This blog covers the fundamental characteristics of agentic AI in simple terms.

The simplest way to picture an AI agent is as a digital teammate. An end user provides a prompt or desired outcome. The agent gathers relevant information, interprets the goal, creates a plan, uses different tools where needed, and verifies results through human review points before delivering the required outcome.

The main charactertsics of Agentic AI are:

Keep learning.. Keep growing.

The diagram shows a simple approach: centralise standards and operations, but keep data processing and storage local to each cloud and region.

1. Common services ( At Top):

The top layer contains services shared across all Databricks environments.

Central CI/CD platform

A central DevOps platform manages how code, notebooks, workflows, and infrastructure are deployed. Instead of manually creating jobs in every Databricks workspace, teams use automated pipelines. The same code can be tested and deployed to AWS or Azure with environment-specific configuration. This improves consistency and reduces manual errors.

Central Identity Provider

A central Identity Provider manages user access through services such as

1. Single Sign-On (SSO)
2. SCIM user and group provisioning
3. Enterprise security groups

Users should log in with the same corporate identity regardless of whether they use Databricks on AWS or Azure. Access should be granted through groups, not individual user permissions.

Common observability

A common observability tool, such as Datadog, can collect operational information from all platforms. It helps teams monitor:

1. Job failures
2. Pipeline delays
3. Cluster usage
4. Platform health
5. Cost and resource consumption
6. Security and audit events

This creates one operational view across clouds and regions.

2. AWS and Azure operate as independent platforms

The lower part of the diagram shows separate Databricks deployments in AWS and Azure.

On AWS, each regional Databricks workspace uses Amazon S3 for storage.

On Azure, each regional Databricks workspace uses Azure Data Lake Storage Gen2 (ADLS Gen2).

Each workspace should have its own:Databricks workspaceUnity Catalog and governance setupCloud storage

Each workspace should have its own:

1. Databricks workspace
2. Unity Catalog and governance setup
3. Cloud storage
4. Network configuration
5. IAM roles or managed identities
6. Encryption keys
7. Compute policies
8. Monitoring and cost controls

This separation is important because a Databricks workspace is regional. Simply using the same AWS account or Azure subscription does not make the platform multi-region automatically.

Conclusion

A successful multi-cloud, multi-region Databricks design is not about creating one giant global platform. It is about creating independent regional data platforms with:

1. Common identity
2. Common deployment standards
3. Common monitoring
4. Consistent governance
5. Local storage and processing
6. Controlled data sharing

In short: centralise the standards, decentralise the execution.

AI Fundamentals — What is Context Layer ?

Many business organizations fail in AI implementation not because of a lack of technology or data, but because of missing or fragmented context. Business context is often scattered across systems, documents, processes, and teams. As a result, AI systems cannot easily understand what the data means, how it should be used, or which business rules apply. When that context is missing, AI tries to fill the gaps through inference, which can lead to incorrect or unreliable outcomes and not aligned with business context.

With the evolution of AI agents, the problem is now more complex. Agents often try to improve answers through loops, reasoning steps, and iterative processing. While this can increase accuracy in some cases, it can also make responses slower, more expensive, and still incorrect if the underlying context is incomplete.

To generate meaningful outcomes from data through AI intelligence, organizations need a centralized context layer. This layer helps AI systems understand data in the right business, technical, governance, and operational context. It enables organizations not only to implement AI and agentic solutions more effectively, but also to solve real business challenges and create measurable business value.

The fundamental question, what is a context layer?

A context layer in a data platform is the layer that transforms raw data into something understandable, discoverable, trustworthy, and usable by people, applications, analytics platforms, and AI systems. It sits above the raw storage and processing layers and adds the business, technical, governance, and operational meaning required to use data effectively.

For an enterprise data ecosystem, data context refers to the information that helps people, applications, analytics platforms, and AI systems understand what data means, where it comes from, how it should be used, and whether it can be trusted. It typically includes:

· Metadata — Technical details about the data, such as data type, format, schema, structure, and source system.

· Business definition — The business meaning of the data, including definitions, rules, and interpretation.

· Lineage — The end-to-end flow of data from source to target, including transformations, dependencies, and consumption points.

· Data ownership — Information about accountable owners, such as business owners, data owners, data stewards, and technical custodians.

· Data quality — Details about the quality of the data, including quality rules, metrics, completeness, validity, accuracy, and null versus non-null percentages.

· Relationships — How the data relates to other data entities, domains, systems, or business concepts.

· Freshness — How frequently the data is updated, refreshed, or synchronized, and when it was last available.

· Granularity — The level of detail represented by the data, including what each row or record represents from a business perspective.

· Classification — The sensitivity and regulatory classification of the data, such as public, internal, confidential, personal, or restricted.

· Usage — Guidance on how the data can be used, including access restrictions, permitted use cases, departmental limitations, and compliance constraints.

The high-level flow for context layer is:

In practice, the context layer helps answer critical questions such as:

· What does this field mean?

· Can I trust this dataset?

· Who owns it?

· How fresh is the data?

· Which reports, dashboards, models, or applications use it?

· Which other datasets or entities can it be joined with?

· Am I allowed to use it for this purpose?

· Which metric definition should I use?

For AI and self-service analytics, the context layer is especially important. It prevents users, models, and AI agents from guessing meanings, applying incorrect joins, using inconsistent metric definitions, or exposing restricted data. In a mature data platform, the context layer becomes the shared understanding between data producers, data consumers, governance teams, analytics platforms, and intelligent applications.

Building any enterprise data platform starts with a foundational step — bringing data from various source systems into a centralized platform. This process is known as Data Ingestion. Today, the data engineering landscape offers a wide range of ingestion tools which are no-code, connector-based, and purpose-built to replicate or move source system data into a target platform with minimal friction.

In this blog, we explore Lakeflow Connect — Databricks’ fully managed, no-code/low-code data ingestion solution designed to simplify and accelerate how teams bring data into the Databricks Data Intelligence Platform.

The key features for LakeFlow connect is as follows:

The following diagram provides high level flow how LakeFlow connect work.

To make this concrete, let’s walk through a live demo using an AWS RDS PostgreSQL database as the source system.

Before you begin, make sure the following are in place:

1. AWS RDS PostgreSQL instance — a running Postgres database on AWS RDS.
2. Network connectivity — Databricks must be able to reach the Postgres instance. For this demo, all inbound connections on port 5432 have been allowed.
3. Database user — a DB user provisioned with the necessary privileges to read from the source.

Note: Setting up the AWS RDS PostgreSQL instance itself is outside the scope of this blog. We assume it is already available.

Step -1 : Configure AWS Postgres Database for configuration.

1. Enable the logical replication on RDS database, To do the same, update the value of rds.logical_replication = 1 in parameter groups.

Note — You can not update the default parameter group, create a new parameter group of the same database type, update the logical_replication parameter and attach the new parameter group to the database.

2. Create a dedicate user for the lakeflow connect

a. Run the below commands in Postgres ( connect postgres database via any client tool and execute the below command)

· create user databricks_lakeflow with password ‘xxxx’

· grant connect on database postgres to databricks_lakeflow

· grant usage on schema public to databricks_lakeflow

· grant select on all tables in schema public to databricks_lakeflow

· grant rds_replication to databricks_lakeflow

Note:

If you are not using AWS RDS database, you need to give the replication privilege to the user — ALTER USER <<user>> WITH REPLICATION;

3. Create a table, in this case we created a customer table and insert sample 100 rows.

        CREATE TABLE public.customer (
    id BIGSERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    email_id VARCHAR(255) UNIQUE NOT NULL,
    telephone VARCHAR(20),
    enroll_date DATE NOT NULL
);

Give the permission to the table as well ( We faced select issue) so explicitly provide the access.

Grant select on table public.customer to darabricks_lakeflow.

4. For each table, to replicate run, run the below command in postgres. For our customer table, the command is:

ALTER TABLE public.customer REPLICA IDENTITY DEFAULT;

This tells PostgreSQL what columns to include in the WAL (Write-Ahead Log) for CDC. DEFAULT uses the primary key. Use FULL if the table has no primary key.

5. Create a Publication and Replication Slot

CREATE PUBLICATION databricks_pub FOR ALL TABLES;

If you want to run for specific table, you can put specific table details

CREATE PUBLICATION databricks_pub FOR TABLE public.customer;

SELECT pg_create_logical_replication_slot(‘databricks_slot’, ‘pgoutput’);

Publication = what data to replicate. Replication Slot = the pointer that tracks how far Databricks has read the change log.

Step -2 : Set up Lakeflow connect in Databricks

1. Go to Databricks UI -> Data Engineer ( Left bar) -> Click on Data Ingestion.

2. Search for Postgres -> select postgres -> click on create connection

3. Give the connection name, give the postgres user name and password and host details and click on create connection.

4. Select the connection and click next, give the pipeline name and select catalog and schema name to whom data will be ingested and click next.

5. Search for postgres and it will list the source details. Select the schema and table name, then select the cursor column ( on which incremental data can be fetched) and the primary key and make historical tracking ON and then click next.

6. Select the destination and then finally click save and run the pipeline. User can adjust the schedule of the pipeline as well.

7. Go to Jobs and Pipeline ( left menu) and look for the job created above. It will start running.

8. To validate the data, run the sample query in SQL editor

9. To run incremental, insert sample data to the table and wait for the job to run in defined schedule.

This gives a complete e2e details from zero to a live CDC data ingestion pipeline using Lakeflow connect.

Keep learning, keep growing and keep sharing.

At its core, an LLM (Large Language Model) takes an input and produces an output — but its answers come strictly from what it learned during training, not from your specific domain or private data.

RAG (Retrieval-Augmented Generation) bridges that gap by enabling the model to answer using up-to-date, domain-specific, and private information. A RAG-powered AI application searches a knowledge base first, pulls the most relevant information, and combines it with the user’s prompt to deliver a precise and contextually accurate response. In simple terms:

Without RAG — the AI returns a generic answer that may have no relevance to your context

With RAG — the AI searches your knowledge base, finds the best match, and feeds that to the LLM for a focused, accurate output

The three core steps of RAG are:

In this blog, we focus on chunking — one of the most critical components of any RAG application. Chunking is the process of breaking large text into smaller, meaningful pieces so an AI application can process and retrieve them efficiently.

We will walk through the core chunking strategies conceptually, and the next blog will cover hands-on implementation. To get the best results from a RAG pipeline, choosing the right chunking strategy matters. Several approaches are available, each suited to different document types and use cases.

The below table provides the comparison matrix for different Chunking Strategies.

Databricks: Part-14: Time Travel & Vacuum !!

In the world of Data Lakehouse architecture, understanding Time Travel and Vacuum are two critical concepts. This blog post will take a deep dive into these functionalities, explaining their features, providing examples, and outlining best practices w.r.t Databricks Delta Tables.

What is Time Travel?

Time Travel in Delta Lake provides the ability to query previous versions of the Delta tables. This is achieved by leveraging the immutable snapshots and transactional logs that Delta Lake maintains.

Key Features of Time Travel:

1. Data Accessibility: As long as the transactional log and the underlying data files exist, user can access any past version of your data.
2. Powerful Use Cases: Time Travel is invaluable for:

• Auditing: Trace back changes to your data over time for compliance and analysis.
• Rollback: Easily revert to a previous good state in case of data corruption or erroneous updates.
• Disaster Recovery: A lifesaver in scenarios like accidental table drops or mass deletions, allowing you to restore data effortlessly.

1. Default Retention: By default, Delta Lake retains transaction logs and data files for 30 days. This means you can “time travel” back up to a month into your table’s history.
2. Configurable Retention: This default retention period is not rigid. You can easily modify it using the delta.logRetentionDuration TBLPROPERTIES. For example, user might increase it for highly sensitive data or decrease it for less critical tables to save storage.

Understanding Vacuum

While Time Travel keeps historical data, VACUUM serves the opposite but equally important purpose: cleaning up old, unreferenced data files. Its primary goal is to free up storage space by removing data files that are no longer part of any active snapshot of your Delta table.

Key Features of Vacuum:

1. Default Deletion Threshold: By default, VACUUM targets files older than 7 days for deletion. This period ensures that files still potentially referenced by short-term time travel windows or concurrent queries are not prematurely removed.
2. Manual/Scheduled Execution: VACUUM is not an automatic process. It needs to be executed manually or scheduled as a regular job to effectively manage storage. This gives you control over when and how frequently cleanup occurs.
3. Configurable Retention: Similar to Time Travel, the retention period for VACUUM can be adjusted. You can modify this via the delta.deletedFileRetentionDuration TBLPROPERTIES, allowing you to fine-tune how long old files are kept before being eligible for deletion.

Example: Time Travel and Vacuum in Action

Let’s walk through an example to illustrate how Time Travel and Vacuum work with a Databricks Delta table.

Step 1: Create a Delta Table and Insert Data

First, we’ll create a simple employee table and populate it with some sample records.

SQL

CREATE OR REPLACE TABLE employee (

id INT,

name STRING

);

INSERT INTO employee VALUES

(1,’sanjeeb1'),

(2,’sanjeeb2'),

(3,’sanjeeb3'),

(4,’sanjeeb4'),

(5,’sanjeeb5');

SELECT * FROM employee;

Step 2: Verify and Update Default Retention Properties

Now, let’s check the default Time Travel and Vacuum settings for our new table and then modify them.

SQL

DESCRIBE DETAIL employee; — Provides detailed metadata including retention properties

SHOW TBLPROPERTIES employee; — Shows specific table properties

— Let’s update the time travel log retention to 90 days and deleted file retention for vacuum to 30 days

ALTER TABLE employee SET TBLPROPERTIES (

‘delta.logRetentionDuration’ = ’90 days’,

‘delta.deletedFileRetentionDuration’ = ’30 days’

);

Step 3: Perform Data Modifications and Explore Time Travel

We’ll perform some DELETE and UPDATE operations to create different versions of our table, then use Time Travel to query past states.

SQL

— Create a new version by deleting records

DELETE FROM employee WHERE id > 4;

— Create another version by updating a record

UPDATE employee SET name=’sanjeeb_update’ WHERE id = 3;

— View the current state of the table

SELECT * FROM employee;

— To see all versions (snapshots) of the table

DESCRIBE HISTORY employee;

— Time travel by version number (e.g., version 1)

SELECT * FROM employee VERSION AS OF 1;

— Time travel using the ‘@v’ syntax

SELECT * FROM employee@v1;

— Time travel by timestamp (replace with an actual timestamp from your history)

— SELECT * FROM employee TIMESTAMP AS OF ‘2025–12–27 10:00:00.000’;

Step 4: Using Vacuum to Clean Up

Finally, we’ll demonstrate how to use VACUUM. Remember, VACUUM will only remove files that are older than the delta.deletedFileRetentionDuration (which we set to 30 days) and are no longer referenced by active snapshots. Since we just created the table and performed recent operations, there might not be many files eligible for deletion immediately.

SQL

— 1. Always do a dry run first to see which files would be deleted

VACUUM employee DRY RUN;

— 2. To actually run the vacuum and delete files

VACUUM employee;

— Note: Since we’ve just created the table and changed retention,

— it’s likely no files will be deleted by VACUUM immediately,

— unless enough time has passed for files to exceed the retention period.

Best Practices for Time Travel and Vacuum

To effectively manage your Delta Lake tables, consider these best practices:

1. Always execute DRY RUN first before running actual VACUUM. This allows you to preview the files that would be deleted without actually removing them, preventing accidental data loss.
2. Always ensure Time Travel parameter (delta.logRetentionDuration) is greater than the Vacuum parameter (delta.deletedFileRetentionDuration). If your VACUUM retention is shorter than your Time Travel retention, VACUUM might delete data files that Time Travel still expects to find, potentially breaking your ability to query older versions. A good rule of thumb is delta.logRetentionDuration >= delta.deletedFileRetentionDuration.
3. Do not set retention values too low. Setting delta.logRetentionDuration or delta.deletedFileRetentionDuration to very low values can have adverse effects. A low logRetentionDuration might limit your ability to recover from disasters or audit changes. A low deletedFileRetentionDuration could potentially lead to VACUUM deleting files that concurrent long-running queries might still need, leading to query failures.

ML life cycle is the step by step process which outlines understanding the business problem statement to developing, deploying and maintaining ML models

In this blog, I presented ML life cycle in a single page. This will help everyone who wants ML Life cycle fundamentals.

Keep Learning.. Keep Growing.

Databricks: Part-13: Fundamentals — All About Databricks Genie!!

Databricks Genie provides a platform to ask questions about your data in natural language. This is designed to enable the easy access of data to end business users to analyze the data without writing complex SQL code. This is a self-analytics platform for data engineer to business users.

Key Capabilities of Genie

1. Natural language to SQL — Ask questions in plain and simple English

2. Automatic SQL generation — Converts natural language to SQL code

3. Context Knowledge — user can give the context to Genie and it also understands the follow up questions and maintain context history for better response.

4. Schema Understanding — From Unity catalog, data structure, Genie understand the data structure. for better refinement, user can provide the more instructions and description about the data.

5. Data Profiling — Genie provide the details on data quality insights by automatic profiling the data.

6. Intelligent Data Relation discovery — Genie identifies relationship between tables which enable to wrote complex query which involved join as well.

7. Auto generated Charts — Depending upon the user prompt, Genie can create charts which can be editable and added to Dashboard as well.

8. Row Level Security — Genie support access control set in unity catalog.

Let’s do a demo. This demo is created using Databricks free tier.

Step 1: Create a Genie workspace

1. Login to Databricks -> Click on Genie -> Click on New to create a new Genie workspace -> select your data set ( tables) -> click create

2. Give a suitable name of Genie space, here we provided the name as demo_genie

Step -2: Explore Sample questions

1. Genie provided a sample questionnaires for the users so that they can understand the data. In this case, users can select any sample questions and see the result.

Step -3: Additional configurations for Genie

1. Add Data — User can add more data sets to Genie space, by select the data -> Add data -> select the data set -> confirm. This enable user to ask complex question which involve joins.

2. Instructions — To provide more context to Genie, users can add more context to Genie space by select instruction -> provide the context details. User can provide join details and sample query for Genie as well and click Save.

Step -3: Provide sample prompt and verify the result

Once the details have been configured, users can ask any question in natural language, genie will convert the natural language to SQL and run the sql and provide the details to users. Users can also verify the SQL and if they want to modify it, they can do it.

Here we have asked a simple question “Provide me the total number of customers”

Let’s ask another complex question to Genie,

Question: Provide a graph which provide the total number of customer w.r.t state

Summary:

In this blog, we have explored Databricks Genie, how to configure and use it. Users can onboard many datasets, configure joins and set more details instructions to get more value from Genie.

Keep learning, keep growing.

Databricks: Part-12: Call Lambda Function from Databricks !!

For any enterprise data platform, integration with external data source or external services is required sometimes. This enable to execute some process in the external services.

In this blog we will explore step by step guide to connect external service like AWS lambda and execute a AWS lambda from Databricks notebook. The same approach can be followed for services like AWS Glue and AWS secrets manager etc.

To configure connections to external services, we need to create a securable object in Unity catalog called service credential. A service credential is a long term cloud credential which provide the access to external services.

The below steps are followed to call a lambda function from Databricks.

Let’s do the demo.

Step -1: Create a IAM Role and Policy in AWS

To execute AWS lambda function from AWS, first create a IAM role and attach the require policy to that Role. This role arn will be added to service credential at Databricks side.

1. Go to AWS management console -> search IAM -> click on IAM Role to create a IAM role and click create role -> select the AWS Service -> select Lambda and click Next

2. Do not select any policy, we will customer policy, give the role name (in this case we gave the name as lambda-databricks, the remaining settings are default and click create role to create it.

To create trust between Databricks and AWS, update the trust policy with below code

Code: Change the role arn, and Databricks account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
                    "<<Give the arn of the role>>"
                ]
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<< Give the databricks account id>> "
                }
            }
        }
    ]
}

3. Create a policy and attach the policy to the role.

To create a custom policy, click on policy on IAM role -> click create policy -> give the policy name and click Json and paste the below policy ( Note — change the role arn).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:DescribeStacks",
                "cloudformation:ListStackResources",
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricData",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "kms:ListAliases",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListRolePolicies",
                "iam:ListRoles",
                "lambda:*",
                "logs:DescribeLogGroups",
                "states:DescribeStateMachine",
                "states:ListStateMachines",
                "tag:GetResources",
                "xray:GetTraceSummaries",
                "xray:BatchGetTraces"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": "lambda.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "logs:FilterLogEvents",
                "logs:StartLiveTail",
                "logs:StopLiveTail"
            ],
            "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/*"
        },
        {
      "Action": ["sts:AssumeRole"],
      "Resource": ["<<Give the IAM role arn>>"],
      "Effect": "Allow"
    }
    ]
}

Attach the policy to the role we created in the previous step.

Step -2: Set up Service credential in Databricks.

To set up a service credential in Databricks, login to Databricks (In our case we used Databricks Free Edition) -> click on Catalog -> click settings icon and select create external connection -> Select Service credential -> Give the credential name -> give the role arn (from AWS) -> click create

To test validate the credential, click validation configuration, if all setup is good, the Test connection will successful.

Step -3: Create a lambda function in AWS

To create a test lambda function, go to AWS management console -> search for lambda -> click create function -> select Author from Scratch -> give the lambda function name -> select Run time python (as we will create the lambda in python) -> select an execution role (here we have used a lambda execution role)

Run the below sample code for lambda , click on deploy to save the code, click on Test and create a test event to test the code.

Code:

import json

def lambda_handler(event, context):
    # TODO implement
    print("This is my databricks lambda")
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Step -4: Create a note book and call lambda function from Databricks

Create a note book in Databricks. To interact with AWS service, we need to install the boto3 package.

Command:

%pip install boto3

Use the below code to call lambda from Databricks.

Code:

import boto3
import json

# --- Lambda details ---
lambda_function_name = "test-databricks"  
region_name = "ap-south-1"                   

boto3_session = boto3.Session(botocore_session=dbutils.credentials.getServiceCredentialsProvider('lambda-datrbricks'), region_name=region_name)
lambda_client = boto3_session.client("lambda")

# --- Payload to send to Lambda ---
payload = {
    "key1": "value1",
    "key2": "value2"
}

# --- Invoke Lambda ---
response = lambda_client.invoke(
    FunctionName=lambda_function_name,
    InvocationType="RequestResponse",  # "RequestResponse" waits for result, "Event" = async
    Payload=json.dumps(payload)
)

# --- Parse the response ---
response_payload = json.loads(response["Payload"].read())
print("Lambda response:", json.dumps(response_payload, indent=2))

The high level details of the code is available below.

Summary:

In this blog, we walkthrough the details how aws services can be called from Databricks.

Keep learning, Keep Growing..

Databricks: Part-11: Fundamentals — All About Databricks Alerts!!

Imagine you have a system that keeps an eye on your data, jobs and lets you know the moment something goes wrong. That’s exactly what Databricks Alerts do. It is a powerful tool for monitoring your Databricks environment in real-time. When there is any abnormal behavior, if it spots anything, it sends a notification as per set up. This help organization fix the issue before encounter a bigger problem.

The key features of Databricks Alert is available below.

Let us do a demo to understand how Databricks alerts work. We are using Databricks Free edition for this demo.

This demo will show you how to set up an alert to notify you whenever a job fails. First, we’ll need a table to track our job executions.

Prerequisites

Create a simple table that logs job executions and populate it with some sample data. This table should include a column to track errors.

Command (for creating table)

create table job_execution_log
(job_id bigint,
job_name string,
start_time timestamp,
end_time timestamp,
status string,
error_description string)

Note — The status column in the table contain the job status which is R -> Running, S -> Success and F -> Failure

Setting up the Databricks Alert

Step -1: Create an alert

Go to Databricks Workspace -> Navigate to SQL -> Click on Alerts -> Click on Create Alert to create an alert

Step -2: Provide the details on Alert configuration

To set up an alert, define the query. In this case we defined the query from last 30 Mins, if the failure job count > 0, send an alert. You can test the query by click Run command.

Query:

select count(1) as error_count
from dev.bronze.job_execution_log
where status = 'F'
and end_time >= current_timestamp() - INTERVAL '30' MINUTES

Set the below conditions.

· Under the Condition tab, select SUM to get the total number of errors.

· Choose the column that tracks errors (e.g., `error_count`).

· Select the condition `>` (greater than) and set the value to `0`. This means the alert will trigger if even one job has failed.

· Click Test condition to see if your setup works.

Step -3: Schedule the Alert

Under Schedule, choose how often you want the alert to check the query. The schedule uses a format called cron syntax.

Step -4: Choose Notification Method

In the Notification section, select the users who want to be notified when the alert is triggered.

Step -5: Save the Alert

Give the alert a clear name, such as `job_failure_alert`, and choose a folder to save it in. Click Create to finish.

Step -6: Test the alert

User can manually run the alert to see if it works. If everything is set up correctly, user will receive a notification, such as an email, telling you that the alert was triggered.

User can find all alerts in the Alerts section. To delete an alert, open it, click the three dots (`…`), and select Delete.

Summary

In this blog, we have explored Databricks alert which help in monitoring and notifying different target when certain conditions meet.

keep learning keep growing.