Career·2026-02-27·5 min read·

CI/CD for Databricks with Asset Bundles: deploying pipelines like real software

How to use Databricks Asset Bundles to version, test, and deploy your notebooks and jobs across environments.

For most of the history of Databricks, deploying a pipeline meant manually copying a notebook to a workspace, clicking through the Jobs UI, and hoping nothing broke in production. Databricks Asset Bundles (DABs) changes that. It's Databricks' native infrastructure-as-code framework — your jobs, pipelines, clusters, and permissions defined in YAML, deployed via CLI, promoted across environments with a single command.

This is the workflow I've moved my team to, and it's the biggest operational improvement I've made in the last two years.

What Asset Bundles are

A Bundle is a project — a directory with a databricks.yml config file and your code (notebooks, Python files, SQL files). The bundle defines:

  • Resources: jobs, DLT pipelines, experiments, model serving endpoints
  • Environments: dev, staging, prod with different cluster configs and permissions
  • Variable substitution: different catalog names, storage paths, and secrets per environment
my-lakehouse-bundle/
├── databricks.yml          # Bundle configuration
├── src/
│   ├── bronze_ingestion.py
│   ├── silver_transform.py
│   └── utils/
│       └── delta_helpers.py
└── tests/
    └── test_transforms.py

The databricks.yml file

bundle:
  name: lakehouse-pipelines

variables:
  catalog:
    description: Unity Catalog catalog name
    default: dev
  storage_root:
    description: ADLS root path for this environment

targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: dev
      storage_root: "abfss://dev@storage.dfs.core.windows.net"
    workspace:
      host: https://adb-dev-workspace.azuredatabricks.net

  prod:
    mode: production
    variables:
      catalog: prod
      storage_root: "abfss://prod@storage.dfs.core.windows.net"
    workspace:
      host: https://adb-prod-workspace.azuredatabricks.net
    permissions:
      - level: CAN_VIEW
        group_name: data-analysts
      - level: CAN_MANAGE_RUN
        group_name: data-engineers

resources:
  jobs:
    customers_silver_job:
      name: "[${bundle.target}] Customers Silver Pipeline"
      email_notifications:
        on_failure:
          - data-platform-team@company.com
      tasks:
        - task_key: bronze_ingest
          notebook_task:
            notebook_path: ./src/bronze_ingestion.py
            base_parameters:
              catalog: ${var.catalog}
              storage_root: ${var.storage_root}
          job_cluster_key: pipeline_cluster

        - task_key: silver_transform
          depends_on:
            - task_key: bronze_ingest
          notebook_task:
            notebook_path: ./src/silver_transform.py
            base_parameters:
              catalog: ${var.catalog}
          job_cluster_key: pipeline_cluster

      job_clusters:
        - job_cluster_key: pipeline_cluster
          new_cluster:
            spark_version: "15.4.x-scala2.12"
            node_type_id: Standard_DS3_v2
            num_workers: 2
            spark_conf:
              spark.databricks.delta.schema.autoMerge.enabled: "true"

Deploy workflow

# Install Databricks CLI (v0.200+)
pip install databricks-cli

# Authenticate
databricks auth login --host https://adb-dev-workspace.azuredatabricks.net

# Validate config before deploying
databricks bundle validate

# Deploy to dev (default target)
databricks bundle deploy

# Deploy to prod explicitly
databricks bundle deploy --target prod

# Run a job after deploying
databricks bundle run customers_silver_job

# Destroy all resources in dev (clean up after testing)
databricks bundle destroy

The bundle CLI handles creating, updating, and deleting resources idempotently. Deploy twice and only changes are applied. Delete a resource from the YAML and the next deploy removes it from the workspace.

Environment promotion

The power of bundles is that the same code runs in every environment, with only config differences:

# In dev: uses small cluster, dev catalog, dev storage
targets:
  dev:
    variables:
      catalog: dev
      cluster_node_type: Standard_DS3_v2
      cluster_workers: 1

  prod:
    variables:
      catalog: prod
      cluster_node_type: Standard_DS5_v2
      cluster_workers: 8

Your notebook code doesn't change. The widget catalog receives dev or prod at runtime based on which environment you deployed to.

CI/CD with GitHub Actions

Here's a complete pipeline that deploys to dev on every PR and to prod on merge to main:

# .github/workflows/deploy.yml
name: Deploy Databricks Bundle

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

env:
  DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST_PROD }}
  DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN_PROD }}

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Databricks CLI
        uses: databricks/setup-cli@main

      - name: Validate bundle
        run: databricks bundle validate

  deploy-dev:
    needs: validate
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    env:
      DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST_DEV }}
      DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN_DEV }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - name: Deploy to dev
        run: databricks bundle deploy --target dev

  deploy-prod:
    needs: validate
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - name: Deploy to prod
        run: databricks bundle deploy --target prod

Every PR triggers a deploy to dev — reviewers can immediately run the job in a real environment. Merge to main deploys to prod. No manual steps, no clicking through UIs.

Testing before deploy

The pipeline above doesn't run tests. Here's how to add them:

  test:
    needs: validate
    runs-on: ubuntu-latest
    env:
      DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST_DEV }}
      DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN_DEV }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main

      - name: Deploy test bundle
        run: databricks bundle deploy --target dev

      - name: Run unit tests
        run: |
          pip install pytest pyspark delta-spark
          pytest tests/ -v --tb=short

      - name: Run integration test job
        run: databricks bundle run integration_test_job --target dev

Unit tests run locally with pyspark installed. Integration tests run as a Databricks job against real data. Both must pass before any deploy proceeds.

Secrets management

Never put credentials in databricks.yml. Use Databricks Secrets:

# Create a secret scope
databricks secrets create-scope pipeline-secrets

# Store a secret
databricks secrets put-secret pipeline-secrets sql-server-password

Reference in your notebook:

sql_password = dbutils.secrets.get(scope="pipeline-secrets", key="sql-server-password")

Reference in databricks.yml for cluster config:

spark_conf:
  spark.hadoop.fs.azure.account.key.storage.dfs.core.windows.net: "{{secrets/pipeline-secrets/adls-key}}"

The secret value is never written to the bundle config or to any log output.

Migrating existing notebooks

If you have existing notebooks deployed manually, migration is straightforward:

  1. Export notebooks from the workspace as .py source files (#%% cell separators)
  2. Add them to the src/ directory
  3. Define the corresponding jobs in databricks.yml
  4. Run databricks bundle deploy --target dev and verify
  5. Delete the old manually-deployed jobs from the workspace

The first deploy takes 10-15 minutes per job (cluster creation, import, etc.). Subsequent deploys for config changes take under a minute.

The operational investment in setting up Asset Bundles pays back on the first time you catch a breaking change in dev instead of prod.