Skip to main content

Snowflake, dbt and Airflow Tutorial Series

A practical, step-by-step tutorial that builds a complete data pipeline with Snowflake, dbt and Apache Airflow on Snowflake's built-in TPC-H retail data. Every part adds one capability, includes a diagram and tested code, and ends with something you can run and check. Read it in order, from Part 1 to Part 10.

1 · Overviewarchitecture2 · Snowflakesetup3 · LoadCOPY INTO4 · dbtsetup5 · Modelingstar schema6 · Testingand docs7 · Incremental+ snapshots8 · Airfloworchestration9 · CI/CDGitHub Actions10 · ProductionmonitoringSnowflake (yellow) → dbt (green) → Airflow (purple) → CI/CD (pink) → production (orange)

Companion code: every file in the series is in tpch_analytics on GitHub. It also runs locally on DuckDB if you do not have a Snowflake account yet.

The 10 parts

  1. Part 1: The stack and what we will build (publishing soon)
    A practical 10-part series building a data pipeline with Snowflake, dbt and Airflow on TPC-H data: architecture, tools, project layout and setup.
  2. Part 2: Setting up Snowflake for dbt (publishing soon)
    Separate warehouses, RAW and ANALYTICS databases, functional roles, a key-pair service user and a credit budget.
  3. Part 3: Loading raw data into Snowflake (publishing soon)
    Stages, file formats, COPY INTO with audit columns, load metadata, ON_ERROR options, COPY_HISTORY checks and Snowpipe.
  4. Part 4: Your first dbt project on Snowflake (publishing soon)
    Install dbt Core with the Snowflake adapter, configure profiles and schemas, declare sources and build your first staging model.
  5. Part 5: Modeling with dbt: staging to marts (publishing soon)
    Staging models, an intermediate model with business logic, and a star schema of facts and dimensions.
  6. Part 6: Testing and documenting with dbt (publishing soon)
    Generic and custom SQL tests, source freshness, documentation and grants.
  7. Part 7: Incremental models and snapshots (publishing soon)
    Incremental models with merge and a load timestamp, safe rebuilds, and customer history with snapshots (SCD type 2).
  8. Part 8: Orchestrating dbt with Airflow (publishing soon)
    Airflow 3 and Astronomer Cosmos: load with COPY INTO, then every model, test and freshness check as its own task.
  9. Part 9: CI/CD for dbt with GitHub Actions (publishing soon)
    Build only changed models on every pull request with state:modified and defer, in isolated CI schemas.
  10. Part 10: Running the pipeline in production (publishing soon)
    Failure alerts, dbt run reports, Snowflake cost and query monitoring, and a runbook.

Who it is for

Data engineers, analytics engineers and BI developers who know SQL and want to see how the modern stack fits together end to end: loading, modelling, testing, orchestration, CI/CD and operations. No prior dbt or Airflow experience is needed.

Related guides

Comments

Post a Comment