Databricks Streaming Tables Explained | Spark Declarative Pipelines Bronze Layer Episode 2

Описание: 🚀 Full Databricks Lakeflow Masterclass (32+ Episodes)

   • Databricks Lakeflow Masterclass  

📚 Start the course here:

1️⃣ Lakeflow Architecture
   • Databricks Lakeflow Explained (2026) | Arc...  

2️⃣ Lakeflow Connect
   • 1️⃣ Lakeflow Connect Explained (2026) | Da...  

This video is Episode 2 of the Databricks Lakeflow Declarative Pipelines Masterclass 2026, where we build production-grade data pipelines using Spark Declarative Pipelines.

In this episode we focus on Streaming Tables, the foundation of the Bronze layer in a Lakeflow pipeline.

Streaming tables allow you to continuously ingest raw data into the Lakehouse using Auto Loader and Spark Structured Streaming, while preserving the full data history.

Using the @dp.table decorator, we define declarative ingestion pipelines that automatically process new data, maintain checkpoints, and scale efficiently.

What You Will Learn

• What Streaming Tables are in Databricks Lakeflow
• The role of the Bronze layer in Medallion architecture
• How to configure Auto Loader for incremental ingestion
• Schema inference vs explicit schema definition
• Handling schema evolution in streaming ingestion
• Working with JSON, CSV, and Parquet ingestion pipelines
• Adding metadata columns for observability and debugging
• Performance optimization for high-volume ingestion pipelines

GitHub Code

All code used in this tutorial is available here:

https://github.com/AhmedMahmoud2/data...

Example file used in this episode:

02_streaming_tables_bronze.py

Lakeflow Declarative Pipelines Masterclass 2026

Episode 1 — Introduction to Spark Declarative Pipelines
Episode 2 — Streaming Tables (Bronze Layer)
Episode 3 — Materialized Views (Silver Layer)
Episode 4 — Data Quality Expectations
Episode 5 — CDC Pipelines
Episode 6 — Schema Evolution
Episode 7 — Slowly Changing Dimensions (SCD Type 2)

▶ Previous Episode
Databricks Lakeflow Pipelines Explained | Spark Declarative Pipelines Introduction (Episode 1)
   • Databricks Lakeflow Pipelines Explained | ...  

▶ Next Episode
Databricks Materialized Views Explained | Spark Declarative Pipelines Silver & Gold Layer Episode 3
   • Databricks Materialized Views Explained | ...  



About the Author

Ahmed Mahmoud
Principal Data Engineer & AI Architect

Sharing practical tutorials on:

• Databricks
• Lakehouse Architecture
• Data Engineering
• AI-ready Data Platforms