Databricks Data Quality as Code | Spark Declarative Pipelines Expectations (Episode 4)

Описание: 🚀 Full Databricks Lakeflow Masterclass (32+ Episodes)

   • Databricks Lakeflow Masterclass  

📚 Start the course here:

1️⃣ Lakeflow Architecture
   • Databricks Lakeflow Explained (2026) | Arc...  

2️⃣ Lakeflow Connect
   • 1️⃣ Lakeflow Connect Explained (2026) | Da...  

This video is Episode 4 of the Databricks Lakeflow Declarative Pipelines Masterclass 2026.

In this episode we implement Data Quality as Code using @dp.expect decorators.

You will learn how to integrate validation rules directly into your data pipelines and enforce quality at different levels:

@dp.expect → log violations

@dp.expect_or_drop → remove invalid rows

@dp.expect_or_fail → stop the pipeline

We also explore business rule validations, reference data checks, and temporal validation strategies to ensure reliable data pipelines.

▶ Previous Episode
Databricks Materialized Views Explained | Spark Declarative Pipelines Silver & Gold Layer Episode 3
   • Databricks Materialized Views Explained | ...  

▶ Next Episode
Databricks Medallion Architecture Pipeline | S Declarative Pipelines Bronze Silver Gold (Episode 5)
   • Databricks Medallion Architecture Pipeline...