Incremental Loading in Apache Airflow - Handle Large Data More Efficiently
Автор: KP Data Dev
Загружено: 2025-02-27
Просмотров: 534
Описание:
Big data can be difficult data. Design your Airflow pipelines to process data more efficiently with incremental loading.
We'll see how to use template variables to process small chunks of our source data.
👉 Sign up for the newsletter: https://kpdata.dev/newsletter/
Chapters:
00:00 Introduction
00:25 What is Incremental Loading?
01:53 Airflow Data Interval
03:50 Example DAG: Donut App Orders
07:05 Building the DAG
10:18 Rendering Template Variables
11:49 Re-Running a DAG
14:53 Outro
Code:
https://github.com/kishanpatel789/kp_...
Blog post:
https://kpdata.dev/blog/airflow-incre...
Other Resources
Airflow Docs: https://airflow.apache.org/docs/apach...
Airflow Template Variables: https://airflow.apache.org/docs/apach...
#airflow #dataengineering #python
Повторяем попытку...
Доступные форматы для скачивания:
Скачать видео
-
Информация по загрузке:
