Dagster for AI & ML Pipelines: What Works, What Breaks, and Why We Chose It

Описание: In this AI Tech Experts Webinar, Nikodem Tadrowski, ML Engineer, shares practical lessons from adopting Dagster as a framework for production-grade data preparation pipelines.

The talk walks through real challenges faced by the team when creating a single source of truth for tabular data and feature logic, and explains why Dagster was chosen over tools like Airflow, Prefect or Luigi.

🔹 why repeated preprocessing breaks ML reliability
🔹 Dagster’s asset-centric model and data lineage tracking
🔹 core concepts: assets, ops, graphs and components
🔹 how metadata, typing and observability help catch issues early
🔹 common pitfalls when abstracting pipelines too early
🔹 when Dagster works well — and when it doesn’t

If you have questions for Nikodem, feel free to ask them in the comments and continue the discussion there!

01:12 Why This Topic?
02:50 What are the available options for Python?
05:33 What does Dagster solve out of the box?
07:30 Core Dagster concepts: assets, ops, graphs
18:49 Lessons learned and final conclusions

Check our website: https://deepsense.ai/
Linkedin:   / applied-ai-insider  

#Dagster #DataPipelines #MLPipelines #AIEngineering #DataLineage #FeatureEngineering #Python #MachineLearning #MLOps #DataWorkflows