Ep04 - Data Over Coffee - Data Pipelines

Описание: From raw data to reliable insights, how modern teams actually move and clean data
Welcome back to Data Over Coffee, where Arjun and Ramon take real-world data and tech conversations out of the meeting room and into a relaxed chat.

Today we unpack data pipelines, the practical heart of analytics. If you have ever wondered how messy source data becomes clean, trusted insight, this one is for you.

You will learn
What a data pipeline is in simple terms
Batch refresh vs streaming, when to use each
ETL vs ELT, why it matters for cloud cost and clarity
The Medallion approach, bronze, silver, gold layers for steady progress
Common real-world challenges, networking rules, odd schemas, rising cloud bills
Where AI helps today, faster design and checks, and where humans still lead

Key takeaways
Start with the simplest refresh you need, daily beats real time for most reports
Prefer ETL when clarity and shared understanding matter, transform early so names and formats make sense
Use the Medallion model to build trust step by step, bronze ingest, silver clean, gold publish as data products
Watch costs, compute is what you pay for, schedule sensibly and retire unused jobs
AI accelerates design and validation, you still own the business rules

Chapters
00:00 Music
00:11 Welcome back, what this show is about
02:09 Quick week updates, weather in Auckland
02:59 New laptop chat, Mac vs Windows for data work
05:33 Main topic, what is a data pipeline
07:55 Batch refresh vs streaming, real examples
13:01 How often to refresh, practical guidance
13:46 ETL vs ELT in the cloud, pros and cons
20:05 Why tidy names and formats matter for analysts
21:31 The water filter analogy for pipelines
23:04 Where AI helps, faster filter design, guardrails still needed
30:35 The Medallion model, bronze, silver, gold
35:28 Biggest challenges, networking, schemas, JSON, swagger, vendors
40:17 Cloud cost surprises and how to avoid them
43:59 Final tips, start small, automate the boring parts
45:07 Wrap up and thank you

Connect with us
Excel in BI, Ramon
Website: https://www.excelinbi.com
LinkedIn:   / razamorano  

Softcraft Studio, Arjun
Website: https://softcraftstudio.net
LinkedIn:   / arjunkataria  
Both of us are based in Auckland, New Zealand.