Data Pipeline Lab
Welcome to my channel! I’m Emeka Nweke, a data professional specializing in data engineering, data science, and analytics. Here, I share hands-on projects, tutorials, and insights on building robust data pipelines and leveraging cutting-edge tools like Apache Kafka for streaming data, dbt, Databricks, Snowflake, AWS, GCP, Airflow, and GitHub.
My content covers the entire data lifecycle, extraction, transformation, streaming, modeling, and analysis—using Python, SQL, and visualization tools like Tableau, Power BI, and QuickSight to deliver actionable insights and predictive models. With expertise in version control, CI/CD pipelines via GitHub Actions, anomaly detection, and KPI analysis, I aim to provide practical solutions for real-world data challenges.
Subscribe for the latest projects and tutorials tailored for data enthusiasts and professionals passionate about driving data-driven decisions!
End-to-End Real Estate ELT Data Pipeline with Databricks Asset Bundles on GCP - Full Walkthrough
Part 10: Query Data with Natural Language in Unity Using Genie
Part 9: Databricks Asset Bundle on GCP - Automate CI/CD with GitHub Actions
Part 8: Databricks Asset Bundle on GCP - Unit and Integration Testing
Part 7: Databricks Asset Bundle on GCP: Deploy and Verify in Dev and Prod
Part 6: Databricks Asset Bundle on GCP: Develop and Deploy Jobs with Your Pipeline
Part 5: Databricks Asset Bundle on GCP: Pipeline Configuration Explained
Part 4: Databricks Asset Bundle on GCP: Refine with the Gold Layer
Part 3: Databricks Asset Bundle on GCP: Build the Silver Layer
Part 2: Databricks Asset Bundle on GCP: Ingest Raw Data Using Autoloader
Part 1 – Databricks Asset Bundle Full Setup and Project Kickoff
E2E Data Pipeline: dbt Fusion, Snowflake, S3, VSCode dbt Extension, Macros, Unit Testing, CI/CD more
Real-Time IoT Pipeline with Confluent Cloud, Kafka & BigQuery: Healthcare Demo
CI/CD Pipeline for Airflow & dbt: Deployment on Astronomer Cloud
End-to-End Data Pipeline with Airflow, dbt, Cosmos, GCS, BigQuery & more
Part 7 | Automate CI/CD for dbt Models Using GitHub Actions | End-to-End Data Pipeline
Part 4 | Deploying dbt Cloud Models to Production in BigQuery
Part 6 | How to Build Incremental Models in BigQuery with dbt Core (CLI)
Part 5 | How to Set Up dbt Core (CLI) to Connect Locally with BigQuery
Part 3 | Building dbt Cloud Models to Transform Raw Data in BigQuery
Part 2 | Creating BigQuery External Tables from Google Cloud Storage for Your Data Pipeline
Part 1 | Setting Up Google Cloud Service Account and Mock Data Storage for BigQuery Pipelines
Building a Power BI Fraud Detection Dashboard | Step-by-Step Walkthrough
End-to-End Fraud Detection Data Pipeline | AWS, Snowflake, dbt, & GitHub Actions