Data Analytics Deep Dives - BigQuery using AI/ML to build an AI Lakehouse

Описание: Learn how to build an analytics lakehouse from end to end in Google Cloud. An analytics lakehouse combines the benefits of data lakes and warehouses. We will create a unified platform for users, data engineers, data scientists and analysts. You will see AI along with open architecture, to power your business.

Our lakehouse will use BigLake tables with row/column security, BigLake tables on AWS using OMNI, Apache Iceberg with BigLake Metastore, Serverless Spark, Vertex AI to extract meaning from our images, Unstructured data analytics in BigQuery, along with BigQuery Machine Learning.

All code is on GitHub: https://goo.gle/dagd

Chapters
00:00 Analytics Lakehouse Demo
02:52 Raw zone (Data sources)
03:28 Raw zone (BigLake tables)
04:37 Raw zone (OMNI on AWS)
05:34 Raw zone (Unstructured data)
06:37 Raw zone (Streaming data - Dataflow)
07:52 Enriched zone (Image processing)
09:11 Enriched zone (Apache Iceberg)
10:45 Enriched zone (BigLake Metastore)
11:53 Curated zone (Dataplex Lakes)
12:46 Curated zone (Lakehouse Security)
13:30 Curated zone (Data Catalog / Data Quality / Data Lineage)
14:30 Curated zone (BigSearch / Unstructured AI)
16:30 Curated zone (BigQuery Machine Learning)
18:42 Curated zone (Vertex AI)
19:32 Conclusion