Workshop: Building Near-Real Time Data Products with SQL

Описание: In this workshop Roy and Jason from Upsolver show how to build data products. Upsolver is simplifying pipelines for Data Engineers: No custom coding. No orchestration. No infrastructure maintenance. Just SQL!

00:00 - Intro.
02:08 - Guest Intro.
03:18 - Roy's (Upsolver) background.
04:25 - Jason's (Upsolver) background.
05:40 - Today's agenda.
06:29 - Skills from today's workshop.
08:10 - Time taken for Realizing value from raw data.
15:00 - Challenges to deliver reliable data.
21:17 - Other data integration strategy.
23:59 - Streaming data & data pipelines.
27:16 - Data streaming pipelines cost!
30:57 - What is a Data Product?
37:27 - Consistency in Modeled view
40:50 - Role of Upsolver in data processing.
45:28 - Storing data in Data Lake.
48:24 - Backfill historic data.
53:18 - Future of the Data Engineer.
58:47 - Workshop Architecture.
1:00:38 - Demo: Build a Data Product on AWS S3 with Upsolver.
1:13:21 - Automatic Detection & Alarms
1:13:50 - Evaluate available datasets
1:19:16 - Jobs visibility
1:23:27 - Demo: Sales analysis pipeline
1:32:42 - Editing AWS pipeline
1:35:12 - Collaboration as a data analyst
1:37:58 - Creating a new Job.
1:42:30 - Pipeline aggregation.
1:45:12 - S3 vs Kafka
1:46:21 - Sync between AWS producer & Consumer through Upsolver.
1:48:38 - Git integration in Upsolver
1:49:49 - Result table
1:54:22 - Setting up a performance Cluster.
1:57:47 - Where can people find you & your website?
2:00:43 - Thank You & Wrap up!

#dataengineering #upsolver #datascience #bigdata