Pub/Sub - Dataflow - BigQuery. How to get your data ready to tell you a story?

Описание: This is the first of a series of episodes I will create on how to use the services available in the Google Cloud Platform for Data Engineering and more.

In this Pub/Sub - Dataflow episode, you are going to learn how to send some messages ( sensor data from highway speed sensors ) to Pub/Sub and start a Dataflow pipeline template to read them and sink it to BigQuery for further analytics using Google Data Studio (average link speed, max speed and min speed on each highway ).

In this video, I will show you how to download and install GCloud SDK on your local pc and how to use the UI.
https://cloud.google.com/sdk/docs/dow...

The data we are going to use can be downloaded at this link.
https://github.com/GoogleCloudPlatfor...

The JSON format of the data is also available at this link.
https://storage.googleapis.com/gcprai...

Please let me know if you need any other video to explain some concepts you are curious about in more detail.

Enjoy, Like, Subscribe and Share
If you liked the video, want more, and are part of deciding what's next, please support us at https://www.patreon.com/cslearning?fa....
  / cslearning86  
  / cslearning3  
https://www.facebook.com/profile.php?...
Follow my blog on https://cslearning.blog/
Thank you

00:00 Introduction
02:24 How to install GCP SDK
04:19 Create Pub/Sub Topic
09:07 Publish data on the topic
11:30 Set up BigQuery tables
16:56 Create Dataflow pipeline sending CSV
24:52 Transform CSV To JSON
27:26 Correct BigQuery Schema
32:53 Creating a new Dataflow pipeline
37:33 Query the data
38:19 Show data in DataStudio
40:21 Final words