Building Data Pipelines Part 2: How To Create A Task In Luigi
Автор: Seattle Data Guy
Загружено: 2021-05-15
Просмотров: 19622
Описание:
How do you build your first data pipeline?
First, you need to pick which framework will work best for you.
In this video we will discuss how you can use the Python library Luigi to start building your first task.
Now, if you want a full course, you can check out the top data engineering courses video I have:
• Best Online Courses For Data Engineers #da...
The code for this video is below.
Code From Video
class StockTwitScrape(luigi.Task):
def output(self):
return luigi.LocalTarget("raw_data.json")
def run(self):
url = "https://api.stocktwits.com/api/2/stre..."
content = requests.get(url).text
with self.output().open("w") as f:
json.dump(json.JSONDecoder().decode(content),f)
If you need data consulting help, then reach out to our team here:
https://www.theseattledataguy.com/
Also, if you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
https://seattledataguy.substack.com/
Check out my Medium here:
/ seattledataguy
Повторяем попытку...
Доступные форматы для скачивания:
Скачать видео
-
Информация по загрузке:
