AWS Pipeline project end to end | Event Driven | Data Engineers

Описание: In this video , I will explain you that what are the concepts we are going to learn further in the playlist.

Playlist link :    • AWS Data Pipeline Project end to end | Eve...  
========================================================
Event Driven Pipeline Learning’s in AWS
========================================================

• Overall billing cost of the workshop is : INR 20 / $0.21
• Part 1 : Architecture Diagram for Event Driven pipeline use case
• Part 2 : Creation of s3 bucket & SNS topic
• Part 2: Creation of SQS and how SQS queue will subscribe the SNS topic
• Part 2 : How to send test message from SNS and provide access permission to SNS service so that it can send message to SQS
• Part 2 : How to check the content of message available in SQS using polling method & purge the queue
• Part 3: First glue ( python shell ) job and how to grab command line arguments using getResolvedOptions
• Part 3: Configure a IAM role having required permissions so that it can run glue job
• Part 3: How to hardcode the command line arguments and use them to test the glue job
• Part 4: Create first lambda function using python
• Part 4: In lambda , we will learn to process the event coming from SQS and fetch the required information from it
• Part 4: trigger a glue job from lambda function
• Part 4: Learn to create a test event for lambda and use it to test the lambda
• Part 4: How to create s3 notification to send message to SNS topic and configure the access policy of SNS so that s3 can send message to it
• Part 5: Create a crawler and glue database
• Part 6: Create a PagerDuty service and slack workspace. Do a integration between PagerDuty service and slack channel
• Part 6: Create a Cloudwatch integration in PagerDuty and use it for SNS subscription
• Part 6: Use the concept of deadletter queue
• Part 6: Create a Cloudwatch alarm for lambda failure and use to raise an alert in PagerDuty
• Part 7 Create a EventBridge rule for glue job failure