8. Azure Data Factory | ADLS Gen2 to ADLS Gen2 | Lookup + ForEach + Copy (Metadata Driven)

Описание: In this video, we demonstrate a real-time Azure Data Factory (ADF) scenario to copy data from ADLS Gen2 to ADLS Gen2 using a metadata-driven approach.

We use Lookup / Script activity to read configuration data and ForEach + Copy activity to dynamically copy multiple folders.

This pattern is heavily used in enterprise Azure Data Engineering projects.

📌 Scenario Covered:
Source : Azure Data Lake Storage Gen2
Sink : Azure Data Lake Storage Gen2
Tool : Azure Data Factory

📌 ADF Components Used:
✔ Linked Services (2)
Source ADLS Gen2
Sink ADLS Gen2
✔ Datasets (2 – Parameterized)
Defines structure & location of data
✔ Pipeline (1)

📌 Pipeline Flow:
1️⃣ Lookup / Script Activity
Reads config file (CSV / JSON / SQL result)
Converts rows into items
Options:
✔ First row only
✔ All rows

2️⃣ ForEach Activity
@activity('Lookup').output.firstRow
@activity('Lookup').output.value
Execution modes:
✔ Sequential (one-by-one folders)
✔ Batch (parallel execution – up to 50 items)

3️⃣ Copy Activity
Source Dataset → Sink Dataset
Fully parameterized

📌 Sample Folder Mapping:
global/india/raw/cust → global/india/bronze/cust

📌 Configuration Columns:
src_container
src_subfolder
sink_container
sink_subfolder

📌 Expressions Used:
@item().src_container
@item().src_subfolder

📌 Advanced Scenarios Explained:
✔ Copy multiple folders from On-Prem File System to ADLS Gen2 using Script + ForEach + Copy
✔ Handle large metadata (5000 rows / 4MB data)
✔ Script activity output usage:
@activity('script').output.resultSets[0].rows
✔ Copy multiple tables from On-Prem SQL Server to Data Lake using Script + ForEach

👉 Join now in Cloudpandith Training Institute and enhance your career by learning MS Azure Data Engineering course by real-time experts and with live project scenarios.

👩 For more details about Course Content visit:
https://www.cloudpandith.com

👩 Azure Data Engineer Demo Every Saturday 10 AM.

✍️ Message for more Details on WhatsApp:
https://wa.me/918904424822

✍️ Book Slot with Trainer to Meet:
https://meetpro.club/mallaiahsomula

🫵 Read about your Trainer:
https://goo.gl/maps/9jGub6NfLH2jmVeGA

☎️ Call Us @ 8904424822

Join our WhatsApp Group for Free Question & Answers Documents:
https://chat.whatsapp.com/Gow6Qxi77FN...

Join our Telegram Group for Free Question & Answers Documents:
https://t.me/+PGnxzlaFpvo1MWJl

👍 Like | Share | Subscribe to CloudPandith for more Azure Data Factory real-time scenarios
#AzureDataFactory
#ADF
#ADLSGen2
#LookupActivity
#ForEach
#ScriptActivity
#MetadataDriven
#AzureDataEngineering
#CloudPandith