AzureFunBytes Episode 43 - Intro to @Azure Data Factory with @KromerBigData 
Tags: azure, bigdata, etl, beginners
Published: 2021-05-13 19:08:39

AzureFunBytes is a weekly opportunity to learn more about the fundamentals and foundations that make up Azure. It's a chance for me to understand more about what people across the Azure organization do and how they do it. Every week we get together at 11 AM Pacific on [Microsoft LearnTV](https://cda.ms/226) and learn more about Azure.

Data drives so many of our decisions.  Whether it's determining which products to have viewed first in our online retail store, or creating reports for business intelligence, we've got so much data!  It's time to figure out how to learn how to take that data and provide human-readable information that will help us continue to make the right decisions.  

This week on AzureFunBytes, I am joined by [Principal Program Manager, Mark Kromer](https://twitter.com/KromerBigData) about how to store and process our big data with Azure Data Factory.  Mark will discuss the [ETL (Extract, Transform, Load)](https://cda.ms/27y) process that gets our data into [Azure Data Factory](https://cda.ms/27w).  I ask Mark [how can we transfer the data we might have to Azure](https://cda.ms/27B)? We look into how to create pipelines to automate the ingestion of our data from various data stores.


{% youtube OuOE6OG0G8U %}

[00:04:35 - Intro to Mark](https://youtu.be/OuOE6OG0G8U?t=275)
[00:09:45 - Let's meet Data Factory](https://youtu.be/OuOE6OG0G8U?t=585)
[00:14:48 - CI/CD With Data Factory Pipelines](https://youtu.be/OuOE6OG0G8U?t=862)
[00:20:32 - Azure Data Factory connector overview](https://youtu.be/OuOE6OG0G8U?t=1232)
[00:31:57 - Demo Time](https://youtu.be/OuOE6OG0G8U?t=1917)


Our Agenda:

* Intro to Data Factory
* Differences between ADF & Synapse
* Data Flows in ADF & Synapse
* Data lake ETL patterns
* Build an ETL flow using taxi sample data (Demo)
* Q&A

From the Azure Documentation "[What is Azure Data Factory?](https://cda.ms/27r)"

![Overview of Data Factory](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9jbmy22v1lufk8rimbic.png)

[Azure Data Factory](https://cda.ms/27w) is the platform that solves such data scenarios. It is the *cloud-based ETL and data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale*. Using Azure Data Factory, you can create and schedule data-driven workflows (called pipelines) that can ingest data from disparate data stores. You can build complex ETL processes that transform data visually with data flows or by using compute services such as [Azure HDInsight Hadoop](https://cda.ms/27D), [Azure Databricks](https://cda.ms/27F), and [Azure SQL Database](https://cda.ms/27G).

Integrate all your data with Azure Data Factory—a fully managed, serverless data integration service. Visually integrate data sources with more than 90 built-in, maintenance-free connectors at no added cost. Easily construct ETL and ELT processes code-free in an intuitive environment or write your own code. Then deliver integrated data to Azure Synapse Analytics to unlock business insights. 


---

Learn about Azure fundamentals with me!

Live stream is available on Twitch, YouTube, and LearnTV at 11 AM PT / 2 PM ET Thursday. You can also find the recordings here as well:

[AzureFunBytes on Twitch](https://twitch.tv/azurefunbytes)
[AzureFunBytes on YouTube](https://aka.ms/jaygordononyoutube)
[Azure DevOps YouTube Channel](https://www.youtube.com/channel/UC-ikyViYMM69joIAv7dlMsA)
[Follow AzureFunBytes on Twitter](https://twitter.com/azurefunbytes)

Useful Docs:

[Get $200 in free Azure Credit](https://cda.ms/219)
[Microsoft Learn: Introduction to Azure fundamentals](https://cda.ms/243) 
[Microsoft Learn: Integrate data with Azure Data Factory or Azure Synapse Pipeline](https://cda.ms/27K)
[Microsoft Learn: Data integration at scale with Azure Data Factory or Azure Synapse Pipeline](https://cda.ms/27L)
[Azure Data Factory](https://cda.ms/27w) 
[Azure Data Factory documentation](https://cda.ms/27x) 
[Azure Data Factory Tutorials](https://cda.ms/27J)
[Extract, transform, and load (ETL)](https://cda.ms/27y)
[Transferring data to and from Azure](https://cda.ms/27C) 
[Big data architecture style](https://cda.ms/27H)
[Watch our snack-sized video tutorials here to learn more about building ETL with data flows](https://www.youtube.com/channel/UC2S0k7NeLcEm5_IhHUwpN0g/videos) 
[Follow the Delta Lake tutorial here to build your own lake](https://cda.ms/27s)
[Branching and chaining activities in an Azure Data Factory pipeline using the Azure portal](https://cda.ms/27t) 
[For access to the taxi medallion sample data to build these pipelines on your own, visit Mark's sample data repo here and look for trip data and trip fare](https://github.com/kromerm/adfdataflowdocs/tree/master/sampledata)