Trigger Azure Data Factory Pipeline from Event Grid (Using Webhook Endpoint)
Tags: azure, eventgrid, dataengineering, learnandshare
Published: 2023-04-12 09:25:30

---
title: Trigger Azure Data Factory Pipeline from Event Grid (Using Webhook Endpoint)
published: true
description: 
tags: Azure, EventGrid, DataEngineering, learnandshare
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2023-04-10 08:57 +0000

---

In this post, I will explain how we can use Azure Event Grid Topics to trigger Data Factory pipeline. We will be using Access key for authentication and POST request for publishing a topic to Event Grid. I have also attached a snapshot for each step which can help to understand the steps easily.

> **Prerequisite:**
Microsoft.EventGrid should be registered under the subscription.
To confirm this, go to your subscription, then click Resource Provider.
Search for Event Grid, if `Microsoft.EventGrid` status is Registered, we are good to go else you need to select `Microsoft.EventGrid` and click on the register button.

![Register EventGrid to Subscription](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r9yhb831s9s95vhv7h7o.png)


We will create 2 Azure resources: Event Grid Topic and Azure Data Factory.


####Create Event Grid Topic Resource####
Here, I created a resource with name `eg-tp-useast-dev-01`

![Create EventGrid Topic Resource](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/erj5l34idfq8blij5yle.png)

While creating the resource, make sure that `Enable access key and Shared Access Signature (SAS)` is enabled.
Remaining configurations you can keep as default.

![Enable access key and Shared Access Signature](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/110w1soxvs6ivsjgeau5.png)

---

Once Event Grid Topic Resource is deployed, go to the resource. Under the Overview tab, you can see there is no subscription found at this moment. Once we configure the ADF trigger, we will see a subscription here.

![No Subscription](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k2gv3q6j46ffnv58jw5f.png)

---

####Create Azure Data Factory resource and launch the Studio####
I created a Data Factory resource `df-useast-dev-01`.
![Create Azure Data Factory resource](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d5iyc8ddf1qci5savr7m.png)
Go to Author Tab and create a sample pipeline. For simplicity, I just added a Wait activity inside a sample pipeline `PL_Sample`.

![sample pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d4815dlwnmvm23s7t20m.png)
Once you created the pipeline, Publish it.

---

####Create a custom trigger####
Create a custom trigger for the pipeline `PL_Sample` which we just created. Click on Add Trigger and create a new trigger with the below details:
- give trigger name 
- select type as Custom events
- select Subscription
- you should see all available event grid topics, select the topic which we just created (eg-tp-useast-dev-01)
- type the subject: tp-subject-01
- type the event type: event-type-01
- check `Start trigger on creation`
- click OK
- publish Trigger

> subject and event type values are used to filter the events.

![create a custom trigger](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/43ykm580rej7cp75bkg9.png)

---

> **NOTE:** If Event Grid topic is not visible in the `Event Grid topic name` drop-down, follow these 4 steps:
1. Open Event Grid topic resource in a new tab, Go to the overview tab, click on JSON view.
![Resource ID](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tkyezaubyeg99i722m2u.png)
2. Copy the Resource ID.
![Resource ID](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/owka3v0xouxwr5vy9e3n.png)
3. Again, switch back to the ADF trigger page and switch to Enter manually for `Account selection method` and 
paste the Resource ID in the scope field.
4. Click on Ok and Publish the trigger.

Once Trigger is published successfully, go to Event Grid topic, you should see 1 event subscription.

![Event subscription](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w4yeo1oyo5laarnodppz.png)

---

So far we have configured a ADF pipeline with the Event Grid topic.

####Publish an event to the Event Grid Topic####
Now we will publish an event to the Event Grid Topic by sending a POST request. I am using Postman to send requests.

In order to send a request, we need an endpoint and an access key of the topic.

- Go to Overview tab and copy the `Topic Endpoint` value
- Go to Access Keys and copy a Key

![Event Grid Topic access key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/az4z2mwnfy5n9ci2wfnc.png)

---

####Create A POST request####
`URL`: Use the Topic Endpoint as request URL
`header`: Add a header in the request with key `aeg-sas-key` and access key as value.
`body`: Add the below body in JSON format
```
[
  {
    "id": "I01",    
    "eventType": "event-type-01",
    "subject": "tp-subject-01",
    "data":{
      "key1": "val1",
      "key2": "val2"
    },
    "eventTime": "2023-04-06T11:26:07+05:30",
    "dataVersion": "1.0"
  }
]
```

>
`id`: id of the event (must be unique for each request)
`eventType`: same as eventType mentioned while creating ADF trigger
`subject`: same as subject mentioned while creating ADF trigger
`data`: parameters that can be passed to ADF trigger (explained at the end)

![Create A POST request](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/op60t9u4ynhrn779k9pc.png)

---

Click on Send, If the status is `200 OK`, ADF pipeline should trigger.
Go to `ADF monitor tab > Pipeline Runs > Triggered`.
You should see the Pipeline has been triggered by the custom event trigger 

![ADF monitor tab](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dha5ihxlm9rs13kzgcad.png)

---


As I mentioned earlier that we can pass the parameter from event topic to ADF pipeline. For that follow below steps:

- Create a pipeline parameter (here I created `par_input_file`)
- Edit the trigger and pass `@triggerBody().event.data.key1` in Trigger Run Parameters.

When pipeline gets triggered, it should initialize the pipeline parameter with `event.data.key1` value.

![pipeline paramete](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/voqgjv5jtk6fvu28jfpy.png)

---

>**Note: Resources should be deleted after testing in order to avoid costs.**

Thank you for reading this post. Hope it helped you understand and implement this event-driven pipeline execution. I could not find all these steps in a single post, so thought to consolidate all the steps into one post and share them with other developers. Please let me know if you stuck in between while implementing this, I would be happy to help. Happy learning.

###