Azure Data Factory - Trigger on SFTP upload
Published: 2023-01-09 21:44:58


In this tutorial, we'll focus on configuring Storage Event Triggers with SFTP-enabled storage accounts. We'll walk through the basic steps of setting up a trigger, as well as the specific configuration changes you'll need to make to trigger an ADF pipeline in response to SFTP-related storage events.

---

## The Story

I was implementing a system where I had to trigger a pipeline in Azure Data Factory (ADF) whenever a file was uploaded on a specific location of a Storage Account, from an external stakeholder.

Well, this was pretty straightforward (I thought), as ADF offers Trigger solutions out of the box.

The problem was that, I was seeing the file being uploaded, but my Trigger was not being activated 🧐

I started the usual debug process. Opened the Microsoft Azure Storage Explorer Application, manually uploaded the file to the location, and the pipeline was triggered. What the hell? Everything is working...

Started going down the road of Google and Stackoverflow queries, but found nothing that suited my situation.

Some colleague discussions and a couple of hours later, we realised that we had given our external stakeholder SFTP credentials from which he could work in the directory. This had to be it.

Right after this conclusion, bam:

![ADF doesn't support SFTP triggers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i5j4znabnq0fqkw9m562.png)

ADF doesn't support trigger activation over SFTP...

## The Solution
Scheduling the trigger was not an option. So I had to find other ways of doing this.

One could go via Logic Apps route, believe me I tried. Wasted more time on this than I expected. **It did not work!**

Then I found a pretty cool drafted solution by [Amrinder Singh](https://techcommunity.microsoft.com/t5/azure-paas-blog/working-with-adf-storage-event-trigger-over-sftp/ba-p/3659652). But I was not able to understand the implementation as is. Thought this might be some legacy stuff. Read some more documentation, and the more I was reading the more I was understanding this previous implementation, but more guidance was originally required.

### Step 1: Create the ADF Trigger
Let's create a Trigger in ADF, by providing the details of our SFTP-enabled storage account, the container name, and the pattern (blob start/blob end) that we want to use for triggering conditions.


![ADF Simple Trigger](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s0tgxu800ag0dxjnw0rt.png)


When you perform this step, ADF is automatically creating under the hood, something called **Event Grid configurations**. This is the key information which we will use in the next steps

## Step 2: Edit Event Grid
Now the cool stuff begins. You need to search for the **Event Grid** Resource in the Azure Portal and proceed to System topics.

![Event Grid System Topics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qw2ugb28kr8sukbm3ygm.png)

Here you will find each automatically created **System Topic** by your Data Factories. Select the one corresponding to your desired ADF and resource group.


![Event Subscriptions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o08j14a6lyg5i3y7bjiu.png)

Under **Event Subscriptions** you will find every trigger you own. With more configuration possibilities than you ever had the chance to add by using only ADF's UI.

We will just focus in adding SFTP capabilities to it, but you can find more possible configurations in here: [Azure Blob Storage as an Event Grid source](https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?tabs=event-grid-event-schema#sftp-events)

Go to your desired trigger, and go to **Filter -> Advanced Filters**.


![Default Event filter](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eyr2azs38mqewzxqy7z5.png)

This is how the default Trigger Filters looks like. Note that SFTP related filters are missing.


![Modified SFTP Trigger](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8z2k0mus6ior2xn5afld.png)

To make things work, just add these values to it:
- `SftpCreate`
- `SftpCommit`

These will trigger the **BlobCreated Event**. More specifically, this event is triggered when clients use the `put` operation, which corresponds to the `SftpCreate` and `SftpCommit` APIs. An empty blob is created when the file is opened and the uploaded contents are committed when the file is closed.

**Save the changes** and your workflow is complete!

## Success 🎉

We've reached the end of our journey!


![SFTP](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/01xll1ftllzbs1ojnm5o.png)


Testing the file upload to our Storage container over SFTP.

![ADFT Trigger Success](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dow7imlw2c42u46ay2th.png)

We have a **Great Success** Tick mark ✅ 

Hope you found this useful for your current and future endeavour's! 

Dinis 💪