WebSub to EventGrid via CloudEvents, and Beyond
Tags: azure, eventgrid, logicapps, azurefunctions
Published: 2021-01-29 00:00:20

---
title: WebSub to EventGrid via CloudEvents, and Beyond
published: true
description: In this post, I'm going to discuss how to implement social media marketing tool for YouTube videos to be amplified to other social media channels like Twitter, LinkedIn and Facebook, by transferring YouTube WebSub event to Azure EventGrid through CloudEvents, Azure Logic Apps and Azure Functions.
tags: azure, azureeventgrid, azurelogicapps, azurefunctions
canonical_url: https://devkimchi.com/2021/01/27/websub-to-eventgrid-via-cloudevents-and-beyond/
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/t7cxhqqyefao8lfhact7.png
series: 

---
Let's say you've got a YouTube channel uploading videos in a regular cadence. When a new video is published, you want to cross-post it to other social media channels that you're running. What could be the best way to do so? There are hundreds of commercial tools on the market for online content marketing. There are hundreds of companies to help you, and those companies have their proprietary solutions for it. It could make sense to utilise those tools or companies. However, with various reasons or circumstances, what if you need to build your own one? What if the existing tools don't fulfil your requirements? Then, it's a good time to build the solution by yourself, isn't it?

This post will discuss an end-to-end workflow story, from YouTube video update to other social media exposure. For the workflow, let's use Azure serverless services like [Azure EventGrid][az evtgrd], [Azure Functions][az fncapp] and [Azure Logic Apps][az logapp].

> If you like to see the source codes of the solution, find this GitHub repository. It's completely open-source, and you can use it under your discretion with care.
> 
> {% github https://github.com/devrel-kr/youtube-websub-subscription-handler %}


## Subscribing YouTube Notification Feed ##

YouTube uses a protocol called [PubSubHubbub][pshb] for its notification mechanism. It's now become a web standard called [WebSub][websub] since 2018, after the [first working draft in 2016][websub wd 1].

Google registers all YouTube channels to their [WebSub Hub][yt websub hub]. Therefore, if you want to get the update notification from a specific channel, you can simply send a [subscription request][yt websub sub] to the Hub. To subscribe, enter the message handler URL and YouTUbe channel URL and click the `Do It!` button. Too easy!

![Subscribing YouTube Channel via WebSub][image-01]

Please ensure that the subscription process is completed only after passing the [message handler verification request][websub verification].


## Verifying WebSub Subscription Request ##

To verify the WebSub subscription request, the WebSub Hub sends an API call to the message handler. When it arrives, the handler MUST deal with the following.

* The verification request sends a `GET` request with the following query parameters:
  * `hub.mode`: The `subscribe` string.
  * `hub.topic`: The YouTube channel URL.
  * `hub.challenge`: A random string generated by the WebSub Hub, used for the verification.
  * `hub.lease_seconds`: The validation period in seconds from the time of the request. The request will be void unless the request is not verified within this period.
* The response of the verification request MUST include the `hub.challenge` value to the response body, with the HTTP status code of 200:
  * If the response body includes anything other than the `hub.challenge` value, the WebSub Hub won't accept it as the valid response.

Here's the verification request handling logic in the [Azure Function][az fncapp] method:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=01-callback-1.cs %}

Once the message handler is verified, the WebSub Hub keeps sending the notification message to the handler whenever a new video update is made, from the subscribed channel.


## Converting WebSub Notification Feed ##

As the mechanism of WebSub follows the same [Publisher/Subscriber (Pub/Sub) pattern][eip pubsub], it's not that new. The only difference of WebSub is the event data that makes use of the existing ATOM feed format. Therefore, as long as any subscriber understands the feed format, it should be OK. In other words, the subscriber has a strong dependency on the event data format the publisher sends. In the modern application environments, we recommend decoupling between the publisher and subscriber as much as we can, so that each can organically grow independently. In other words, the subscriber don't have to know the ATOM feed format. How can we make them decoupled, then? The event data format or message format needs to be canonicalised. Then, converting the canonical data into the subscriber-specific format should be done by the subscriber's end.

Therefore, we are going to use [CloudEvents][ce] as the canonical data format. There are two steps for the conversion&ndash;1) canonicalisation and 2) domain-specific conversion. Let's have a look.


### 1. Canonicalisation: WebSub Feed ➡️ CloudEvents ##

The purpose of this step is to decouple between WebSub Hub and your application. The XML data delivered from the WebSub Hub is just wrapped with the CloudEvents format. When a new video is updated onto YouTube, it sends a notification to the WebSub Hub, which looks like the following:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=02-websub-feed.xml %}

As the message handler takes this request through `POST`, it is stringified like this:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=03-callback-2.cs %}

The request header also contains the following `Link` information:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=04-websub-request-header.txt %}

As it includes the YouTube channel URL as the message source, you need to extract it.

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=05-callback-3.cs %}

Then, set the event type and content type like the following.

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=06-callback-4.cs %}

As I mentioned in my [previous post][post prev 1], at the time of this writing, the [Azure EventGrid Binding][az fncapp binding evtgrd] for [Azure Function][az fncapp] currently has [a limitation to support the CloudEvents format][az evtgrd ce]. Therefore, you should handle it manually:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=07-callback-5.cs %}

So far, the WebSub data is canonicalised with the CloudEvents format and sent to EventGrid. The canonicalised information looks like this:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=08-cloudevents-from-websub.json %}

Now, you have cut the dependency on the WebSub Hub.

![YouTube WebSub to Azure EventGrid][image-02]

Let's move onto the next step.


### 2. Domain-Specific Conversion: WebSub XML Data Manipulation ##

At this step, the XML data is actually converted into the format we're going to use for social media amplification.

The WebSub XML data only contains bare minimum information like the video ID and channel ID. Therefore, you need to call a YouTube API to get more details for social media amplification. An event handler should be registered to handle the published event data on [Azure EventGrid][az evtgrd]. Like the WebSub subscription process, it also requires [delivery authentication][az evtgrd delivery auth]. One of the good things using [Azure Logic Apps][az logapp] as the event handler is that it automatically does all the verification process internally. Therefore, you just use the Logic App to handle the event data.

The Logic App handler's first action is to verify whether the event data is what you are looking for&ndash;it should meet your channel ID and event type of `com.youtube.video.published`. If either channel ID or the event type is different, this handler stops processing.

![Verifying Event Data][image-03]

If the event data is what you are looking for, the handler passes it to Azure Functions for further manipulation.

![Manipulating Event Data][image-04]

The Azure Functions app calls the YouTube API to get more details of the video, manipulates them, and turns it back to Logic App. The converted data looks like:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=09-video-details.json %}

YouTube data has been massaged for our purpose.

![Diagram: Event Data Conversion][image-05]


## Social Media Exposure ##

The event handler now needs to help spread the YouTube video update to the world through designated social media. There are two approaches:

* The event handler directly connects to APIs of individual social media, or
* The event handler publishes another event to Azure EventGrid for other event handlers takes care of social media amplification.

Although both approaches are valid, the first one has strong coupling between the handler and amplifiers. If you need to add a new amplifier or remove an existing one, the Logic App handler must be updated, which is less desirable, from the maintenance perspective. On the other hand, The second approach publishes another event containing the converted data. All social media amplifiers act as event handlers, and they are all decoupled. I chose the second one.


### 1. Event Publish: Converted YouTube Video Details ###

In order to publish the converted YouTube video details to Azure EventGrid, the data needs to be wrapped with the CloudEvents format. The screenshot shows the action on how to wrap the video details data with CloudEvents. This time, the event type will be `com.youtube.video.converted`.

![Converting Video Details to CloudEvents][image-06]

The next action is to send an HTTP request to Azure EventGrid, with the CloudEvents payload. You can notice that many metadata headers are starting with `ce-`, defined in the [cross reference check spec over HTTP][ce binding http].

![Sending Data to EventGrid][image-07]

The message handler now completes its workflow. From now on, each social media handler takes care of the new event data.

![Diagram: Sending Data to EventGrid][image-08]


### 2. Event Handlers: Social Media Amplification ###

YouTube video details are now ready for amplification! Each social media handler takes care of the event data by adapting their circumstances. The event data received from EventGrid looks like this:

{% gist https://gist.github.com/justinyoo/cb04844306f9a44dcdcaaa70a6a55326 file=10-video-details-to-cloudevents.json %}


#### Twitter Amplification ####

As Logic Apps provides the [Twitter connector][az logapp connector twitter] out-of-the-box, you don't need to use the API by yourselves. Therefore, simply use the actions like below:

![Posting to Twitter][image-09]


#### LinkedIn ####

Logic Apps also provides a built-in [LinkedIn connector][az logapp connector linkedin]. So, simply you use it.

![Posting to LinkedIn][image-10]


#### Facebook ####

Unlike the other two connectors, the [Facebook connector][az logapp connector facebook] has been deprecated. Instead, it's now become an [open-source project][az logapp connector facebook custom]. So, you should use this open-sourced custom connector or something else. Fortunately [IFTTT][ifttt] provides the [Facebook Page connector][ifttt facebook page], so you just use it.

![IFTTT Facebook Connecto][image-11]

From the Logic App point of view, calling IFTTT is just another HTTP call. So it's not that tricky. The only thing to remember is that the request payload can only include no more than `value`, `value2` and `value3`.

![Posting Facebook][image-12]

The actual process result in the IFTTT end looks like this:

![Posting Facebook on IFTTT][image-14]

We've amplified to social media of Twitter, LinkedIn and Facebook.

![End-to-end Event Processing Workflow][image-13]

If you want to add another social media, you can simply add another Logic App as the event handler.

---

So far, we've implemented a workflow solution that posts to designated social media platform when a new YouTube video update is notified through WebSub, by using [CloudEvents][ce], [Azure EventGrid][az evtgrd], [Azure Functions][az fncapp] and [Azure Logic Apps][az logapp]. As steps are all decoupled, we don't need to worry about the dependencies during the maintenance phase. In addition to that, although a new social media platform is planned to add, it wouldn't impact on the existing solution architecture.

If you or your organisation is planning online content marketing, it's worth building this sort of system by yourself. And it would be an excellent opportunity to make a well-decoupled and event-driven cloud solution architecture.


[image-01]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-01.png
[image-02]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-02-en.png
[image-03]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-03.png
[image-04]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-04.png
[image-05]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-05-en.png
[image-06]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-06.png
[image-07]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-07.png
[image-08]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-08-en.png
[image-09]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-09.png
[image-10]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-10.png
[image-11]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-11.png
[image-12]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-12.png
[image-13]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-13-en.png
[image-14]: https://sa0blogs.blob.core.windows.net/devkimchi/2021/01/websub-to-eventgrid-via-cloudevents-and-beyond-14.png


[post prev 1]: /azure/processing-cloudevents-for-azure-eventgrid-via-azure-functions-186l

[gh sample]: https://github.com/devrel-kr/youtube-websub-subscription-handler

[pshb]: https://github.com/pubsubhubbub/PubSubHubbub
[websub]: https://www.w3.org/TR/websub/
[websub wd 1]: https://www.w3.org/TR/2016/WD-pubsub-20161020/
[websub verification]: https://indieweb.org/How_to_publish_and_consume_WebSub#The_hub_verifies_the_subscription_request

[eip pubsub]: https://www.enterpriseintegrationpatterns.com/patterns/messaging/PublishSubscribeChannel.html

[yt webhook]: https://developers.google.com/youtube/v3/guides/push_notifications
[yt websub hub]: https://pubsubhubbub.appspot.com/
[yt websub sub]: https://pubsubhubbub.appspot.com/subscribe

[ce]: https://cloudevents.io/
[ce binding http]: https://github.com/cloudevents/spec/blob/master/http-protocol-binding.md#3-http-message-mapping

[az evtgrd]: https://docs.microsoft.com/azure/event-grid/overview?WT.mc_id=devops-dotnet-12869-juyoo
[az evtgrd ce]: https://docs.microsoft.com/azure/event-grid/cloudevents-schema?WT.mc_id=devops-dotnet-12869-juyoo#use-with-azure-functions
[az evtgrd delivery auth]: https://docs.microsoft.com/azure/event-grid/security-authentication?WT.mc_id=devops-dotnet-12869-juyoo

[az logapp]: https://docs.microsoft.com/azure/logic-apps/logic-apps-overview?WT.mc_id=dotnet-12869-juyoo
[az logapp connector twitter]: https://docs.microsoft.com/connectors/twitter/?WT.mc_id=dotnet-12869-juyoo
[az logapp connector linkedin]: https://docs.microsoft.com/connectors/linkedinv2/?WT.mc_id=dotnet-12869-juyoo
[az logapp connector facebook]: https://docs.microsoft.com/connectors/facebook/?WT.mc_id=dotnet-12869-juyoo
[az logapp connector facebook custom]: https://github.com/microsoft/PowerPlatformConnectors/tree/master/custom-connectors/Facebook

[az fncapp]: https://docs.microsoft.com/azure/azure-functions/functions-overview?WT.mc_id=dotnet-12869-juyoo
[az fncapp binding evtgrd]: https://docs.microsoft.com/azure/azure-functions/functions-bindings-event-grid?WT.mc_id=dotnet-12869-juyoo

[ifttt]: https://ifttt.com/
[ifttt facebook page]: https://ifttt.com/facebook_pages