Subscribe to GCP Topic in Azure from C#
Tags: gcp, azure
Published: 2022-06-28 08:32:25

Today in this Blog I would like to share 

> 
_How to subscribe google cloud publisher subscriber (Pubsub) topics into Azure services (azure function) using Microsoft visual studio C#_

**Overview**
Pub/Sub is an asynchronous messaging service designed to be highly reliable and scalable. The service is built on a core Google infrastructure component that many Google products have relied upon for over a decade. Google products including Ads, Search and Gmail use this infrastructure to send over 500 million messages per second, totaling over 1TB/s of data. This article describes the salient design features that enables Pub/Sub to provide this type of scale reliably.

**The Basics of a Publish/Subscribe Service**
Pub/Sub is a publish/subscribe (Pub/Sub) service: a messaging service where the senders of messages are decoupled from the receivers of messages. There are several key concepts in a Pub/Sub service:

**Message**: the data that moves through the service.

**Topic**: a named entity that represents a feed of messages.

**Subscription**: a named entity that represents an interest in receiving messages on a particular topic.

**Publisher (also called a producer)**: creates messages and sends (publishes) them to the messaging service on a specified topic.

**Subscriber (also called a consumer)**: receives messages on a specified subscription.

The following diagram shows the basic flow of messages through Pub/Sub:

![PubSub Architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l8nzc7nfombj0oz2gk8d.png)
 

Publishers A and B push a message to a single topic, where two subscriptions listen for messages and push them to multiple subscribers.

In this scenario, there are two publishers publishing messages on a single topic. There are two subscriptions to the topic. The first subscription has two subscribers, meaning messages will be load-balanced across them, with each subscriber receiving a subset of the messages. The second subscription has one subscriber that will receive all of the messages. The bold letters represent messages. Message A comes from Publisher 1 and is sent to Subscriber 2 via Subscription 1, and to Subscriber 3 via Subscription 2. Message B comes from Publisher 2 and is sent to Subscriber 1 via Subscription 1 and to Subscriber 3 via Subscription 2

**Prerequisites** 
Now as we have some idea about the architecture of this google service, let's create a topic in google cloud.

first you need to have google cloud account where we can create a GCP topic and subscriber.

You can register free with your gmail account to create a google cloud account.

[Register here](https://cloud.google.com/apigee/docs/hybrid/v1.5/precog-gcpaccount)


Once you have google cloud account you can login to cloud console using this [URL](https://console.cloud.google.com/)

By default a google cloud project will be created, you create a new one using Project drop down

![Create new Google project](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4r1y6v9nykbxa47u7dru.png)

Once you are in your project you can select Pubsub service from the left menu bar

![Search Topic Service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xqwdwha1c3ndjvle4eik.png)
Or you can search on top search bar with keyword "Topic"

![Topic service board](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tcuq7k2b4v5ueghnwhrr.png)


Once you select the Pubsub service you will be navigated to Topic service Dashboard where you can find all the created topics and it's details. 

![Search Topic Service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uqsg0sgxauceron5oq67.png)

**Create Topic**
To create new topic click on **Create Topic** button

 ![Create Topic](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5wibifv4l92c0e2praw1.png)

![Create Topic](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/94ep7lvsputg78ir920k.png)

Once you create a topic a default subscription will be created.

![View Subscription](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/588rgelcwzdkdg5nzu8p.png)
 
Now let's publish a new message to this topic manually (in your actual application it might possible some process will publish a message)


click on message tab

![Publish message](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o8duw7xz7u0hajugb4da.png)
  
![Create msg](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/glrrg07pkq82w4ekod9y.png)

Now on the dashboard you can select subscription (default one in this case)  and view the messages

![view msg](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/71o98gdczz5zxjjsyqus.png)

**Next we requires to create a service account and role to access the Pubsub service**

**Create Service account**

To create a Service Account search IAM in the top search bar select **IAM**

![Search IAM](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/25gewf6fham4s6byfoas.png)

Select Service account option from the left menu, Click on 'Create new Service Account' button to create a new service account.

Enter the detail and click on **Create Account and Proceed**

![Create Account](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u7ax18vj19uwbp4x7y8c.png)

Select Role 
![Select Role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wt66fon4kokp084irjb7.png)

Select Pub/Sub Publisher and Pub/Sub Subscriber

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gjxzykea592h7kutn918.png)
 
Click on Continue and click 'Done' (Don't make any changes to _Grant users access to this service account_ option)

after this an account will be created now we need to create a Key for this service account

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hwsqzzb5peenwv30dx5h.png)

Select service account and click on **Key** tab

Click on 'Add Key' button to add new key for this service account

![Add key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/blmwow7nyu6o0yddzu0u.png)
 
This will save a private key on your work station

![Download key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhzu5cgz7dyq5zd04rjn.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/atxrs41r7a2h5475objn.png)

This is the key which you can use in your program

```Json

{
  "type": "service_account",
  "project_id": "xenon-raceway-268218",
  "private_key_id": "9620fb3dd81a33e2bbe0e8defd327858633459ef",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC2Brf0+VVIv2vs\nUVAUSPKiP57z/9NOp/7qJyVgJsnk7brqvpU7RiJpCF84wmp4Emlgxh3vbwZa2+BU\n9MtCcd3Xb4JVnA0dxSafRIRQE6Fljp8KtKk42VZg2mggwdHhCXO1wfCbWtd5yng6\nIGV0OxWlzQbATmk+HvHZoOVZ1bWv22Baf2rF5w3msj9i4Jh5wU8fAQHX9+oWEVJRGCEZslB\nJ+vaWVVYluxfDpmv7YCZ3gs=\n-----END PRIVATE KEY-----\n",
  "client_email": "spn-pub-sub@xenon-raceway-268218.iam.gserviceaccount.com",
  "client_id": "116457436621716168449",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/spn-pub-sub%40xenon-raceway-268218.iam.gserviceaccount.com"
}

```
 
 This json is the one which will be used to in the program to access to service data.

You can find more details about the environment variables here

```
https://cloud.google.com/docs/authentication/getting-started
```

Using above mentioned steps you can complete the setup of your Infra structure for PubSub topic

Lets create a new Console application or azure function application with C# as language

The code requires to use google cloud C# SDK using the NuGet packages

![NuGet package](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/63ufbf5pdyt3q20n8b6l.png)
 
![Package added](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q9lc3h9ryqx3dm3ls6aq.png)

Now you can write azure function code to write following code to read the message from service topic from google. 


```C#
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", Path.Combine(Environment.CurrentDirectory, "credentials.json"));
    string projectId = "xenon-raceway-268218";
    string topicId = "order-topic";
    string subscriptionId = "order-topic-sub";
    ProjectName projectName = ProjectName.FromProject(projectId);

    var subscriptionName = new SubscriptionName(projectId, subscriptionId);
    while (true)
    {
        var subscriber = await SubscriberClient.CreateAsync(subscriptionName);
        try
        {
            await subscriber.StartAsync((msg, ct) =>
            {
                Console.WriteLine(msg.Data.ToStringUtf8());
                return Task.FromResult(SubscriberClient.Reply.Ack);
            });
        }
        catch (Exception e)
        {
            Console.WriteLine($"Exception: {e}");
        }
    }
```

This is how the output will looks like

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/67rr24wc6la5ca68ertp.png)
 
**References** 

https://cloud.google.com/pubsub/docs/reference/libraries#client-libraries-usage-csharp

https://cloud.google.com/dotnet/docs/reference/Google.Cloud.PubSub.V1/latest/index


  
I hope you have found this useful.