How to sync user accounts with G Suite push notifications
Tags: gsuite, sysadmin, django, python
Published: 2020-04-08 16:23:31

If your organization uses G Suite to manage your users, you can synchronize those users’ accounts with other systems. This is a little different to just using Google Sign-in with your own custom app. Let me explain why.

We were working with the team at [sennder](https://www.sennder.com/), one of our portfolio companies, to synchronize a backend system with G Suite. Their system has a private codename, so I’m just gonna call it “Reinhardt” because it is based on Django (a web framework named after [Django Reinhardt](https://en.wikipedia.org/wiki/Django_Reinhardt)).

For our project, it wasn’t enough to have people sign into Reinhardt with their Google accounts. Reinhardt is a huge system with many roles and permissions levels. The goal was to centrally manage user accounts and permissions from within the G Suite and automatically create matching user accounts within Reinhardt.

To help you understand how to do this yourself, I’m going to break down our process with code examples.

We used the push notification feature of Google’s Directory API to perform the following major tasks:

*   **Have Google notify Reinhardt when someone creates a user in G Suite**:  
    Whenever someone creates a new user in the G Suite, we have Google automatically notify Reinhardt by sending a request to a webhook address like this one:  
    _https://reinhardt.sennder.com/gsuite/user-webhook_
*   **Create the user in Reinhardt based on the data that G Suite provides us:**  
    The webhook process reads the request, extracts the primary email, then uses the primary email as a key to retrieve the full user details from G Suite. Once we have the full user details, we create the user in Reinhardt’s database.

If you want to try this yourself, make sure that you’ve read my companion article “ [Beginning a G Suite integration — what to prepare before you start coding](https://medium.com/@merlin.pav/151a7361cf27?source=friends_link&sk=3f9cc981888f87b2f1a4f7c429f56e17)”. It gives you an overview of all the prerequisites such as service accounts and permission scopes.

Once you have that out of the way, you’re ready to start.
&nbsp;
## Installing the Google APIs Python Client

To interact with Google APIs, it’s easiest to use one of their [client libraries](https://developers.google.com/api-client-library). In our case, we used the Python client library for Google APIs so we’ll be using Python examples from our implementation.

You can install the Google APIs Client by entering the following commands in a bash shell.
```bash
pip install virtualenv  
virtualenv   
source /bin/activate  
/bin/pip install google-api-python-client
```
&nbsp;
## Installing the Django and the Django REST Framework

If you want to follow along and try out the examples, you’ll need to install Django and the accompanying REST framework which we use for our webhook receiver.

You can install Django with the following command:

`python -m pip install Django`

And the REST framework like so:

`pip install djangorestframework`

You’ll also need to add the REST framework to your Django _INSTALLED_APPS_ configuration like this:

```python
INSTALLED_APPS = [  
    ...  
    'rest_framework',  
]
```

If you need more help with the Django related stuff, check out the documentation on [configuring Django apps](https://docs.djangoproject.com/en/3.0/ref/applications/) and the [Django REST framework](https://www.django-rest-framework.org/#installation).
&nbsp;
## Creating a G Suite client wrapper

In our case, we created a wrapper around the standard Google APIs client library. This gave us a convenience class that was more tailored to working with the G Suite APIs. We called it the _GSuiteClient_, and here’s what it looks like:

```python
from typing import List
from google.oauth2 import service_account
from googleapiclient.discovery import build


class GSuiteClient:
    def __init__(
        self, permission_file_path: str, permission_scopes: List[str], admin_user: str
    ):
        self._permission_file_path = permission_file_path
        self._permission_scopes = permission_scopes
        self._admin_user = admin_user
        self._user_service = None
        self._credentials = None

    @property
    def user_service(self):
        if self._user_service is None:
            self._user_service = build(
                "admin",
                "directory_v1",
                credentials=self.credentials,
                cache_discovery=False,
            )
        return self._user_service.users()

    @property
    def credentials(self):
        if self._credentials is None:
            credentials = service_account.Credentials.from_service_account_file(
                filename=self._permission_file_path, scopes=self._permission_scopes
            )
            self._credentials = credentials.with_subject(self._admin_user)
        return self._credentials
```

⚠️**Beware of the API version parameter**

*   The **API version** parameter required for the Directory API endpoint is not what we expected it to be.

*   According to Google’s [official documentation](https://github.com/googleapis/google-api-python-client/blob/master/docs/start.md#building-and-calling-a-service), you’re supposed to initialize an API with the following syntax:
`service = build(‘api_name’, ‘api_version’, …)`

*   Like us, you might assume the directory API is initialized like so:
`service = build(‘Directory’, ‘**v1**’, …)`

*   But actually, the API name “Directory” is appended to the version parameter like this:
`service = build(‘admin’, ‘**Directory**_v1’, …)`

*   It took us a while to figure this one out. If you look at the reference documentation, a POST request to the directory API looks like this:
`POST https://www.googleapis.com/admin/directory/v1/users/watch?customer=my_customer&event=add`
We didn’t anticipate that we’d need to concatenate the API name with the version number.
&nbsp;
# Initializing the G Suite client settings

Our G Suite client wrapper expects three main settings:

*   A path to the **credentials** that you’re supposed to download when you created a service account.  
    Again, this step is covered in a [companion article](https://medium.com/@merlin.pav/151a7361cf27?source=friends_link&sk=3f9cc981888f87b2f1a4f7c429f56e17).
*   The **permissions scopes** that your integration needs (also covered in the same article).
*   The **primary email** of a G Suite user who has administrative privileges for your G Suite instance.  
    The integration must always be associated with a “real “user.

We give it these settings when we instantiate our _GSuiteClient_ like this:

```python
# initlialize our GSuite client wrapper with the required settings
gsuite_client = GSuiteClient(
    permission_file_path="C:/gs_example/gsuite-integration-211111-627e77777b0a.json",
    permission_scopes=["https://www.googleapis.com/auth/admin.directory.user"],
    admin_user="hugobert@sennder.com",
)
```
&nbsp;
## Creating a notification channel to watch for changes

As Google’s documentation acknowledges, the word “channel” is a bit ambiguous and has a different meaning across Google’s product portfolio. Here is how they define the concept of “channel” in the content of push notifications.

> _A channel specifies routing information for notification messages. As part of the channel setup, you identify the specific URL where you want to receive notifications. Whenever a channel’s resource changes, the Directory API sends a notification message as a POST request to that URL._

### Creating a Channel Generator Class

To create a channel, we first wrote a class that could be instantiated later on. It calls the [Users: watch](https://developers.google.com/admin-sdk/directory/v1/reference/users/watch) method based on the parameters that you provide it.

Here’s what the class looks like:

```python
from uuid import uuid4
from gs_example.gsuite_part.gsuite_client import GSuiteClient

class GSuiteChannelGenerator:
    def __init__(self, client: GSuiteClient):
        self._client = client

    def create_channel(
        self, url: str, domain: str, event: str, time_to_live: int
    ) -> None:
        param = {
            "domain": domain,
            "event": event,
            "body": {
                "type": "web_hook",
                "id": str(uuid4()),
                "address": url,
                "params": {"ttl": time_to_live},
            },
        }

        self._client.user_service.watch(**param).execute()
```
&nbsp;
## Calling the Channel Generator

Our logic to call the channel generator was somewhat complex so we’ve simplified it for this article. The complex part involved keeping the channel alive, but I’ll get into that in a moment.

In this example, we’re creating the channels when we start the Django server. We’ve updated Django's standard [URL configuration file](https://docs.djangoproject.com/en/3.0/topics/http/urls/) that initializes when the server starts.

---
**NOTE**

_Don’t do this in production_, because it will create channels every time you start the server. We’ve done it to simplify the example. The Google API sends a sync message as soon as you create a channel, and if it gets a 404 (i.e. your server isn’t yet running), it will not send notifications. That’s why we create the channels during the server initialization.

---

Here’s what our URL configuration looks like:

```python
from django.contrib import admin
from django.urls import path

from gs_example.gsuite_part.channel_generator import GSuiteChannelGenerator
from gs_example.gsuite_part.gsuite_client import GSuiteClient
from gs_example.gsuite_part.web_hook import WebHookReceiver

urlpatterns = [
    path("admin/", admin.site.urls),
    path("gsuite/user-webhook", WebHookReceiver.as_view()),
]

# initlialize our GSuite client wrapper with the required settings
gsuite_client = GSuiteClient(
    permission_file_path="C:/gs_example/gsuite-integration-268011-627e32157b0a.json",
    permission_scopes=["https://www.googleapis.com/auth/admin.directory.user"],
    admin_user="hugobert@sennder.com",
)

channel_generator = GSuiteChannelGenerator(client=gsuite_client)
channel_generator.create_channel(
    url="https://reinhardt.sennder.com/gsuite/user-webhook",
    domain="sennder.com",
    event="add",  # for this case we watch user creation
    time_to_live=60 * 60 * 8,
)
channel_generator.create_channel(
    url="https://reinhardt.sennder.com/gsuite/user-webhook",
    domain="sennder.com",
    event="update",  # for this case we watch user edit
    time_to_live=60 * 60 * 8,
```
*   The variable _urlpatterns_ defines the path that triggers our webhook receiver (again, we’ll get to the webhook receiver a bit later).
*   The variable _gsuite_client_ instantiates our _GSuiteClient_ with our required settings.
*   The variable _channel_generator_ instantiates the _GSuiteChannelGenerator_ with our preconfigured _GSuiteClient._
*   We then call the _create_channel_ function twice, so that we get two channels:  
    One channel to send notifications about _new_ users, and another to send notifications about _updated_ users. After you’ve successfully created channels, comment out these two _channel_generator.create_channel_ calls. You won’t need them until the channels expire.
*   The _time_to_live_ argument sets each channel to expire after 8 hours.

After each channel is created, you should get a response that resembles the following example:

```python
{
  "kind": "api#channel",
  "id": "01234567-89ab-xxx-012BAX6789ab", # ID generated for the channel.
  "resourceId": "B4ibMJiIhTx123xf2K2bexk8G4", # ID of the watched resource.
  "resourceUri": "https://www.googleapis.com/admin/directory/v1/users?domain=sennder.com&event=add", # Version-specific ID of the watched resource.
  "token": "target=GSuiteCreateUser", # Present only if one was provided.
  "expiration": 1384823632000, # Actual expiration time as Unix timestamp (in ms), if applicable.
}
```

After Google has created the channel, it will send your webhook receiver a [sync message](https://developers.google.com/admin-sdk/directory/v1/guides/push#sync-message) to indicate that it’s ready to send notifications. This message has includes the header parameter `X-Goog-Resource-State: sync.`

In the [companion article,](https://medium.com/@merlin.pav/151a7361cf27?source=friends_link&sk=3f9cc981888f87b2f1a4f7c429f56e17) I talked about how to set up a test server with the help of [ngrok](https://ngrok.com/). Here’s how a sync message looks in the ngrok dashboard.

![NGROK screenshot](https://dev-to-uploads.s3.amazonaws.com/i/8et1y0ahgnpeuk480kly.png)
&nbsp;
## How long can a channel live?

We couldn’t really find a definitive answer to this question in Google’s documentation on the Directory API, but we compared it with the [documentation for the Drive API](https://developers.google.com/drive/api/v3/push#making-watch-requests) and found this note.

> _For the Drive API, the maximum expiration time is 86400 seconds (1 day) after the current time for File resources and 604800 seconds (1 week) for Changes. If you don’t set the expiration property in your request, the expiration time defaults to 3600 seconds after the current time._

Since Google Drive is part of G Suite, we assumed that channels for watching user accounts would also expire after a week. But we didn’t want our channels to last that long, because…

⚠️**Currently, you can’t stop a notification channel**

*   This is a bug. Suppose that you want to temporarily stop notifications (eg: you want to fix a bug in the callback receiver). Google provides the [stop](https://developers.google.com/admin-sdk/directory/v1/reference/channels/stop) endpoint for doing just that.
*   However, we found this [Stack Overflow issue](https://stackoverflow.com/questions/40392097/trying-to-stop-google-directory-api-push-notifications-with-a-client-returns-404) where a user reported that they get a 404 when they make a stop request in the client or call the stop endpoint directly (there is also general confusion on the correct address of the stop endpoint).  
    There are multiple reports of this issue in various client support forums, and at the time of writing, it doesn’t appear to be fixed. For that reason, we didn’t want a channel to live too long, in case we needed to stop notifications.
*   Judging by [this bug report](https://github.com/googleapis/google-api-python-client/issues/403) for the G Suite Admin SDK, it looks like this issue won't be fixed any time soon.

To keep a channel alive, we had a process that ran every hour and checked to see if the channel had expired. If it was due to expire within 30mins, we created a new channel. Whenever we created a channel, we [stored its metadata in a Redis cache](https://realpython.com/caching-in-django-with-redis/) (we didn’t use Memcached because we needed the cache to say alive after a process ended).

Our scheduled process could then check the channel expiry date without making another request to Google.
&nbsp;
## Acting on a notification

When someone created a user in G Suite, Google sent a request to our webhook receiver with the following headers and body.

```
POST https://reinhardt.sennder.com/gsuite/user-webhook # Your receiving URL.
Content-Type: application/json; utf-8
Content-Length: 189
X-Goog-Channel-ID: addChannel
X-Goog-Channel-Token: 245tzzzzzzzzzz333
X-Goog-Channel-Expiration: Mon, 09 Dec 2013 22:24:23 GMT
X-Goog-Resource-ID: B4ibMJzzzzzzzzzzbexk8G4
X-Goog-Resource-URI: https://www.googleapis.com/admin/directory/v1/users?domain=sennder.com&event=add&alt=json
X-Goog-Resource-State: delete
X-Goog-Message-Number: 226440
  
{
 "kind": "admin#directory#user",
 "id": "111225555555555555558702",
 "etag": "\"Mf8RAmnAzzzzzzzzzzzzzzdRE/evLIDzzzzzzzzzzzz7Pzq8UAw\"",
 "primaryEmail": "hugobert@sennder.com"
}
```
&nbsp;
We then used this information to recreate the user in the other system. Here’s the code of the webhook receiver that performs this task.

```python
from django.contrib.auth.models import User
from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView

from .gsuite_client import GSuiteClient
from .gsuite_user_fetcher import GSuiteUserFetcher

# initlialize our GSuite client wrapper with the required settings
gsuite_client = GSuiteClient(
    permission_file_path="C:/gs_example/gsuite-integration-211111-627e77777b0a.json",
    permission_scopes=["https://www.googleapis.com/auth/admin.directory.user"],
    admin_user="hugobert@sennder.com",
)

gsuite_client_fetcher = GSuiteUserFetcher(user_service=gsuite_client.user_service)

class WebHookReceiver(APIView):
   def post(self, request: Request) -> Response:
       data = request.data
       if "primaryEmail" in data:
           user_email = data["primaryEmail"]
           user = gsuite_client_fetcher.fetch_user_by_email(user_email=user_email)
           try:
               # Check if the user already exists in DB
               u = User.objects.get(username=user.email)
               print('User exists, updating user: ', user)
               # Refresh all user data:
               # We're just refreshing the first/last names here
               # but you can add more fields where necessary..
               u.first_name = user.first_name
               u.last_name = user.last_name
               u.save()
           except User.DoesNotExist:
               print('User did not exist previously')
               User.objects.create_user(username=user.email,
                                        first_name=user.first_name,
                                        last_name=user.last_name,
                                        email=user.email,
                                        password='default_password')
       else:
           print("We got a Sync from Google!!")
```
*   First, we instantiate the _GSuiteUserFetcher_ class which we’ll use to retrieve the user details based on their primary email.
*   The function in the _WebHookReceiver_ class checks to see if the notification contains the primary email (if not, we assume it’s just a standard sync message from Google).
*   Once we have confirmed that the primary email is present, we use it to fetch the full details of the user (by calling the _fetch_user_by_email_ method of the G Suite Client Fetcher).

The response from Google resembles the following example:

```json
{
    "kind": "admin#directory#user",
    "id": "10455555555555552791",
    "etag": "enlFCt4L0-k8zzzzzzzzzz/3hRG05XNzzzzzzzzzzzobrHi6Erg",
    "primaryEmail": "hugobert@sennder.com",
    "name": {
        "givenName": "Hugobert",
        "familyName": "Dinkelbaum",
        "fullName": "Hugobert Dinkelbaum"
    },
    "isAdmin": false,
    "isDelegatedAdmin": false,
    "lastLoginTime": "1970-01-01T00:00:00.000Z",
    "creationTime": "2020-02-03T10:54:01.000Z",
    "agreedToTerms": false,
    "suspended": false,
    "archived": false,
    "changePasswordAtNextLogin": true,
    "ipWhitelisted": false,
    "emails": [
        {"address": "hugobert@sennder.com", "primary": true},
        {"address": "hugobert@sennder.com.test-google-a.com"}
    ],
    "nonEditableAliases": ["hugobert@sennder.com.test-google-a.com"],
    "customerId": "C0zzzzzpm",
    "orgUnitPath": "/",
    "isMailboxSetup": true,
    "isEnrolledIn2Sv": false,
    "isEnforcedIn2Sv": false,
    "includeInGlobalAddressList": true,
    "customSchemas": {
        "PM_Teams": {
            "team_name1": true,
            "team_name2": true
        },
        "Roles": {
            "role_name1": true,
            "role_name2": true

        },
        "User_Level": {"Team_Lead": true, "Admin": true, "Basic": true},
```

You can see that the response contains a number of custom G suite fields such as “User_Level”. We these fields to determine the equivalent permissions when creating the user entry in Reinhardt.

After we receive the response, we use Djangos [built-in serializer](https://www.django-rest-framework.org/api-guide/serializers/) to serialize the user details into an object called “UserEntity” like this:

```python
class GSuiteUserSerializer(Serializer):
    primaryEmail = CharField()
    name = DictField()
    isAdmin = BooleanField()

    def create(self, validated_data) -> UserEntity:

        return UserEntity(
            email=validated_data["primaryEmail"],
            first_name=validated_data["name"]["givenName"],
            last_name=validated_data["name"]["familyName"],
            is_admin=validated_data["isAdmin"],
        )
```
Then, we check if the user already exists in the Reinhardt database. If they are a newly created G Suite user, we call the standard Django [_create_user()_ method](https://docs.djangoproject.com/en/3.0/ref/contrib/auth/#django.contrib.auth.models.UserManager.create_user) to insert the user entry into the Reinhardt database as well.

```python
if "primaryEmail" in data:
        user_email = data["primaryEmail"]
        user = gsuite_client_fetcher.fetch_user_by_email(user_email=user_email)
        try:
            # Check if the user already exists in DB
            u = User.objects.get(username=user.email)
            print('User exists, updating user: ', user)
            # Refresh all user data:
            # We're just refreshing the first/last names here
            # but you can add more fields where necessary..
            u.first_name = user.first_name
            u.last_name = user.last_name
            u.save()
        except User.DoesNotExist:
            print('User did not exist previously')
            User.objects.create_user(username=user.email,
                                     first_name=user.first_name,
                                     last_name=user.last_name,
                                     email=user.email,
                                     password='default_password')
```

---
**A note about passwords**

Unsurprisingly, we can’t get the user password from the Directory API. The problem is, Django’s _create_user()_ method requires a password.

To get around this, we just randomly generate a password and require the user to reset their password when they first log in to Reinhardt.

---

If the notification relates to an existing user, we make sure that the user already exists in the Reinhardt database and we then update the entry, field by field.

We use [Django helper functions](https://docs.djangoproject.com/en/3.0/topics/auth/default/#user-objects) to update all the fields in the database with the new data G Suite admin API. It doesn’t matter if 1 field or 5 fields were changed, all fields are updated with the latest values from G Suite.
&nbsp;
## Synchronizing user accounts saves a lot of time

Before we created this integration, sysadmins had to create each user twice. Once in G Suite and again in Reinhardt. This was particularly cumbersome at the beginning of the month when large batches of new hires would need to be onboarded. For one user, it doesn’t take that long to create duplicate accounts. But it’s more laborious and error-prone when you have to it for 20 consecutive accounts.

Although this integration turned out to be much more complex than we expected, we’re happy that we could spare people some monotonous busywork. And hopefully, we’ve also made it easier for you to do the same thing — but without the “gotchas” that slowed us down.