LGTM Devlog 11: Writing the Serverless Function for receiving GitHub webhooks with Pydantic validation
Tags: devjournal, webdev, github, python
Published: 2020-12-31 17:54:39

Now that we know what the github forks look like, it's time to write the function for receiving them. You can see the code matching this post at this commit: [40ac1367](https://github.com/meseta/lgtm/tree/40ac1367ceebcd1b84a03d7f33ce54082e4178f7)

# Data validation with Pydantic
First, validating data and extracting it. I need to pull out four or five pieces of data from the json being sent. While this is a relatively simple case of reading the various nested keys from the json, there's problems with that: what if some of the keys don't exist? We need to throw an error, ideally one that explains what the problem is. Suddenly pulling out 4 values becomes having to write four if statements and for error messages.

While in this particular instance, we don't expect the github API to be omitting values, it's still to do things rigorously. One of the best options for this validation problem is to use a module like [Pydantic](https://pydantic-docs.helpmanual.io/). Not only does it do this validation for you, it generates meaningful errors, has static type annotations, and a whole bunch of very wholesome reasons why you'd use Pydantic over manually decoding JSONs.

In fact, it's so useful that Python API libraries like Starlette/FastAPI built-in support for Pydantic models for defining what API endpoints require as payload, or output. Because we're running on Google's serverless functions, we don't have those built-in support for Pydantic, but we can still make use of it, we just have to manually run the validation.

So first, we define the data model(s). I noticed that the hook json payload has two members: `forkee` and `repository`. They both describe a repo. And inside each repo, is an `owner` that contains the details of the person who owns the repo. So I decided to buidl my data models like this:

(This is the entire contents of the `app/utils/models.py` file)
```py
""" Models for validation of Github hooks """
from pydantic import BaseModel, Field, Extra  # pylint: disable=no-name-in-module

# pylint: disable=too-few-public-methods,missing-class-docstring
class GitHubUser(BaseModel):
    login: str = Field(..., title="User's user login name")
    id: int = Field(..., title="User's ID")

    class Config:
        extra = Extra.ignore


class GitHubRepository(BaseModel):
    id: int = Field(..., title="Repo's ID")
    full_name: str = Field(..., title="Repo's full name (including owner)")
    owner: GitHubUser = Field(..., title="Owner of repo")
    url: str = Field(..., title="API RUL of the repo")

    class Config:
        extra = Extra.ignore


class GitHubHookFork(BaseModel):
    forkee: GitHubRepository = Field(..., title="The fork created")
    repository: GitHubRepository = Field(..., title="The repository being forked")

    class Config:
        extra = Extra.ignore
```

The `extra` config tells Pydantic to not error out when it receives more data than was given, since I don't want to fully define the github object, just the objects I want to access later.

Now in my main function, I can tell it to parse the raw data from the request and return this object:
```py
    try:
        hook_fork = GitHubHookFork.parse_raw(request.data)
    except ValidationError as err:
        logger.err("Validation error", err=err)
        return abort(400, "Validation error")
```
This `hook_fork` object has the properties we defined, for example, we can check the username of the person who forked our repo using `hook_fork.forkee.owner.login`, and! in the case validation fails, we'll get an exception containing what part of the data didn't match.

# Signature validation
We discovered in the last post that GitHub can secure webhooks using a signature that is calculated from a pre-shared secret value that you tell it to use, and the payload. I found an example of this implementation on Google, but updated it to use the new SHA256 which GitHub wants you to use. I'm going to store the pre-shared secret in an environmental variable called `SECRET` (may rename later), so the code looks like this (contents of `app/utils/verify.py`):

```py
""" Verify the GitHub webhook secret """
import os
import hmac
import hashlib

from flask import Request

SECRET = bytes(os.environ["SECRET"], "utf-8")


def verify_signature(request: Request) -> bool:
    """ Validates the github webhook secret. Will return false if secret not provided """
    expected_signature = hmac.new(
        key=SECRET, msg=request.data, digestmod=hashlib.sha256
    ).hexdigest()
    incoming_signature = request.headers.get("X-Hub-Signature-256", "").removeprefix(
        "sha256="
    )
    return hmac.compare_digest(incoming_signature, expected_signature)
```

A couple of things to note here: the new Python 3.9 `.removeprefix()` method for strings, and using `hmac.compare_digest()` to compare the two digests. This method is equivalent to just doing `==` but it introduces some random timing to prevent time analysis, which makes it a bit more secure (though in our case that is unlikely to be a major consideration, it's still best-practice)

# Main function
So finally, the main function that we will deploy looks like this (full contents of `app/main.py`):

```py
""" Listens to webhooks from GitHub """

import structlog  # type: ignore
from flask import Request, abort
from pydantic import ValidationError

from utils.verify import verify_signature
from utils.models import GitHubHookFork

logger = structlog.get_logger()

OUR_REPO = "meseta/lgtm"


def github_webhook_listener(request: Request):
    """ A listener for github webhooks """

    # verify
    if not verify_signature(request):
        return abort(403, "Invalid signature")

    # decode
    try:
        hook_fork = GitHubHookFork.parse_raw(request.data)
    except ValidationError as err:
        logger.err("Validation error", err=err)
        return abort(400, "Validation error")

    # output
    if hook_fork.repository.full_name == OUR_REPO:
        logger.info("Got fork", data=hook_fork.dict())

    return "OK"
```

Note that at this point, I'm not actually doing anything with the data, just decoding it and logging it. Next sprint's task will be to actually take this data and use it to create accounts and whatever.

# Tests
To ensure we have full test coverage, I wrote a couple of tests, as well as downloaded the payload and hash for a valid hook, and fabricated an invalid one. The `tests/conftest.py` file now looks like this, with fixtures to provide the two test payloads as well as the test client:

```py
""" Setup for tests """

import os
import json
import pytest
from functions_framework import create_app  # type: ignore

TEST_FILES = os.path.join(
    os.path.dirname(os.path.realpath(__file__)),
    "test_files",
)


class Payload:  # pylint: disable=too-few-public-methods
    """ Container for holding header/payload pairs during testing"""

    def __init__(self, header_path, payload_path):
        with open(os.path.join(TEST_FILES, header_path)) as fp:
            self.headers = json.load(fp)
        with open(os.path.join(TEST_FILES, payload_path), "rb") as fp:
            self.payload = fp.read()


@pytest.fixture()
def good_fork():
    """ A payload containing (raw) data, this is recorded from GitHub """
    return Payload("good_fork_headers.json", "good_fork.bin")


@pytest.fixture()
def bad_fork():
    """A payload containing (raw) data, that has been edited to be missing stuff
    but paired wiht a valid signature"""
    return Payload("bad_fork_headers.json", "bad_fork.bin")


@pytest.fixture(scope="package")
def client():
    """ Test client """
    return create_app(
        "github_webhook_listener", os.environ["FUNCTION_SOURCE"]
    ).test_client()

```

And to test our model (contents of `tests/test_model.py`), we simply ask it to parse the good and the bad payloads to see if it'll work or raise the appropriate exceptions:

```py
""" Tests for pydantic models """

import pytest
from pydantic import ValidationError
from app.utils.models import GitHubHookFork

# pylint: disable=redefined-outer-name
def test_model(good_fork):
    """ Test that our validator works """

    hook_fork = GitHubHookFork.parse_raw(good_fork.payload)
    assert hook_fork.forkee.owner.login


def test_model_invalid(bad_fork):
    """ Test that our validator fails correctly """

    with pytest.raises(ValidationError):
        GitHubHookFork.parse_raw(bad_fork.payload)
```

And to test our main function, there's four tests that test various combinations of valid/invalid/missing signatures, valid/invalid payloads (contents of `tests/test_main.py`)

```py
""" Tests for main.py """

# pylint: disable=redefined-outer-name
def test_good_fork(client, good_fork):
    """ For a good fork that's working fine """

    res = client.post("/", headers=good_fork.headers, data=good_fork.payload)
    assert res.status_code == 200


def test_model_validation_fail(client, bad_fork):
    """ Test model validation failure with bad payload but correct signature for it"""

    res = client.post("/", headers=bad_fork.headers, data=bad_fork.payload)
    assert res.status_code == 400


def test_signature_fail(client, good_fork, bad_fork):
    """ Test signature validation failure with good payload but incorrect signature for it"""

    res = client.post("/", headers=bad_fork.headers, data=good_fork.payload)
    assert res.status_code == 403


def test_no_signature(client, good_fork):
    """ When signatures are not supplied """

    res = client.post(
        "/", headers={"Content-Type": "application/json"}, data=good_fork.payload
    )
    assert res.status_code == 403

```

Our final test results show all tests pass, with 100% code coverage, which isn't definitive that our code is bug free, but gives us a little confidence, and helps us make sure there's no regressions when we edit the code later

![Test coverage](https://dev-to-uploads.s3.amazonaws.com/i/nknrklfw8j016nwnyrb2.png)

# Running a real test
To run a real test, I fire up that ngrok endpoint I set up previously (by running `ngrok http 5000`), started up the test server using `pipenv run serve github_webhook_listener`, updating the GitHub Webhook URL to the new temporary ngrok path, and telling GitHub to re-send the previous webhook.

The server logs show that a webhook is received, and you can read the decoded variables that are now in the `hook_fork` object!

![Validating the hook](https://dev-to-uploads.s3.amazonaws.com/i/2j1l14tg2uzan7seri8c.png)

For those interested, all of this is running inside VSCode's Terminal pane, opened into WSL, and inside a tmux session. It's a nice way to run code while developing.

___
This represents all that is needed for a basic GitHub webhook receiving serverless function, that is built in a reasonably robust way. Next we need to upload it, but I will set up the CI to do this instead of me having to upload it every time.