Lambda Function to send your Github Traffic Statistics to New Relic Insights
Tags: python, newrelic, lambda, github
Published: 2020-07-19 16:51:52

For repositories that you have push access to within Github, the [Github traffic API](https://developer.github.com/v3/repos/traffic/) provides access to the information provided in the [graphs section](https://help.github.com/articles/about-repository-graphs/#traffic) of the repositories UI. 

This information includes referral sources, referral paths, views, and clones. The API returns data over the past 14 day period. As the API only returns data over the past 14 day period, this solution aims to give users access to historical Traffic data for their Github repositories within Insights. 

This Lambda function calls each endpoint every 15 days and creates custom events that you can then query and create visualisations within New Relic Insights. 

See the New Relic One Dashboard below:

![Image](https://i.imgur.com/Y6lRjXa.png)

## How it works

The function uses Github's token based API authentication to gain access to the user's repository data. For each of the users repostories, the script makes a call to each Traffic API endpoint and records a custom event using the [Event API](https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/introduction-event-api) if there has been any Traffic data reported.

```python
def lambda_handler(event, context):
  base_url = "https://api.github.com"
  user = os.getenv("GITHUB_USERNAME")
  print("Getting repositories for %s" % user)
  req = api_req("%s/users/%s/repos" % (base_url, user))

  ur = [(r["name"]) for r in req]
  print("Found %s repositories" % len(ur))

  print(" ")

  for repo_name in ur:

    print("Getting referral data for %s" % repo_name)
    referrals = api_req(
      "%s/repos/%s/%s/traffic/popular/referrers" % (base_url, user, repo_name)
    )
    if len(referrals) > 0:
      for ref in referrals:
        referred = {
          "eventType": "Referral",
          "repo_name": repo_name,
          "referrer": ref["referrer"],
          "count": ref["count"],
          "uniques": ref["uniques"],
        }
        print(referred)
        record_custom_event(referred)

        print("Getting top referral path data for %s" % repo_name)

        paths = api_req(
          "%s/repos/%s/%s/traffic/popular/paths" % (base_url, user, repo_name)
        )
        if len(paths) > 0:
          for ref in paths:
            paths = {
              "eventType": "ReferralPath",
              "repo_name": repo_name,
              "path": ref["path"],
              "title": ref["title"],
              "count": ref["count"],
              "uniques": ref["uniques"],
            }
            print(paths)
            record_custom_event(paths)

            print("Getting views for %s" % repo_name)
            req = api_req("%s/repos/%s/%s/traffic/views" % (base_url, user, repo_name))
            if req["count"] > 0 or req["uniques"] > 0:
              viewed = {
                "eventType": "View",
                "repo_name": repo_name,
                "count": req["count"],
                "uniques": req["uniques"],
              }
              print(viewed)
              record_custom_event(viewed)

              print("Getting clones for %s" % repo_name)
              req = api_req("%s/repos/%s/%s/traffic/clones" % (base_url, user, repo_name))
              if req["count"] > 0 or req["uniques"] > 0:
                cloned = {
                  "eventType": "Clone",
                  "repo_name": repo_name,
                  "count": req["count"],
                  "uniques": req["uniques"],
                }
                record_custom_event(cloned)
                print(" ")
```

You can view the complete source code on my [Github](https://github.com/AnthonyBloomer/github-traffic-insights).  

## Setup

Clone the repository: `git clone https://github.com/AnthonyBloomer/github-traffic-insights.git`

Update the following environment variables in the `template.yml` file:

- `GITHUB_TOKEN` - Create a personal access token using your [Personal access tokens settings page](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line). 
- `GITHUB_USERNAME` - Your GitHub username.
- `NEW_RELIC_ACCOUNT_ID` - Your New Relic [Account ID](https://docs.newrelic.com/docs/accounts/install-new-relic/account-setup/account-id)
- `NEW_RELIC_INSERT_KEY` - Your New Relic [Insert Key](https://docs.newrelic.com/docs/apis/get-started/intro-apis/types-new-relic-api-keys#event-insert-key)

Deploy using the [AWS CLI](https://aws.amazon.com/cli/):

```
aws cloudformation deploy --template template.yml --stack-name github
```

### Sample Dashboard

The project also includes the [sample dashboard JSON](https://github.com/AnthonyBloomer/github-traffic-insights/blob/master/example_dashboard.json) shown above which you can import into your New Relic account using the [Dashboards API](https://docs.newrelic.com/docs/insights/insights-api/manage-dashboards/insights-dashboard-api).