Visualize your Fastly traffic on a real time globe using Glitch
Tags: data, visualization, webgl, tutorial
Published: 2023-08-09 13:37:34


On the [Fastly developer hub](https://developer.fastly.com), we recently added a visualization of the traffic flowing across the Fastly edge network.  It was a big hit and customers ask me how they can visualize their own traffic like this.

![Screenshot of developer.fastly.com showing a globe visualization](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3qcdu3r68p9aw8tcr0wt.png)

Fear not.  I've repackaged the React component we made to do this as a Glitch app, so you can create your very own globe.  Not only that, but a lot of the feedback I was getting was stuff like "This would look great on the giant screen in our lobby" so we made a non-interactive version designed to fill a 16:9 screen.  Let's dive in.

## Make a log processor app

First, you need somewhere for Fastly to send your log data.  A Glitch app is ideal for this (Glitch is a collaborative and free code playground which is now part of Fastly).

1. Remix the [fastly-globeviz-data](https://glitch.com/edit/#!/fastly-globeviz-data) Glitch app
1. Note the URL that Glitch assigns to your new app, eg `monthly-ferret-taxi.glitch.me` or `glitch.com/edit/#!/monthly-ferret-taxi`.

What does this app do?  It's a one-file ExpressJS app that has a few route handlers:

* POST requests to `/` are unpacked and each line is treated as an event to emit on a [Server sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) stream.
* GET requests to `/` receive a streaming response that includes any events as they are created.
* A GET request to `/.well-known/fastly/logging/challenge` is answered as required by Fastly's [log challenge rules](https://developer.fastly.com/learning/integrations/logging/#http-challenge) for HTTP endpoints.

Now you have your own log collector, it's time to send some data to it!

## Instrument your Fastly service

Configure your Fastly service to emit traffic data to the Glitch app you just created.

1. Create a Fastly service if you don't have one already.  If you do, create a new draft version of the service.
1. Decide on a three character code to identify this traffic, e.g. if you are the New York Times, you could use `NYT`.  This is so that if you want to you can send multiple different website traffic streams to the same globe visualization, you can identify them and colour code them.
1. Decide on a sample rate.  A good rate to sample is 20-50 requests per second.  If that represents roughly a thousandth of your traffic, the sample rate would be 1000.  A sample rate of 1 is 100% of traffic.  Err on the side of sampling too little and dial it up, otherwise you may trigger rate limiting on Glitch.
1. Add the following VCL as an **INIT** VCL snippet:
    ```
    table globeviz_config {
      "service_name": "ABC", // Change this value
      "sample_rate": "1000"  // Change this value
    }
    
    // https://deviceatlas.com/device-data/explorer/#defined_property_values/7/2705619
    table globeviz_known_browsers {
      "Chrome": "C",
      "Chrome Mobile": "C",
      "Firefox": "F",
      "Firefox for Mobile": "F",
      "Edge": "E",
      "Kindle Browser": "K",
      "Safari": "S",
      "Samsung Browser": "A"
    }


    // Record the following data (space separated)
    //    PIO             Name of source service (3 char)
    //    LCY             Name of Fastly POP (3 char)
    //    51.43,-0.23     Client location (var length)
    //    17234           Duration (var length)
    //    EN              Normalized Accept-Language (2 char)
    //    M               Cache state (1 char: H, M, P)
    //    2               HTTP version (numeric, variable length)
    //    1.2             TLS version (numeric, variable length)
    //    C               Browser (1 char: C, F, E, K, S, A)
    //
    sub vcl_log {
      // Record only non-shielding, non-VPN requests at the specified sample frequency
      if (fastly.ff.visits_this_service == 0 && client.geo.proxy_type == "?" && client.geo.latitude != 0 && randombool(1,std.atoi(table.lookup(globeviz_config, "sample_rate", "100000")))) {
        log "syslog " req.service_id " fastly-globeviz :: "
          table.lookup(globeviz_config, "service_name", "-") " "
          server.datacenter " "
          client.geo.latitude "," client.geo.longitude " "
          time.elapsed.usec " "
          std.toupper(
            accept.language_lookup(
              "en:de:fr:nl:jp:es:ar:zh:gu:he:hi:id:it:ko:ms:pl:pt:ru:th:uk",
              "en",
              req.http.Accept-Language
            )
          ) " "
          substr(fastly_info.state, 0, 1) " "
          regsuball(req.proto, "[^\d.]", "") " "
          regsuball(tls.client.protocol, "[^\d.]", "") " "
          table.lookup(globeviz_known_browsers, client.browser.name, "Z")
        ;
      }
    }
    ```
1. Create an HTTPS [log endpoint](https://developer.fastly.com/learning/integrations/logging/) called `fastly-globeviz` pointing to your Glitch URL (choose a content type of "text/plain" and a placement of "none").  You can do that in the UI or with this cURL command:
    ```
    curl -i  -X POST "https://api.fastly.com/service/$SERVICE_ID/version/$VERSION/logging/https" -H "Fastly-Key: $YOUR_FASTLY_TOKEN" -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -d "name=fastly-globeviz&placement=none&content_type=text/plain&url=https://$GLITCH_APP_ID.glitch.me/"
    ```
1. Activate the new version of your service

## Test the log receiver

Open the URL of your glitch app in a browser and you should start to see traffic log data streaming in.  

![Browser screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o019ab4x8wig0mib619a.png)

This is the data your Fastly service is emitting.  This is a good time to check that you're happy with the type of data you're sharing with the world, since the Glitch app you're sending it to is exposing it publicly.

## View it on a map!

Now you can visualize your log data on a globe.  You don't actually need your own version of the globe, so you don't need to remix an app for this, you can just use mine directly - and point it at your data source.

1. Go to [https://fastly-globeviz-ui.glitch.me/](https://fastly-globeviz-ui.glitch.me/) in your browser.  You should see traffic animating, but that's not your traffic right now, it's the default traffic source.
1. Add a query parameter `?SOURCE_URL=<your log collector>` to the URL, e.g. `?SOURCE_URL=https://monthly-ferret-taxi.glitch.me`.  You should now see your data!
1. Go look at [the `src/config.js` file](https://glitch.com/edit/#!/fastly-globeviz-ui?path=src%2Fconfig.js%3A1%3A0) in the UI Glitch app.  This is where you can see all the variables that are configurable.  `SOURCE_URL` was one of them, but they are all overridable by adding more query parameters to your URL.  If you prefer, you can remix the app and edit `config.js` directly.

## Give feedback and contribute

Both the Glitch apps are remixable, so go ahead and make changes, and let us know what you come up with in the [Fastly community forum](https://community.fastly.com)!