How Serverless, Puppeteer, FFMPEG, and AWS Lambda can help you get the GIFs from Twitter
Tags: puppeteer, node, serverless, ffmpeg
Published: 2019-11-14 18:42:24

---
title: "How Serverless, Puppeteer, FFMPEG, and AWS Lambda can help you get the GIFs from Twitter"
published: true
description: "How Serverless, Puppeteer, FFMPEG, and AWS Lambda can help you get the GIFs from Twitter."
tags: [Puppeteer, Nodejs, Serverless, FFMPEG]
canonical_url: "https://evanhalley.dev/post/get-me-the-gif/"
cover_image: "https://thepracticaldev.s3.amazonaws.com/i/zho1ufm9a9s8rp9iayic.gif"
---

Running Puppeteer on Amazon Web Services' (AWS) serverless computing platform, Lambda, is it a bit of a challenge.  Puppeteer and it's bundled version of Chromium easily exceeds the lambda size limitation.  There are a few options you opt for instead:

1. Run or maintain your own farm of Google Chrome / Chromium instances
2. Use a Google Chrome / Chromium-as-a-Service offering like Browserless
3. Use an NPM module, `npm i @serverless-chrome/lambda` to handle this for you

I'm going to be walking you through how I built "Get Me the GIF" (referred to as GMTG hereafter) with Puppeteer, Serverless, Chromium, and Nodejs.

## The Idea

The idea behind GMTG is an easy one.  When Twitter user's tweet GIFs out, Twitter will show these GIFs as videos instead of GIFs. 

{% twitter 985862643159363584 %}

I'm somewhat of a GIF collector.  You never know when a GIF is more useful in communicating an idea than words are.  When I see GIFs on Twitter that are funny, I like to save those for later.  Twitter for the web and Android make this impossible.  One day, I had more free time than usual and decided to open up Chrome DevTools and explore the code behind how a tweet is displayed in the browser.  Long story short, it's relatively easy to capture and download these GIFs (videos).  I wrote a little extra code to use FFMPEG to convert the video I capture from Twitter into a GIF I can add to my collection.

I'm not going to highlight all the pieces I used, but just a few pieces of code I thought were useful in helping me reach my end goal, getting all of those GIFs.

## The Flow

You give GMTG a URL to a tweet containing the GIF, like:

https://twitter.com/EvanHalley/status/1130891914675445760

It should give you a GIF.

![Malcolm X slamming a table](https://thepracticaldev.s3.amazonaws.com/i/llxfnvgj4n46t4w7knps.gif)

The overall flow:

1. Open the Tweet
2. Intercept the network request that rendered the MP4 preview frame
3. Parse out identifier of the MP4 preview frame
4. Build the URL to access the MP4 and download it
5. Use FFMPEG to convert the MP4 to a GIF
6. Make the GIF available to the user

## Getting Chrome to Run in AWS Lambda

<i>Note: For this project, I am using the Serverless Framework to debug and deploy my serverless app to AWS Lambda.  I'm not going to dive into Serverless, but checkout this great [Serverless tutorial](https://serverless.com/framework/docs/providers/aws/guide/quick-start/).</i>

As mentioned earlier, getting a Puppeteer + Chrome based Nodejs app running in AWS Lambda is difficult because of the [deployment package size limits](https://docs.aws.amazon.com/lambda/latest/dg/limits.html).  In order to get around this limitation, I used a NPM module, [serverless-chrome](https://github.com/adieuadieu/serverless-chrome).

Without jumping to far into the details, serverless-chrome handles everything needed to get Chrome up and running in a serverless environment and manages to get around the deployment package size limitations.

> The aim of this project is to provide the scaffolding for using Headless Chrome during a serverless function invocation. Serverless Chrome takes care of building and bundling the Chrome binaries and making sure Chrome is running when your serverless function executes. In addition, this project also provides a few example services for common patterns (e.g. taking a screenshot of a page, printing to PDF, some scraping, etc.)

Once you have a Nodejs, Puppeteer, and Serverless project bootstrapped, you can easily add serverless-chrome:

```bash
npm install --save @serverless-chrome/lambda
```

In your source code, connect to a Chrome instance running in a serverless environment:

```javascript
const launchChrome = require("@serverless-chrome/lambda");

async function getChrome() {
    let chrome = await launchChrome();

    let response = await request
        .get(`${chrome.url}/json/version`)
        .set("Content-Type", "application/json");

    console.log(JSON.stringify(response.body));
    let endpoint = response.body.webSocketDebuggerUrl;

    return {
        endpoint,
        instance: chrome
    };
}
```

The code snippet above, calls `launchChrome()` to start a Chrome process.  Once it's launched, we can query the Chrome instance to find the URL to the Chrome DevTools Protocol (CDP) socket.  Puppeteer uses this URL to connect to Chrome.

Making a GET request to this URL + `/json/version` returns:

```json
{
   "Browser": "HeadlessChrome/78.0.3904.97",
   "Protocol-Version": "1.3",
   "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/78.0.3904.97 Safari/537.36",
   "V8-Version": "7.8.279.23",
   "WebKit-Version": "537.36 (@021b9028c246d820be17a10e5b393ee90f41375e)",
   "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec"
}
```

We are after `webSocketDebuggerUrl`.

Once we have a URL to the CDP socket, connecting with Puppeteer is straighforward:


```javascript
const puppeteer = require("puppeteer");
...
let cdpSocketUrl = 'ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec';
let browser = await puppeteer.connect({
    browserWSEndpoint: cdpSocketUrl
});
```

There is a caveat to know about when using serverless-chrome.  It has not been updated in over a year, which means the latest prebuilt version of Chromium (69.0.3497.81) it uses is over a year old.  This means it is pretty much only guaranteed to work with older versions Puppeteer.

## Intercepting Requests with Puppeteer

Puppeteer has a handy API for intercepting ALL network requests the browser makes when loading a web page.  You can intercept these requests and either continue or abort them.  There are some really useful cases where this level of control is desired, such as capturing a webpage screenshot, but not processing any of the image or javascript.  In my case, I just wanted to identify the MP4 thumbnail network request.

I discovered that the URL to the MP4 thumbnail looks like:

https://pbs.twimg.com/tweet_video_thumb/1234567890.jpg

The link to the MP4 looks like:

https://video.twimg.com/tweet_video/1234567890.mp4

Using Puppeteer I am able to write request interception code that looks for this URL.

```javascript
const VIDEO_THUMBNAIL_PREFIX = 'https://pbs.twimg.com/tweet_video_thumb/';
let videoUrl = null;
page = await browser.newPage();
await page.setRequestInterception(true);

page.on('request', request => {

    if (request.url().startsWith(VIDEO_THUMBNAIL_PREFIX) && request.url().endsWith('.jpg')) {
        let thumbnailUrl = request.url();
        let assetId = thumbnailUrl.replace(VIDEO_THUMBNAIL_PREFIX, '')
            .replace('.jpg', '');
        videoUrl = VIDEO_URL_PREFIX + assetId + '.mp4';
    }
    request.continue();
});
await page.goto(tweetUrl);
```

Once I had the URL to the video thumbnail, I'm easily able to build a URL to the video so I can download it later.

## Converting the Video

FFMPEG is one of the most popular command line utilities for transcoding a wide area of video, audio, and still images.  It's written in C.  However, like many things nowadays, you can instrument it with JavaScript.  I discovered a [GitHub Gist](https://gist.github.com/identiq/d633dfda66698d24c390bbe8422090b3) that was tackling a similar problem, converting a video into a GIF.

Using traditional command line FFMPEG, you could execute the operation with:

```bash
ffmpeg -i input_video.mp4 output.gif
```

Using a Nodejs library, [fluent-ffmpeg](https://github.com/fluent-ffmpeg/node-fluent-ffmpeg), the same operation looks like:

```javascript
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
const ffmpeg = require('fluent-ffmpeg');
ffmpeg.setFfmpegPath(ffmpegPath);

ffmpeg('input_video.gif')
    .on('end', () => {
        // done, do something with output.gif
    })
    .on('error', err => {
        // oh noe error!
    })
    .save('output.gif');
```

The use of `fluent-ffmpeg` requires the use of another Nodejs libray, [@ffmpeg-installer/ffmpeg](https://www.npmjs.com/package/@ffmpeg-installer/ffmpeg).  It installs a version of ffmpeg compatible with the underlying platform (Linux, Mac, or Windows).  You then tell `fluent-ffmpeg` where it has been installed and it takes over from there.

## The End

I've deployed GMTG to AWS Lambda.  You can make HTTP GET calls to it.  The value of the URL parameter should be a URL to a Tweet containing a GIF (short movies probably work as well).

https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=

In this example, we will extract the GIF out of this Tweet

{% twitter 1194435785082187778 %}

Using the following GMTG API call:

https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=https://twitter.com/ThePracticalDev/status/1194435785082187778

The GIF:

![Newer is always better meme](https://thepracticaldev.s3.amazonaws.com/i/zho1ufm9a9s8rp9iayic.gif)

The source code has been uploaded to GitHub.

https://github.com/evanhalley/get-me-the-gif

Let me know if you have any questions by leaving a comment or on [Twitter, @EvanHalley](https://twitter.com/evanhalley).

**If you like reading articles about all the cool things you can do with Puppeteer, please sign up for my email newsletter, [The PuppetHero Digest](https://puppethero.com).**

*This article was originally posted at [EvanHalley.dev](https://evanhalley.dev/post/get-me-the-gif/)*