Building a simple TikTok scraping API 🤖
Tags: webdev, showdev, api, javascript
Published: 2022-08-14 18:26:40

**Note: this no longer works as TikTok now blocks cross-origin media requests.**

As part of a project centered around TikTok (coming soon!) I needed a way to get a video's media. The closest thing that TikTok officially provides is their [`oembed` API](https://developers.tiktok.com/doc/embed-videos/), but unfortunately this only includes basic information like `title`, `thumbnail_url` etc. And other unofficial APIs [cost money](https://tikapi.io/#pricing) or require [python](https://github.com/davidteather/TikTok-Api), which I don't want to do. So I needed another way to retrieve this info.

## Leveraging SSR patterns 📦

As a website that handles large volumes of traffic I knew they had some sort of SSR/client-side hydration going on. This generally involves is shipping data to the client in an object inlined inside a `<script>` tag, reducing the number of necessary client-side requests by directly populating components with this data. In [Next.js](https://nextjs.org/) for example, this happens via [server-side props](https://nextjs.org/docs/basic-features/data-fetching/get-server-side-props).

And lo and behold! There's a handy blob provided alongside the page markup for every video. You can confirm this for yourself by visiting any TikTok video's page (`https://www.tiktok.com/:user/video/:id`) and looking at the contents of the `SIGI_STATE` tag via console:

```js
JSON.parse(document.getElementById("SIGI_STATE").innerText);
```

Knowing this I built a simple handler using [deno](https://deno.land/) to fetch this blob and return it directly.

## Enter cheerio & deno 🥣🦕

If the video URL is attached to the request (via `POST` body or params), a simple handler can fetch (scrape) the content for the page:

```js
const response = await fetch(`https://tiktok.com/${video}`);
const html = await response.text();
```

Then, using [cheerio](https://cheerio.js.org/), this raw text can be turned into a queryable jQuery-like API. This provides an entrypoint to get the content of the server-generated blob within the script tag:

```js
const $ = cheerio.load(html);
const appContext = $("#SIGI_STATE").text();
```

From there, it's only a matter of parsing this as JSON and returning pertinent data in the response (`ItemModule`). The data is currently keyed by some unique identifier, `Object.keys` helps remove that:

```js
const json = JSON.parse(appContext);
const key = Object.keys(json.ItemModule)[0];
const data = json.ItemModule[key];

return new Response(JSON.stringify({ ...data }), {
  headers: { 'Content-Type': 'application/json' },
  status: 200,
});
```

And that's it! Now I can make a client-side request to get up-to-date media for a video and display it directly. This also gets around TikTok's CDN expiration which would prevent saving a reference to this media.

> Side-note: comparing this to the [output](https://tikapi.io/documentation/#tag/Public/operation/public.video) for one of the unofficial API options I came across, the data looks identical. So it's likely they're doing the same thing (but charging almost $200/mo for unlimited usage)!

---

Anyways, this is fragile and could break if they decide to rename the script tag, rate-limit, etc. But this demonstrates how easy it is to spin up an instance for your own project. And maybe with enough bot traffic TikTok will create a public API that accomplishes this instead of making us scrape for it! 😇