The Puppeteer Language Experiment
Tags: webdev, javascript, typescript, productivity
Published: 2024-05-21 15:00:00

If you're using [Puppeteer](https://pptr.dev/) for automation, tests, or web scraping, you've likely encountered the question of how to set the browser's language. Controlling the language explicitly is crucial because the language on your local system might differ from that on another remote system where you want to run Puppeteer, such as CI (e.g., GitHub Actions) or a Serverless environment (e.g., AWS Lambda or CloudFlare Workers).

To my surprise, it’s not well documented how to do that. It seems there are a bunch of options, and you have to figure out what works best for you. After spending a considerable amount of time on research, I’ve compiled a list of options. I have validated each option against [BrowserLeaks](https://browserleaks.com/javascript), which shows you all available information on your browser and its supported features, including the locale and the accepted language, which ultimately determines the content you’re going to see.

## What Language?
There are two ways to determine the requester's language: client-side and server-side. On the client side, with JavaScript, you have the [`navigator.language`](https://developer.mozilla.org/en-US/docs/Web/API/NavigatorLanguage/language) and [`navigator.languages`](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/languages) properties, and the [`Intl.DateTimeFormat().resolvedOptions()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat/resolvedOptions) method. The values do not necessarily match each other because `navigator.language` seems to use the language from Chrome settings, while `Intl.DateTimeFormat().resolvedOptions()` reflects the operating system's language. On the server side, you have the `Accept-Language` header that is sent with every HTTP request from the browser. The browser uses the `navigator.languages` property to fill the header values.

> Side note: I use the term *language* to refer to the first part `de` of a *locale* like `de-DE`.

It must be noted that the server can decide to return the content of a website in the language reflecting the HTTP `Accept-Language` header values, or it can use JavaScript to detect the language and redirect the user to a localized URL (think of `example.com/de-DE/`), or it could dynamically load the actual content from the server via JavaScript based on the properties.

## Set Up Puppeteer
I'm using the `puppeteer` package, which automatically downloads a recent version of [Chrome for Testing](https://developer.chrome.com/blog/chrome-for-testing/) (I think it used to be Chromium). There is also `puppeteer-core` if you want to manage the browser installation yourself or if the environment already provides a browser.

To start the browser, all we need to do is call the `launch` function without any further arguments needed. Puppeteer provides all defaults.

```ts
import puppeteer from "puppeteer";

const LANG = 'de-DE';

const browser = await puppeteer.launch();
const page = await browser.newPage();
```

I defined the constant `LANG` as the target language for the browser. In the next few steps, I'll show you how to apply this setting.

### Command line argument `--lang`
Chrome (and Chromium) provides a plethora of [command line arguments](https://peter.sh/experiments/chromium-command-line-switches/). The argument [`--lang`](https://peter.sh/experiments/chromium-command-line-switches/#lang) can be used to set the language on startup. Keep in mind it must be merged with the [default args](https://github.com/puppeteer/puppeteer/blob/9939c891b3ab1bb25160913129b202a71cef86b3/packages/puppeteer-core/src/node/ChromeLauncher.ts#L158-L268) from Puppeteer.

```ts
console.log(`Using --lang=${LANG}`);

const args = [...puppeteer.defaultArgs(), `--lang=${LANG}`];
const browser = await puppeteer.launch({
  args,
});

const page = await browser.newPage();
```

### Environment variable `LANG`
While the environment variable is not officially documented, there are references to it on the [Chromium bug tracker](https://issues.chromium.org/issues/40535085#comment15) and the [Puppeteer repository](https://github.com/puppeteer/puppeteer/issues/5970#issuecomment-690677674). However, this seems to work only on Linux. Even worse, setting this environment variable causes the browser to fail to start on certain operating systems.

```ts
console.log(`Using env.LANG=${LANG}`);

const browser = await puppeteer.launch({
  env: {
    LANG,
  }
});

const page = await browser.newPage();
```

### HTTP header `Accept-Language`

[Additional HTTP headers](https://pptr.dev/api/puppeteer.page.setextrahttpheaders/) like `Accept-Language` can be set on every page request. This doesn't affect the browser language itself, but the requested website may return the content in the requested language if it respects this header.

```ts
console.log(`Sending HTTP header Accept-Language: ${LANG}`);

const browser = await puppeteer.launch();

const page = await browser.newPage();
await page.setExtraHTTPHeaders({
  'Accept-Language': LANG,
});
```

### Override `navigator.language`

Overriding the `navigator.language` property on a page might seem like a hack, but it is actually an [official example](https://pptr.dev/api/puppeteer.page.evaluateonnewdocument/#example) in the Puppeteer docs.

```ts
console.log(`Overriding navigator.language=${LANG}`);

const browser = await puppeteer.launch();

const page = await browser.newPage();
await page.evaluateOnNewDocument((lang) => {
  Object.defineProperty(navigator, 'language', {
    get() {
      return lang;
    },
  });
  Object.defineProperty(navigator, 'languages', {
    get() {
      return [lang];
    },
  });
}, LANG);
```

### Chrome DevTools Protocol `Network.setUserAgentOverride`

Puppeteer uses the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) (CDP) to communicate with Chrome. The [`Network.setUserAgentOverride`](https://chromedevtools.github.io/devtools-protocol/1-3/Network/#method-setUserAgentOverride) method allows setting `acceptLanguage` as the browser language to emulate. Note that `userAgent` is a required parameter, but we keep the default value.

```ts
console.log(`Using CDP Network.setUserAgentOverride(acceptLanguage: ${LANG})`);

const browser = await puppeteer.launch();

const page = await browser.newPage();

const cdpSession = await page.createCDPSession();
cdpSession.send('Network.setUserAgentOverride', {
  userAgent: await browser.userAgent(),
  acceptLanguage: LANG,
});
```

## Results
I ran all options sequentially and extracted the values for `navigator.language`, `navigator.languages`, `Intl.DateTimeFormat().resolvedOptions()` from [BrowserLeaks JS](https://browserleaks.com/js) and the `Accept-Language` header from [BrowserLeaks IP](https://browserleaks.com/ip). Here are the results for each option:

```txt
System:  darwin/23.1.0, arm64
Browser: Chrome/125.0.6422.60

Using --lang=de-DE
        Internationalization Locale: en-GB
        Navigator Language: en-GB
        HTTP Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

Using env.LANG=de-DE
        Internationalization Locale: en-GB
        Navigator Language: en-GB
        HTTP Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

Sending HTTP header Accept-Language: de-DE
        Internationalization Locale: en-GB
        Navigator Language: en-GB
        HTTP Accept-Language: de-DE

Overriding navigator.language=de-DE
        Internationalization Locale: en-GB
        Navigator Language: de-DE
        HTTP Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

Using CDP Network.setUserAgentOverride(acceptLanguage: de-DE)
        Internationalization Locale: en-GB
        Navigator Language: de-DE
        HTTP Accept-Language: de-DE
```

As you can see, the results vary significantly. None of the options impacted the locale returned from `Intl.DateTimeFormat().resolvedOptions()`. Surprisingly, even the `--lang` option didn't affect the browser language as reflected by `navigator.language`. This is unexpected, as I believe in older versions of Chrome/Chromium or on different operating systems, this flag would be respected. Using the `Network.setUserAgentOverride` command from CDP appears to be the most reliable way to control the browser language, despite being the least documented option.

## Contribution
I implemented this project as a reproducible test suite that one can run locally with minimal setup. If you're interested, please check the repository and run the tests on your local machine. I'm curious to see if the results differ across various operating systems or browser versions.

{% embed https://github.com/zirkelc/puppeteer-language-experiment %}