Multi-lingual models¶

Most of the models available in this library are mono-lingual models (English, Chinese and German). A few multi-lingual models are available and have a different mechanisms than mono-lingual models. This page details the usage of these models.

The two models that currently support multiple languages are BERT and XLM.

XLM¶

XLM has a total of 10 different checkpoints, only one of which is mono-lingual. The 9 remaining model checkpoints can be split in two categories: the checkpoints that make use of language embeddings, and those that don’t

XLM & Language Embeddings¶

This section concerns the following checkpoints:

xlm-mlm-ende-1024 (Masked language modeling, English-German)
xlm-mlm-enfr-1024 (Masked language modeling, English-French)
xlm-mlm-enro-1024 (Masked language modeling, English-Romanian)
xlm-mlm-xnli15-1024 (Masked language modeling, XNLI languages)
xlm-mlm-tlm-xnli15-1024 (Masked language modeling + Translation, XNLI languages)
xlm-clm-enfr-1024 (Causal language modeling, English-French)
xlm-clm-ende-1024 (Causal language modeling, English-German)

These checkpoints require language embeddings that will specify the language used at inference time. These language embeddings are represented as a tensor that is of the same shape as the input ids passed to the model. The values in these tensors depend on the language used and are identifiable using the lang2id and id2lang attributes from the tokenizer.

Here is an example using the xlm-clm-enfr-1024 checkpoint (Causal language modeling, English-French):

import torch
from transformers import XLMTokenizer, XLMWithLMHeadModel

tokenizer = XLMTokenizer.from_pretrained("xlm-clm-1024-enfr")

The different languages this model/tokenizer handles, as well as the ids of these languages are visible using the lang2id attribute:

# Continuation of the previous script
print(tokenizer.lang2id)  # {'en': 0, 'fr': 1}

These ids should be used when passing a language parameter during a model pass. Let’s define our inputs:

# Continuation of the previous script
input_ids = torch.tensor([tokenizer.encode("Wikipedia was used to")]) # batch size of 1

We should now define the language embedding by using the previously defined language id. We want to create a tensor filled with the appropriate language ids, of the same size as input_ids. For english, the id is 0:

# Continuation of the previous script
language_id = tokenizer.lang2id['en']  # 0
langs = torch.tensor([language_id] * input_ids.shape[1])  # torch.tensor([0, 0, 0, ..., 0])

# We reshape it to be of size (batch_size, sequence_length)
langs = langs.view(1, -1) # is now of shape [1, sequence_length] (we have a batch size of 1)

You can then feed it all as input to your model:

# Continuation of the previous script
outputs = model(input_ids, langs=langs)

The example run_generation.py can generate text using the CLM checkpoints from XLM, using the language embeddings.

XLM without Language Embeddings¶

This section concerns the following checkpoints:

xlm-mlm-17-1280 (Masked language modeling, 17 languages)
xlm-mlm-100-1280 (Masked language modeling, 100 languages)

These checkpoints do not require language embeddings at inference time. These models are used to have generic sentence representations, differently from previously-mentioned XLM checkpoints.

BERT¶

BERT has two checkpoints that can be used for multi-lingual tasks:

bert-base-multilingual-uncased (Masked language modeling + Next sentence prediction, 102 languages)
bert-base-multilingual-cased (Masked language modeling + Next sentence prediction, 104 languages)

These checkpoints do not require language embeddings at inference time. They should identify the language used in the context and infer accordingly.

XLM-RoBERTa¶

XLM-RoBERTa was trained on 2.5TB of newly created clean CommonCrawl data in 100 languages. It provides strong gains over previously released multi-lingual models like mBERT or XLM on downstream taks like classification, sequence labeling and question answering.

Two XLM-RoBERTa checkpoints can be used for multi-lingual tasks:

xlm-roberta-base (Masked language modeling, 100 languages)
xlm-roberta-large (Masked language modeling, 100 languages)