  Transformer has demonstrated its great power to learn contextual word
representations for multiple languages in a single model. To process
multilingual sentences in the model, a learnable vector is usually assigned to
each language, which is called "language embedding". The language embedding can
be either added to the word embedding or attached at the beginning of the
sentence. It serves as a language-specific signal for the Transformer to
capture contextual representations across languages. In this paper, we revisit
the use of language embedding and identify several problems in the existing
formulations. By investigating the interaction between language embedding and
word embedding in the self-attention module, we find that the current methods
cannot reflect the language-specific word correlation well. Given these
findings, we propose a new approach called Cross-lingual Language Projection
(XLP) to replace language embedding. For a sentence, XLP projects the word
embeddings into language-specific semantic space, and then the projected
embeddings will be fed into the Transformer model to process with their
language-specific meanings. In such a way, XLP achieves the purpose of
appropriately encoding "language" in a multilingual Transformer model.
Experimental results show that XLP can freely and significantly boost the model
performance on extensive multilingual benchmark datasets. Codes and models will
be released at https://github.com/lsj2408/XLP.
