Adversarial Training on Disentangling Meaning and Language Representations for Unsupervised Quality Estimation (2022)

Abstract: We propose a method to distill language-agnostic meaning embeddings from multilingual sentence encoders for unsupervised quality estimation of machine translation. Our method facilitates that the meaning embeddings focus on semantics by adversarial training that attempts to eliminate language-specific information. Experimental results on unsupervised quality estimation reveal that our method achieved higher correlations with human evaluations.

## introduction quality estimation (qe) is a task of estimating translation quality without reference sentences @xcite . reference-based automatic evaluation methods, such as bleu @xcite and bleurt @xcite , have contributed to research and development of machine translation; however, end-users of machine translation systems unlikely have such reference translations. hence, the development of qe methods that correlate well with human evaluation is practically important. supervised qe models @xcite @xcite based on pre-trained multilingual sentence encoders @xcite have been actively proposed in the qe competitions @xcite . however, these models require bilingual sentence pairs with manually labeled translation quality scores for fine-tuning. creating such a qe dataset is expensive because it requires annotators who are fluent in both of source and target languages. therefore, supervised qe models are limited to several major language pairs included in the competitions. in contrast, unsupervised qe allows quality estimation without human-assessed machine translation outputs. instead of the annotated outputs, unsupervised qe utilizes widely available parallel corpora. multilingual sentence encoders @xcite @xcite are promising for developing unsupervised qe models; however, their sentence embeddings are dominated by language-specific information. due to this characteristic, these sentence embeddings form clusters by language rather than by meaning, which hinders precise estimation of semantic similarity across languages @xcite . to address this problem, dream @xcite disentangles sentence embeddings to meaning and language embeddings. it conducts self-supervised learning using parallel sentence pairs in bilingual corpora as positive examples and random pairs as negative examples; meaning embeddings of positive pairs should be close while those of negative pairs should be distant. however, dream lacks direct supervision to eliminate language-specific information from the meaning embeddings and its architecture is complex. we improve dream by introducing an adversarial training that attempts to remove languagespecific information from the meaning embeddings. 1 our adversarial training eliminates the random pairs that dream needs, which results in a simpler architecture and lighter computational costs for training. experimental results on the wmt20 qe task @xcite revealed that our method achieved higher correlations with human scores than previous unsupervised qe models based on multilingual sentence encoders. com-source embedding target embedding reconstruction loss cross reconstruction loss cross reconstruction loss reconstruction loss source embedding target embedding language embedding loss source embedding 𝑀𝐿𝑃 𝑀 language embedding meaning embedding 𝑀𝐿𝑃 𝐿 𝑀𝐿𝑃 𝐷 adversarial loss target embedding 𝑀𝐿𝑃 𝐿 language embedding meaning embedding 𝑀𝐿𝑃 𝑀 𝑀𝐿𝑃 𝐷 adversarial loss pared to other approaches independent of multilingual sentence encoders @xcite , our method showed higher correlations in low-resource language pairs. ## proposed method our model is an autoencoder comprising two multilayer perceptrons, mlp m and mlp l , trained with bilingual corpora as shown in figure 1 . the former is responsible for extracting meaning and the latter for extracting language-specific information. these outputs are summed to reconstruct the input sentence embedding. we train these mlps using multilingual-multitask learning with the following four loss functions. ## summary and future work we introduced adversarial training to disentangle meaning embeddings from sentence representations of multilingual sentence encoders for unsupervised qe. our method consistently improves the performance of a state-of-the-art multilingual sentence encoder. our future work includes exploring ways to utilize language-specific embeddings for qe in terms of fluency of sentences. combined with the present method of assessing the adequacy of sentences, a better qe may be achieved. we will also apply our method for disentangling styles and meanings of sentences for the style-transfer research.