Multi-Source Text Classification for Multilingual Sentence Encoder with Machine Translation (2024)

Abstract: To reduce the cost of training models for each language for developers of natural language processing applications, pre-trained multilingual sentence encoders are promising.However, since training corpora for such multilingual sentence encoders contain only a small amount of text in languages other than English, they suffer from performance degradation for non-English languages.To improve the performance of pre-trained multilingual sentence encoders for non-English languages, we propose a method of machine translating a source sentence into English and then inputting it together with the source sentence in a multi-source manner.Experimental results on sentiment analysis and topic classification tasks in Japanese revealed the effectiveness of the proposed method.

## introduction fine-tuning of pre-trained sentence encoders @xcite has been remarkably successful in a variety of nlp (natural language processing) application tasks @xcite . pre-trained sentence encoders have developed in the direction not only of higher accuracy @xcite and efficiency @xcite , but also of multilingualization, with pre-trained multilingual sentence encoders such as mbert and xlm-r @xcite , which are capable of handling 100 languages, being widely used @xcite . since it is costly for developers to train models for each language, these multilingual sentence encoders are promising for the efficient multilingual deployment of nlp applications. however, the training data for existing multilingual sentence encoders is dominated by english texts, with only a few percent in other languages. to address this issue, we propose a method to improve the performance of multilingual sentence encoders in non-english languages by exploiting english, which is rich in the training data of multilingual sentence encoders. our proposed method combines a non-english source sentence and its machine translation into english in a multi-source manner for input to a multilingual sentence encoder. we expect that utilizing english translations gives multilingual sentence encoders the benefit of large-scale pre-training. although machine translation may contain translation errors, multi-source modeling with the source sentence can mitigate the negative effects of semantic changes. experimental results on sentiment polarity classification of japanese sns (social networking service) posts foot_1 @xcite and topic classification of japanese news titles foot_2 reveal the effectiveness of our multisource modeling. in addition, our detailed analysis revealed that the performance of the proposed method is insensitive to differences in machine translation quality, that the proposed method is also effective for non-english languages other than japanese, and that the proposed method is effective independent of the training corpus size. ## proposed method to improve the performance of pre-trained multilingual sentence encoders, we propose a method to transform source texts into synonymous expressions that perform well for the model. in this study, we assume that expressions that occur frequently in the pre-trained corpus are easy for the model to process, and machine translate lower-frequency non-english sentences into higher-frequency english sentences, which are then input to a multilingual sentence encoder. to reduce the effect of noise during machine translation, we employ multi-source modeling @xcite , in which sentences before and after the machine translation are concatenated and input. an overview of the proposed method is shown in figure1. we target non-english languages and assume situations where machine translation models from the target language to english are available. first, the given source sentence is machinetranslated, and the sequence "[cls] source sentence [sep] english translation [sep]" is input to the multilingual sentence encoder. as shown in ## experiments we evaluate the effectiveness of the proposed method on japanese text classification tasks for four pre-trained multilingual sentence encoders. ## conclusion this study proposed a multi-source input method that uses machine translation of source texts and a combination of source and english translations to improve the performance of pre-trained multilingual sentence encoders in languages other than english, aiming at the efficient deployment of natural language processing services in multiple languages. the proposed method benefits from the large amount of pre-trained data in english and is expected to improve the performance of multilingual sentence encoders. evaluation experiments on sentiment polarity classification of sns posts and topic classification tasks of news articles in japanese showed that the proposed method with english translations can improve the classification performance compared to the baseline method with only the source sentences. the proposed method with both the source and target sentences consistently improves the performance of the multilingual sentence encoder, while the performance of the method with only the target sentences deteriorates because the machinetranslated sentences can contain translation errors. 5 230