Improving Sentiment Analysis over non- E nglish Tweets using Multilingual Transformers and Automatic Translation for Data-Augmentation (2020)

Abstract: Tweets are specific text data when compared to general text. Although sentiment analysis over tweets has become very popular in the last decade for English, it is still difficult to find huge annotated corpora for non-English languages. The recent rise of the transformer models in Natural Language Processing allows to achieve unparalleled performances in many tasks, but these models need a consequent quantity of text to adapt to the tweet domain. We propose the use of a multilingual transformer model, that we pre-train over English tweets on which we apply data-augmentation using automatic translation to adapt the model to non-English languages. Our experiments in French, Spanish, German and Italian suggest that the proposed technique is an efficient way to improve the results of the transformers over small corpora of tweets in a non-English language.

## introduction monitoring social media at the scale of a continent, like europe, requires to process multiple languages. data from twitter is noisy and can drastically change in term of words distribution in one language when compared with general texts. it becomes even more challenging when tackling a multilingual task. in accordance with the rgpd, it is impossible to make available tweets that have been deleted by their authors. this makes it more difficult to find tweet corpora annotated in sentiment. the semeval challenges assure a big database of tweets in english, with a total of more than 62k examples annotated at the level of the tweet @xcite . for other languages, this is more complicated. bidirectional transformers like bert @xcite revolutionized the world of natural language processing (nlp). even when pre-trained over text from general domain, these models need a substantial amount of data to adapt to a domain where the syntax is different, like twitter @xcite . this paper presents the experiments carried out on several datasets of tweets in five different languages: english, french, spanish, german and italian. the general idea is pretty simple: instead of using a monolingual model, we chose to use a multilingual model that we can train over a large dataset of english tweets, over the original non-english tweets and over their automatic translations. we chose the multilingual transformer model xlm-roberta from @xcite with a dataaugmentation technique using machine translation. we investigated the effects of pre-training with english data and data-augmentation. we also compared performances of multilingual models against their monolingual french @xcite and english @xcite counterparts and found interesting improvements. ## state of the art the related works section of this paper is shared between multilingual sentiment analysis, dataaugmentation and sentiment analysis over tweets. we can count several challenges tackling sentiment analysis over tweets like semeval @xcite in english, tass @xcite in spanish or deft @xcite in french. lang. tweet english i'd rather dump gasoline all over myself and run into a burning building than use excel. french je préférerais jeter de l'essence partout et tomber dans un immeuble en feu plutôt que d'utiliser excel. german ich würde lieber benzin auf mich werfen und in ein brennendes gebäude laufen, als excel zu benutzen. spanish prefiero tirar gasolina sobre mí mismo y correr hacia un edificio en llamas que usar excel. italian preferirei buttarmi la benzina addosso e correre in un edificio in fiamme piuttosto che usare excel. for the last years, the neural networks are ruling the sentiment analysis over tweets. @xcite won the sentiment polarity subtask of the semeval-2017 challenge @xcite using a neural network approach. @xcite improve the state-of-the-art with a different incorporation of the emojis: they use the descriptions instead of their unicodes before a blstm with word embeddings. finally, @xcite proposed a bert model with the roberta pre-training procedure @xcite over 850m english tweets. this model gives state-of-the-art results on the semeval-2017 dataset, but like all the other models, it is not adapted to multilingual data. although data-augmentation is not as developed for textual data as it is for images, there are ways to apply it to text @xcite . @xcite augment their datasets using back-translation for pairing sentences for a mt task. @xcite uses language models in context in order to create new plausible sentences. @xcite use data-augmentation for mt, by automatically translating at the level of words using plausible substitutions in order to create a new sentence. the work that is closest to the one we are presenting in this article remains that of @xcite who tackle a sentiment analysis task over multilingual tweets. they show that use of multilingual, machine-translated data can help to better distinguish relevant features for sentiment classification, using svm models with bag-of-n-grams. we distinguish from this work by using real datasets for testing instead of artificially created test sets made of translated tweets re-edited by humans. ## proposed method the method we propose is very simple. it basically consists in using a multilingual model instead of a monolingual model, pre-trained it over available annotated english tweets, combined with a dataaugmentation techniquethat is based on mt. with this augmentation, we have each tweet in five examples, in five different languages. the languages that we use for the tweets are the same languages that the datasets we used to test the tweets: french, english, german, spanish and italian. ## conclusion we presented a technique that helps to improve the results of a sentiment analysis system over nonenglish tweets. we use multilingual model that is able to process external english data available in big quantities to pre-train the model, and machine translation to augment the dataset. this technique is simple and yet allows to take advantage of the multilingual models for non-english tweet datasets of limited size.