Are the Multilingual Models Better? Improving C zech Sentiment with Transformers (2021)

Abstract: In this paper, we aim at improving Czech sentiment with transformer-based models and their multilingual versions. More concretely, we study the task of polarity detection for the Czech language on three sentiment polarity datasets. We fine-tune and perform experiments with five multilingual and three monolingual models. We compare the monolingual and multilingual models’ performance, including comparison with the older approach based on recurrent neural networks. Furthermore, we test the multilingual models and their ability to transfer knowledge from English to Czech (and vice versa) with zero-shot cross-lingual classification. Our experiments show that the huge multilingual models can overcome the performance of the monolingual models. They are also able to detect polarity in another language without any training data, with performance not worse than 4.4 % compared to state-of-the-art monolingual trained models. Moreover, we achieved new state-of-the-art results on all three datasets.

## introduction in recent years, bert-like models @xcite based on the transformer architecture @xcite and generalized language models brought a significant improvement in performance in almost any nlp task @xcite , especially in english. despite this fact, not much work has been recently done in sentiment analysis for the czech language with the latest transformer models. we partly fill this gap by focusing on the sentiment classification task, also known as polarity detection. polarity detection is a classification task where the goal is to assign a sentiment polarity to a given text. the positive, negative and neutral classes are usually used as the polarity labels. the polarity can also be defined with a different number of labels, i.e., fine-grained sentiment analysis @xcite . the models based on bert were almost exclusively trained for english, limiting their usage to other languages. recently, however, their cross-lingual adaptions like mbert @xcite , mt5 @xcite , xlm @xcite or xlm-r @xcite emerged along with other non-english monolingual versions, for example, czech @xcite , french @xcite , arabic @xcite , romanian @xcite , @xcite or finnish @xcite . our motivation is to reveal the performance limits of the current sota transformer-based models on the czech polarity detection task, check the ability of the multilingual models to transfer knowledge between languages and unify the procedure and data that enable the correct future evaluation of this task. in this paper, we focus on the task of polarity detection applied on czech text by comparing the performance of seven pre-trained transformer-based models (both monolingual and multilingual) on three czech datasets. we fine-tune each model on each dataset and we provide a comprehensive survey of their performance. our experiments show the effectiveness of the transformer models that significantly outperform the older approaches based on recurrent neural networks. we observe that the monolingual models can be notably outperformed by the multilingual models, but only by those with much more parameters. moreover, we achieve new state-of-the-art results on all three evaluated datasets. we are also interested in the ability of the multilingual models to transfer knowledge between languages and its usability for polarity detection. thus, we perform zero-shot cross-lingual classification, fine-tune four cross-lingual transformer-based models on the english dataset and then test the models on czech data. we also perform the same experiment in the reverse direction, i.e., from czech to english. the results reveal that the xlm-r-large model (fine-tuned solely on english) can achieve very competitive results that are only about 4 % worse than the sota model fine-tuned by us on czech data. to the best of our knowledge, this is the first paper that performs zero-shot cross-lingual polarity detection for the czech language. we also noticed that the comparison with the previous works is rather problematic and thus, we provide a split for all czech datasets that allows comparing future works much easier. our code and pre-trained models are publicly available foot_0 . our main contributions are the following: 1) we provide the comprehensive performance comparison of the currently available transformer-based models for the czech language on the polarity detection task along with the models' optimal settings. 2) we test the ability of the multilingual models to transfer knowledge between czech and english. 3) we release all the fine-tuned models and code freely for research purposes and we provide a data split that allows future comparison and evaluation. furthermore, we achieved new state-of-the-art results for all three evaluated datasets. ## related work the previous approaches @xcite @xcite @xcite for english polarity detection and other related tasks mostly relied on transfer learning and pre-trained word embeddings such as word2vec @xcite and fasttext @xcite in combinations with convolutional neural networks (cnn) or long short-term memory (lstm) @xcite , eventually in conjunction with the modified attention mechanism @xcite @xcite . furthermore, the new contextualized word representations such as cove @xcite or elmo @xcite and pre-trained language model ulmfit @xcite were successfully applied to the polarity detection. finally, the latest transformerbased models like bert @xcite , gpt @xcite , roberta @xcite or t5 @xcite ) that are all in general trained on language modeling tasks proved their performance superiority for english over all previous approaches, for example in @xcite . these models are pre-trained on a modified language modeling tasks with a huge amount of unlabeled data. in the end, they are fine-tuned for a specific downstream task. the initial works on czech polarity detection and sentiment analysis usually used lexical features @xcite or bag-of-words text representations along with the naive bayes or logistic regression classifiers @xcite or a combination of supervised and unsupervised approach @xcite . @xcite applied cnn using the architecture from @xcite and the lstm neural network to all three datasets that we use in this paper. another usage of lstm neural network with the self-attention mechanism @xcite can be found in @xcite . similarly, @xcite tried to use curriculum learning with cnn and lstm. @xcite pre-trained a bert-based model for polarity detection with an improved pooling layer and distillation of knowledge technique. the most recent application of the transformer model is in @xcite . the authors created a bert model for czech and, as one of the evaluation tasks, they performed polarity detection on the fb and csfd datasets. to the best of our knowledge, there are no previous works that focus on the zero-shot cross-lingual polarity detection task in the czech language. the recent related work can be found in @xcite , where the authors use the neural machine translation encoder-based model and english data to perform zero-shot cross-lingual sentiment classification on french. in @xcite the authors performed the zero-shot classification from slovene to croatian. another related work can be found in @xcite . ## data to the best of our knowledge, there are three czech publicly available datasets for the polarity detection task: (1) movie review dataset (csfd), (2) facebook dataset (fb) and (3) product review dataset (mallcz), all of them come from @xcite and each text sample is annotated with one of three foot_1 labels, i.e., positive, neutral and negative, see since there is no official partitioning for the czech datasets, we split them into training, development and testing parts with the same class distribution for each part as it is in the original dataset, see ## models description we performed exhaustive experiments with transformed-based models and in order to compare them with the previous works, we also implemented the older models (baseline models) that include the logistic regression classifier and the bilstm neural network. ## experiments & results we perform two types of experiments, i.e., monolingual and cross-lingual. in monolingual experiments, we fine-tune and evaluate the transformer models for each dataset separately on three-class (positive, negative and neutral) and two-class (positive and negative) sentiment analysis. we also implemented the logistic regression (lrc) and lstm baseline models and we compare the results with the existing works. in cross-lingual experiments, we test the ability of four multilingual transformer-based models to transfer knowledge between english and czech. we run the multilingual models only on the twoclass datasets (positive and negative). we fine-tune either on english (imdb) or czech (csfd), and then we evaluate on the other language. thus we perform the zero-shot cross-lingual classification. we decided to use the imdb and csfd dataset because they are from the same domain i.e., movie reviews. each experiment 10 was performed at least five times and we report the results using the macro f 1 score. ## conclusion in this work, we evaluated the performance of available transformer-based models for the czech language on the task of polarity detection. we compared the performance of the monolingual and multilingual models and we showed that the large xlm-r-large model can outperform the monolingual czert-b model. the older approach based on recurrent neural networks is surpassed by a very large margin by the transformers. moreover, we achieved new state-of-the-art results on all three czech polarity detection datasets. we performed zero-shot cross-lingual polarity detection from english to czech (and vice versa) with four multilingual models. we showed that the xlm-r-large is able to detect polarity in another language without any labeled data. the model performs no worse than 4.4 % in comparison to our new state-of-the-art monolingual model. to the best of our knowledge, this is the first work that aims at cross-lingual polarity detection in czech. our code and pre-trained models are publicly available. in the future work, we intend to perform a deep error analysis to find in which cases the current models fail and compare approaches that use the linear cross-lingual transformations @xcite that explicitly map semantic spaces into one shared space. the second step in the cross-lingual settings is to employ more than two languages and utilize the models for different domains.