On the logistical difficulties and findings of Jopara Sentiment Analysis (2021)

Abstract: This paper addresses the problem of sentiment analysis for Jopara, a code-switching language between Guarani and Spanish. We first collect a corpus of Guarani-dominant tweets and discuss on the difficulties of finding quality data for even relatively easy-to-annotate tasks, such as sentiment analysis. Then, we train a set of neural models, including pre-trained language models, and explore whether they perform better than traditional machine learning ones in this low-resource setup. Transformer architectures obtain the best results, despite not considering Guarani during pre-training, but traditional machine learning models perform close due to the low-resource nature of the problem.

## introduction indigenous languages have been often marginalized, an issue that is reflected when it comes to design natural language processing (nlp) applications, where they have been barely studied @xcite . one of the places where this is greatly noticed is latin america, where the dominant languages (spanish and portuguese) coexist together with hundreds of indigenous languages such as guarani, quechua, nahuatl or aymara. in this context, the guarani language plays a particular role. it is an official language in paraguay and bolivia. besides, it is spoken in other regions, e.g. corrientes (argentina) or mato grosso do sul (brazil), alongside with their official languages. overall, it has about 8m speakers. its coexistence with other languages, mostly spanish, has contributed to its use in code-switching setups @xcite @xcite ) and led to jopara, a code-switching between guarani and spanish, with flavours of portuguese and english @xcite . despite its official status, there is still few nlp resources developed for guarani and jopara. @xcite developed a parallel spanish-english-guarani corpus for machine translation. similarly, @xcite developed a guarani-spanish parallel corpus aligned at sentence-level. there are also a few online dictionaries and translators from guarani to spanish and other languages. 1 beyond machine translation, @xcite released a corpus for guarani speech recognition that was collected from the web; and rudnick (2018) presented a system for cross-lingual word sense disambiguation from spanish to guarani and quechua languages. there also are a few resources for pos-tagging and morphological analysis of guarani, such as the work by @xcite and apertium; foot_0 and also for parsing, more specifically for the mbyá guarani variety @xcite , under the universal dependencies framework. in the context of sentiment analysis (sa; @xcite , and more particularly classifying the polarity of a text as positive, negative or neutral, we are not aware of any previous work; with the exception of @xcite . they presented a sentiment corpus for the paraguayan spanish dialect, which also includes words in english and portuguese. however, there were few, albeit relevant, words of guarani (70) and jopara foot_1 (10), in comparison to the amount of the ones in spanish (3, 802) @xcite . overall, sa has focused on rich-resource languages for which data is easy to find, even when it comes to code-switching setups @xcite , maybe with a few exceptions such as english code-switched with languages found in india @xcite @xcite . in this context, although some previous work has developed multilingual lexicons and methods @xcite ; for languages such as guarani and other low-resource cases (where web text is scarce), it is hard to develop nlp corpora and systems. contribution our contribution is twofold. first, we collect a corpus for polarity classification of jopara tweets, which mixes guarani and spanish languages, being the former the dominating language in the corpus. we also discuss on the difficulties that we had to face when creating such resource, such as finding enough twitter data that shows sentiment and contains a significant amount of guarani terms. second, we train a set of neural encoders and also traditional machine learning models, in order to have a better understand of how old versus new models perform in this low-resource setup, where the amount of data matters. ## josa: the jopara sentiment analysis dataset in what follows, we describe our attempts to collect jopara tweets. note that ideally we are interested in tweets that are as guarani as possible. however, guarani is intertwined with spanish, and thus we have focused on jopara, aiming for guaranidominant tweets, in contrast to @xcite . we found interesting to report failed attempts to collect such data, since the proposed methods would most likely work to collect data in rich resource languages. we hope this can be helpful for other researchers interested in developing datasets for low-resource languages in web environments. in this line, twitter does not allow to automatically crawl guarani tweets, since it is not included in its language identification tool. to overcome this, we considered two alternatives: (i) using a set of guarani keywords ( §2.1), and (ii) scrapping twitter accounts that mostly tweet in guarani ( §2.2). ## models due to the low-resource setup, we run neural models and pre-trained language models, but also other machine learning models, such as complement naïve bayes (cnb) and support vector machines (svms) @xcite , since they are less data hungry, and could help shed some light about the real effectiveness of neural models on jopara texts. in all cases, the selection of the hyperparameters was done over a small grid search based on the dev set. we report the details in the appendix a.2. ## experiments reproducibility the baselines and tweet ids 12 are available at https://github.com/ mmaguero/josa-corpus . we run experiments for the unbalanced and balanced versions of josa, evaluating the macroaccuracy (to mitigate the impact of the neutral class in the unbalanced setup). pre-trained embeddings are represented with a prefix together with their language iso 639-1 code (except for m: multilingual). with respect to the experiments with cnns and bilstms encoders, we tested different combinations using character representations, which output is first concatenated to a second external word vector (as explained in §3), and then fed to the encoder. among those, the model that used a character-level cnn and a word-level bilstm encoder obtained the best results. still, the difference with respect to traditional machine learning models is small. we hypothesize this might be due to the low-resource nature of the task. finally, the pre-trained language models that use transformers architectures, in particular beto, obtain overall the best results, despite not being pre-trained on guarani. we believe this is partly due to the presence of spanish words in the corpora and also to the cross-lingual abilities that bert model might explode, independently of the amount of word overlap @xcite . error analysis on the balanced version of josa figure 1 shows the confusion matrices for a representative model of each machine learning family (based on the accuracy): (i) cnb, (ii) the best bilstm-based model (cnn-bilstm), and (iii) spanish bert (beto). there seems to be different tendencies in the miss-classifications that different models make. for instance, cnb tends to over-classify tweets as negative, while both deep learning models show a more controlled behaviour when predicting this class. although for the three models neutral tweets seem to be the easiest to identify, both deep learning models are clearly better at it. finally, when it comes to identify positive tweets, beto seems to show the overall best performance. these different tendencies indicate that an ensemble method could be beneficial for lowresource setups such as the ones that josa represent, since the models seem to be complementary to certain extent. in this context, we would like to explore this line of work in the future, following previous studies such as jhanwar and das (2018), which showed the benefits of combining different machine learning models for hindi-english codeswitching sa. ## conclusion this paper explored sentiment analysis on jopara, a code-switching language that mixes guarani and spanish. we collected the first guarani-dominant dataset for sentiment analysis, and described some of the challenges that we had to face to create a collection where there is a significant number of guarani terms. we then built several machine learning (naïve bayes, svms) and deep learning models (bilstms, cnns and bert-based models) to shed light about how they perform on this particular low-resource setup. overall, transformers models obtain the best results, even if they did not consider guarani during pre-training. this poses interesting questions for future work such as how cross-lingual bert abilities @xcite can be exploited for this kind of setups, but also how to improve language-specific techniques that can help process low-resource languages efficiently.