Sentiment-based Candidate Selection for NMT (2021)

Abstract: The explosion of user-generated content (UGC)—e.g. social media posts and comments and and reviews—has motivated the development of NLP applications tailored to these types of informal texts. Prevalent among these applications have been sentiment analysis and machine translation (MT). Grounded in the observation that UGC features highly idiomatic and sentiment-charged language and we propose a decoder-side approach that incorporates automatic sentiment scoring into the MT candidate selection process. We train monolingual sentiment classifiers in English and Spanish and in addition to a multilingual sentiment model and by fine-tuning BERT and XLM-RoBERTa. Using n-best candidates generated by a baseline MT model with beam search and we select the candidate that minimizes the absolute difference between the sentiment score of the source sentence and that of the translation and and perform two human evaluations to assess the produced translations. Unlike previous work and we select this minimally divergent translation by considering the sentiment scores of the source sentence and translation on a continuous interval and rather than using e.g. binary classification and allowing for more fine-grained selection of translation candidates. The results of human evaluations show that and in comparison to the open-source MT baseline model on top of which our sentiment-based pipeline is built and our pipeline produces more accurate translations of colloquial and sentiment-heavy source texts.

## introduction the web, widespread internet access, and social media have transformed the way people create, consume, and share content, resulting in the proliferation of user-generated content (ugc). ugc-such as social media posts, comments, and reviews-has proven to be of paramount importance both for users and organizations/institutions @xcite . as users enjoy the freedoms of sharing their opinions in this relatively unconstrained environment, corporations can analyze user sentiments and extract insights for their decision-making processes, (timoshenko and hauser, 2019) or translate ugc to other languages to widen the company's scope and impact. for example, @xcite shows that translating ugc between certain language pairs has beneficial effects on the overall ratings customers gave to attractions and shows on tri-padvisor, while the absence of translation hurts ratings. however, translating ugc comes with its own challenges that differ from those of translating well-formed documents like news articles. ugc is shorter and noisier, characterized by idiomatic and colloquial expressions @xcite . translating idiomatic expressions is hard, as they often convey figurative meaning that cannot be reconstructed from the meaning of their parts @xcite , and remains one of the open challenges in machine translation (mt) @xcite . idiomatic expressions, however, typically carry an additional property: they imply an affective stance rather than a neutral one @xcite . the sentiment of an idiomatic expression, therefore, can be a useful signal for translation. in this paper, we hypothesize that a good translation of an idiomatic text, such as those prevalent in ugc, should be one that retains its underlying sentiment, and explore the use of textual sentiment analysis to improve translations. our motivation behind adding sentiment analysis model(s) to the nmt pipeline are several. first, with the sorts of texts prevalent in ugc (namely, idiomatic, sentiment-charged ones), the sentiment of a translated text is often arguably as important as the quality of the translation in other respects, such as adequacy, fluency, grammatical correctness, etc. second, while a sentiment classifier can be trained particularly well to analyze the sentiment of various texts-including idiomatic expressions @xcite -these idiomatic texts may be difficult for even state-of-the-art (sota) mt systems to handle consistently. this can be due to problems such as literal translation of figurative speech, but also to less obvious errors such as truncation (i.e. failing to translate crucial parts of the source sentence). our assumption however, is that with open-source translation systems such as opus mt foot_1 , the correct translation of a sentiment-laden, idiomatic text often lies somewhere lower among the predictions of the mt system, and that the sentiment analysis model can help signal the right translation by re-ranking candidates based on sentiment. our contributions are as follows: ## related work several papers in recent years have addressed the incorporation of sentiment into the mt process. perhaps the earliest of these is @xcite , which examined the effects of using honorific marking in training data to help mt systems pick up on the t-v distinction (e.g. informal tu vs. formal vous in french) that serves to convey formality or familiarity. @xcite used sentiment-labeled sentences containing one of a fixed set of sentiment-ambiguous words, as well as valence-sensitive word embeddings for these words, to train models such that users could input the desired sentiment at translation time and receive the translation with the appropriate valence. lastly, @xcite experimented with training sentimentisolated mt models-that is, mt models trained on only texts that had been pre-categorized into a set number of sentiment classes i.e., positive-only texts or negative-only texts. our approach is novel in using sentiment to re-rank candidate translations of ugc in an mt pipeline and in using precise sentiment scores rather than simple polarity matching to aid the translation process. in terms of sentiment analysis models of non-english languages, @xcite experimented with using an rnn-based english sentiment model to analyze the sentiment of texts translated into english from other languages, while @xcite used smt to generate sentiment training corpora in non-english languages. @xcite provides an overview and comparison of various techniques used to tackle multilingual sentiment analysis. as for mt candidate re-ranking, hadj @xcite provides an extensive overview of the various features and tools that have been used to aid in the candidate selection process, and also proposes a feature ensemble approach that doesn't rely on external nlp tools. others who have used candidate selection or re-ranking to improve mt performance include @xcite and @xcite . to the best of our knowledge, however, no previous re-ranking methods have used sentiment for re-ranking despite findings that mt often alters sentiment, especially when ambiguous words or figurative language such as metaphors or idioms are present or when the translation exhibits incorrect word order @xcite . ## method: sentiment-based candidate selection we propose the use of two language-specific sentiment classifiers (which, as we will describe later in the paper, can be reduced to one multilingual sentiment model)-one applied to the input sentence in the source language and another to the candidate translation in the target language-to help an mt system select the candidate translation that diverges the least, in terms of sentiment, from the source sentence. using the baseline mt model described in section 3.2, we first generate @xmath0. we decided on 10 as our candidate number based on the fact that one can expect a relatively low drop off in translation quality with this parameter choice @xcite , while also maintaining a suitably high likelihood of getting variable translations. additionally, decoding simply becomes too slow in practice beyond a certain beam size. once our model generates the 10 candidate translations for a given input sentence, we use the sentiment classifier trained in the appropriate language to score the sentiment of both the input sentence and each of the translations in the interval [0, 1]. to compute the sentiment score s(x) for an input sentence x, we first compute a softmax over the array of logits returned by our sentiment model to get a probability distribution over all m possible classes (here, @xmath1, since we only used positive-and negative-labeled tweets). representing the negative and positive classes using the values 0 and 1, respectively, we define s(x) to be the expected value of the class conditioned on x, namely s(x) = m @xmath2, where c i is the ith class and v i is the value corresponding to that class. in our case, since we have only two classes and the negative class is represented with value 0, s(x) = p (positive class | x). after computing the sentiment scores, we take the absolute difference between the input sentence x's score and the candidate translation t i 's score for @xmath3, 2, ..., 10 to obtain the sentiment divergence of each candidate. we select the candidate translation that minimizes the sentiment divergence, namely @xmath4. our method of selecting a translation differs from previous works in our use of the proposed sentiment divergence, which takes into account the degree of the sentiment difference (and not just polarity difference) between the input sentence and the candidate translation. ## discussion our experimentation with the various mt models generated a number of interesting example cases concerning the translation of idiomatic language. for example, given the tweet "time warner road runner customer support here absolutely blows," the baseline mt gives a literal translation of the word "blows" as "pukulan" (literally, "hits") in indonesian; google translate gives a translation "hebat" ("awesome") that is opposite in sentiment to the idiomatic sense of the word "blows" ("sucks") in english; and our model gives a translation closest in meaning and sentiment to "blows," namely "kacau" (approx. "messed up" in indonesian). there are also cases where our model gives a translation that is closer in degree of sentiment than what google translate produces. given the source text "yo @apple fix your shitty imessage," google translate produces "yo @apple perbaiki imessage buruk anda" ("yo @apple fix your bad imessage"), which has roughly the same polarity as the source tweet. by contrast, our proposed model produces "yo @apple perbaiki imessage menyebalkan anda," using the word "menyebalkan" ("annoying") instead of "buruk," which conveys a closer sentiment to "shitty" than simply "bad". the evaluators of the english-spanish translations provided us with rich qualitative commentary as well. for the sentence "just broke my 3rd charger of the month. get your shit together @apple," which is translated by the professional translator as "se acaba de romper mi tercer cargador del mes. sean más eficientes @apple," one evaluator acutely notes that "the expression 'get your shit together' was translated in a more formal way (it loses the vulgarism). i would have translated it as 'poneos las pilas, joder' to keep the same sentiment. we could say that this translation has a different diaphasic variation than the source text." this demonstrates that sentiment preservation is a problem not only for nmt systems, but for human translators as well. there are also problems attributed to challenges in machine translating informal texts. acronyms such as "tbh" and "smh" made for another interesting case, as they weren't translated by any of the mt models for any language pairing, despite their common occurrence in ugc. the same evaluator also notes that "the acronym 'tbh' was not translated" in the sentence "@apple tbh annoyed with apple's shit at the moment," and says "this acronym is important for the sentiment because it expresses the modality of the speaker." in another example, we see our sentiment-sensitive pipeline helping the baseline distinguish between such a semantically fine-grained distinction as that between "hope" and "wish": the baseline translates the sentence "@iberia ojalá que encuentres pronto tu equipaje!!" as "@iberia i wish you'd find your luggage soon!!," while our pipeline correctly chooses "@iberia i hope you will find your luggage soon!!." we observe similar issues contribute to sentiment divergence in spanish and indonesian despite the fact that these are typologically disparate languages with different amounts of training data in the mt system. in terms of automatic mt evaluation, our method improves over the baseline for the spanish tweets on which the human evaluation was conducted. this result supports the efficacy of our model in the context of highly-idiomatic, affective ugc. and while google translate still outperforms the baseline and our pipeline in terms of bleu score on tatoeba (for both languages) and the tweets (for which only spanish had a gold-standard benchmark)-given that the baseline model that we built our pipeline on is not sota-our pipeline can be added to any mt system and can also improve sota mt for ugc. furthermore, our approach also lends itself to many practical scenarios, e.g. companies who are interested in producing sentiment-preserving translations of large bodies of ugc but who lack the sufficient funds to use a subscription api like google cloud translation. in these contexts, it may be beneficial-or even necessary-to improve free, open-source software in a way that is tailored to one's particular use case (thus the idea of "customized mt" that many companies now offer), instead of opting for the sota but more costly software. more generally, since our approach shows that we can improve performance of an mt model for a particular use case i.e., ugc translation using signals beyond translation data that is relevant for the task at hand i.e., sentiment, it will be interesting to explore other signals that are relevant for improving mt performance in other use cases. it will also be interesting to explore the addition of these signals in a pipeline (our current method), as implicit feedback such as in @xcite , or as explicit feedback in an end-to-end mt model for example, as additional loss terms in supervised @xcite , weakly-supervised @xcite , or unsupervised @xcite mt models. beyond the potential engineering contribution for low-resource, budget-constrained settings, our experiments also offer rich qualitative insights regarding the causes of sentiment change in (machine) translation, opening up avenues to more disciplined efforts in mitigating and exploring these problems. ## conclusion in this paper, we use several distinct sentiment classifiers trained on twitter data to help machine translation models select sentiment-preserving translations of highly idiomatic source texts. diverging from previous works, we use continuous (rather than binary or categorical) sentiment scores to select minimally divergent translations, and we test the performance of our pipeline with automated and human evaluations for english-spanish and english-indonesian translations. furthermore, we implement our sentiment-aware translation pipeline on free, open-source mt models available on hugging face 8 . although many of these models are non-sota, our choice to use them represents a real-world scenario: many users and companies do not have the resources or budget to subscribe to a sota translation api or train their own mt model from scratch. our pipeline poses a lightweight solution for getting more with less, in a somewhat niche yet ubiquitous translation context (social media posts). in future work, we would like to evaluate the effect of sentiment classifier performance on the downstream mt results, including the effects of classifier architecture, the number of sentiment categories and their distribution in the training data (e.g., ugcs with more informal words may contain more affective texts), etc. we would also like to investigate how continuous sentiment scoring compares with binary or categorical scoring for this task, using a larger evaluation set for idiomatic texts (e.g. in english @xcite or constructed in other languages @xcite ), or from a dataset we create ourselves. finally, further work should establish benchmarks and put forth improvements for cross-lingual sentiment classification (i.e. the extent to which sentences that are translations of each other are assigned similar sentiments)-including the problem of zero-shot transfer-adding onto recent work in cross-lingual performance benchmarks @xcite .