A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical A rabic UGT (2022)

Abstract: In the online world, Machine Translation (MT) systems are extensively used to translate User-Generated Text (UGT) such as reviews, tweets, and social media posts, where the main message is often the author’s positive or negative attitude towards the topic of the text. However, MT systems still lack accuracy in some low-resource languages and sometimes make critical translation errors that completely flip the sentiment polarity of the target word or phrase and hence delivers a wrong affect message. This is particularly noticeable in texts that do not follow common lexico-grammatical standards such as the dialectical Arabic (DA) used on online platforms. In this research, we aim to improve the translation of sentiment in UGT written in the dialectical versions of the Arabic language to English. Given the scarcity of gold-standard parallel data for DA-EN in the UGT domain, we introduce a semi-supervised approach that exploits both monolingual and parallel data for training an NMT system initialised by a cross-lingual language model trained with supervised and unsupervised modeling objectives. We assess the accuracy of sentiment translation by our proposed system through a numerical ‘sentiment-closeness’ measure as well as human evaluation. We will show that our semi-supervised MT system can significantly help with correcting sentiment errors detected in the online translation of dialectical Arabic UGT.

## introduction incorporating automatic translation tools by websites such as twitter, amazon.com and booking.com has become common practice to cater for their multilingual users. in this context, sentiment preservation is of great importance because deci-sions about purchasing a product or service, as well as analysis of public trends, are based on accurate translation of the user's affect message. arabic ugt constitutes a significant challenge for mt systems because it is commonly a mix of dialectical arabic (da) and modern standard arabic (msa) which differ significantly on the lexicogrammatical level. research has shown that the code-switching between da and msa by online users can lead to a serious mistranslation of sentiment for several reasons @xcite . first, there are lexical and structural differences between the two versions of the arabic language which cause confusion to mt systems in choosing the correct sentiment-carrying word. on the lexical level, there are polysemous words used in both msa and da which can have exact opposite sentiment poles. to give one example, the word ' ' means 'rigid' in msa, but in da, within the ugt domain, it often means 'great or awesome'. hence, we find the positive goodreads review ' ' (a very good book) 1 is mistranslated by the online mt tool into 'a very rigid book', incorrectly reflecting a negative sentiment. the same word, however, in another book review written in msa -' ' -is correctly translated as 'the author's way of narrating events is very rigid', rightly reflecting the dissatisfaction of the author. second, the arabic writing system does not have letters for short vowels; instead short vowels are realised as diacritic symbols on or below letters. ugt commonly lacks diacritics and hence it of-ten contains words spelled alike in msa and da but different in meaning due to different pronunciation. an example of these homographs is in the da tweet foot_1 ' ' where the noun ' ' commonly means 'fraud' in da with the diacritic 'fatha' (a short /a/ sound) on the first letter; the tweet should read 'enough of the fraud'. the online mt system flips the negative polarity as it mistakes this word with its common homograph in msa meaning 'monument', pronounced with 'damma' (a short /u/ sound) on its first and second letters. the mistranslation of the homograph produces a neutral statement, 'enough monument', which completely misses the negative polarity of the source. the third problem is that the way sentiment is expressed by the da used in ugt is different than the structured da data that is commonly used to train da-en nmt systems (e.g. @xcite ; @xcite ; @xcite ; @xcite ; @xcite ). some of the main differences is that ugt typically contains profanity and aggressive words that are not to be found in the available dialectical data. moreover, the da used on online platforms such as twitter usually contains unusual orthography to express emotions or to obfuscate aggression and, at times, nuanced words that are understood only within context. a review of the literature shows that the authentic parallel datasets for da-en consist mainly of hand-crafted structured data which significantly differ from this type of noisy da used in ugt. on the other hand, there is a considerable number of large parallel msa-en datasets in various domains (e.g opus foot_2 open-source parallel msa-en datasets include un documents, tedx talks, subtitles, news commentary, etc.). since da in the ugt domain has peculiar qualities and since it differs on the lexico-grammatical level from standard arabic and, at times, same words can have opposite sentiment in the two versions, the freely available msa datasets are not optimal for translating sentiment in ugt written in a dialectical version. given the scarcity of any substantial goldstandard da-en data within the ugt domain, we propose to improve the transfer of sentiment in arabic ugt by training a semi-supervised nmt system where we leverage the relatively large goldstandard msa-en data with da monolingual data from the ugt domain. we take advantage of pretraining a cross-lingual language model with both a masked language modelling (mlm) objective and a translation language modelling (tlm) objective for creating a shared embedding space for english, msa and da. we show that initialising our nmt model with these cross-lingual pretrained word representations has a significant impact on the translation performance in general and on the transfer of sentiment in particular. in this research, therefore, we make the following contributions: to present our contributions, the paper is divided as follows: section 2 provides a summary of relevant approaches to supervised and unsupervised mt as well as research attempts for the translation of da. section 3 describes our semi-supervised nmt system set up and its requirements. section 4 presents the experiments we conducted on our compiled datasets as well as the assessment methods used to evaluate the improvement of sentiment translation in da ugt. finally, section 5 presents our conclusions on the different experiments and the limitations of the study. ## related work the earliest attempt to solve the problem of translating da has been introduced by @xcite . they created the largest existing parallel data for da to english which is relied upon in most mt research for da. the dataset consists of around 250k parallel sentences. they used mechanical turk to translate sentences from da to en. most of the da is in the levantine and egyptian dialects, but none of the texts used belong to the ugt domain. they show that when translating the dialectical test sets, the da-en mt system performs 6.3 and 7.0 bleu points higher than an mt system trained on a 150m-word msa-en parallel corpus. another approach to solve the data scarcity problem was introduced by @xcite who propose pivoting to msa instead of directly translating from da to en. they transform da sentences into msa by a large number of hand-written morphosyntactic transfer rules. there have been other attempts to create da-en and da-msa parallel datasets such as the multidialectical mdc and madar datasets @xcite , the qca speech corpus @xcite , and the padic parallel corpus which includes five dialects and msa, but not english @xcite . these datasets, however, are relatively too small (max 14.7k parallel sentences) and differ considerably from the ugt domain. since the problem of da-en scarcity of data still exists up to the time of writing this research, the most recent attempts to improve the translation of da to english have focused either on augmenting the available datasets by bootstrapping techniques @xcite or on training with the large available msa datasets and fine-tuning on the smaller da datasets @xcite . a recent research line in mt which has been introduced to overcome the sparsity of gold-standard parallel data for low-resource languages is unsupervised mt which relies solely on monolingual data of the source and target languages in training @xcite @xcite . the key idea is to build a common latent space for two languages (or more) which can be used to reconstruct a sentence in a given language from a noisy version of it @xcite , or to obtain the translated sentence by using a back-translation procedure @xcite . the use of high quality cross-lingual word embeddings pretrained by state-of-the-art cross-lingual language models to initialise the unsupervised mt systems has recently contributed to a significant improvement in their performance @xcite @xcite . in this research, we combine both methods of supervised and unsupervised mt to compensate for the sparsity of the da-en data from the ugt domain. our semisupervised system is explained in the following section. 3 semi-supervised nmt system set up 3.1 cross-lingual language model due to their lexico-grammatical differences, we treat dialectical and standard arabic as two distinct languages. hence, we construct a multi-directional nmt system between the permutations of da-msa-en with the objective of obtaining the highest translation accuracy in the da-en direction. the setup of this system is shown in figure 1. for constructing our semi-supervised nmt system we require the following data: 1. msa-en clean parallel data usually used for training nmt, 2. msa-da clean parallel data from any domain, 3. da-en silver-standard parallel data from the ugt domain with sentiment lexicon infused, and 4. da monolingual data from the ugt domain. it should be noted that the arabic ugt is not written in da per se, it is usually a mix of da and msa. since we are treating da and msa as two distinct languages, we need to extract only the da instances from the ugt dataset. for this purpose, we build our own da detection classifier as per step (1) in figure 1 . in step (2), we pretrain a cross-lingual language model to initialise our nmt system. we follow @xcite approach to train a cross-lingual language model with the combination of the following two objectives: masked language model (mlm): the mlm we train has a similar objective to bert @xcite masking technique but adopting lample and conneau (2019 )'s approach by including the use of text streams of an arbitrary number of sentences (truncated at 200 tokens) instead of pairs of sentences. we optimise the mlm objective on the msa and en source data as well as the da monolingual data mentioned in data requirements 1, 2, and 4 above. translation language model (tlm): we use the tlm objective to improve cross-lingual training where the language model is trained on the gold-standard parallel sentences (i.e msa-en and msa-da in data requirements 1 and 2 above). the training is achieved by randomly masking words in both the source and target sentences. thus, to predict a word masked in a da sentence, for example, the model can either attend to surrounding da words or to the en/msa side of the parallel data if the da context is not sufficient to infer the masked da word. by relying on the parallel data, the tlm objective helps in the alignment of embedding spaces across the three languages. ## conclusion in this research, we tackled the intricate problem of translating sentiment in different arabic dialects in the ugt domain such as tweets and online reviews. we overcome the problem of the scarcity of goldstandard parallel data by training an nmt model with both a supervised and an unsupervised objective functions using monolingual as well as parallel data. we compared this model to a baseline that was trained solely on parallel data and a da-en mt model where we pivoted on msa as an intermediary step. our semi-supervised model showed improved performance over these two models not only in terms of translation quality but specifically in the preservation of the sentiment polarity of the source. we also conducted automatic and manual evaluation of the models' performance and pro-posed a lexicon-based metric that takes into account the sentiment distance between the source and the mt output. overall, our error analysis has revealed that despite some structural inaccuracies the semi-supervised model is more capable of transferring the correct sentiment specifically in aggressive tweets. future research will address the challenge of trading off translation fluency for sentiment accuracy to improve the translation of sentiment-oriented arabic online content. a appendix ex1 source google translate our system reference slay me pissed me off he made me quite angry ex2 source google translate our system reference very wedged very sorry we are very sorry ex3 source google translate our system reference may god protect you, i love you may god protect you, i don't like to lie may god protect you, i don't like to lie ex4 source google translate our system reference god doesn't help him god does not grant him success may god not grant him success. ex5 source google translate our system reference ok let's talk a little, i mean steal the best heck? sorry, let's talk a little he steals the best like this? let's just talk a bit, so does he better steal like this? ex6 source google translate our system reference without getting upset, let's talk without getting upset, so be from talking without getting upset, so be it from talking