Sentiment Preservation in Review Translation using Curriculum-based Re-inforcement Framework (2021)

Abstract: Machine Translation (MT) systems often fail to preserve different stylistic and pragmatic properties of the source text (e.g. sentiment and emotion and gender traits and etc.) to the target and especially in a low-resource scenario. Such loss can affect the performance of any downstream Natural Language Processing (NLP) task and such as sentiment analysis and that heavily relies on the output of the MT systems. The susceptibility to sentiment polarity loss becomes even more severe when an MT system is employed for translating a source content that lacks a legitimate language structure (e.g. review text). Therefore and we must find ways to minimize the undesirable effects of sentiment loss in translation without compromising with the adequacy. In our current work and we present a deep re-inforcement learning (RL) framework in conjunction with the curriculum learning (as per difficulties of the reward) to fine-tune the parameters of a pre-trained neural MT system so that the generated translation successfully encodes the underlying sentiment of the source without compromising the adequacy unlike previous methods. We evaluate our proposed method on the English–Hindi (product domain) and French–English (restaurant domain) review datasets and and found that our method brings a significant improvement over several baselines in the machine translation and and sentiment classification tasks.

## introduction product and/or service reviews available in the e-commerce portals are predominantly in the english language, and hence a large number of population can not understand these. machine translation (mt) system can play a crucial role in bridging this gap by translating the usergenerated contents, and directly displaying them, or making these available for the downstream natural language processing (nlp) tasks e.g. sentiment classification foot_0 @xcite @xcite . however, numerous studies @xcite @xcite have found a significant loss of sentiment during the automatic translation of the source text. the susceptibility to sentiment loss aggravates when the mt system is translating a noisy review that lacks a legitimate language structure at the origin. for example, a noisy review contains several peculiarities and informal structures, such as shortening of words (e.g. "awesome" as "awsm"), acronyms (e.g. "oh my god" as "omg"), phonetic substitution of numbers (e.g. "before" as "b4"), emphasis on characters to define extremity of the emotion (e.g. "good" as "gooooooood"), spelling mistakes, etc. unfortunately, even a pervasively used commercial neural machine translation (nmt) system, google translate, is very brittle and easily falters when presented with such noisy text, as illustrated through the following example. review text (english): i found an awsome product. (positive) google transliteration (hindi): mujhe ek ajeeb utpaad mila. (neutral) the example shows how the misspelling of a sentiment bearing word "awesome" gets this positive expression translated to a neutral expression. in the above context, if an unedited raw mt output is directly fed to the downstream sentiment classifier, it might not get the expected classification accuracy. thus, in this work we propose a deep-reinforcement-based framework to adapt the parameters of a pre-trained neural mt system such that the generated translation improves the performance of a cross-lingual multi-class sentiment classifier (without compromising the adequacy). more specifically, we propose a deep actor-critic (ac) reinforcement learning framework in the ambit of curriculum learning (cl) to alleviate the issue of sentiment loss while improving the quality of translation in a cross-lingual setup. the idea of actor-critic is to have two neural networks, viz. (i). an actor (i.e. a pre-trained nmt) that takes an action (policy-based), and (ii). a critic that observes how good the action taken is and provides feedback @xcite . this feedback acts as a guiding signal to train the actor. further, to better utilize the data, we also integrate curriculum learning into our framework. recently, @xcite demonstrated that an mt system (actor) can be customised to produce a controlled translation that essentially improves the performance of a cross-lingual (binary) sentiment classifier. they achieved this task-specific customisation of a "generic-mt" system via a policy-based method that optimizes a task-specific metric, i.e. f 1 score (see section 2). however, this often miserably fails to encode the semantics of the source sentence. recent studies @xcite demonstrated that the non-opinionated semantic content improves the quality of a sentiment classifier. accordingly, the transfer of such information from the source to the target can be pivotal for the quality of the sentiment classifier in a cross-lingual setup. towards this, we investigate the optimization of a harmonic-score-based reward function in our proposed rl-based framework that ensures to preserve both sentiment and semantics. this function operates by taking a weighted harmonic mean of two rewards: (i). content preservation score measured through sentence-level bleu or sbleu; and (ii). sentiment preservation score measured through a function that performs element-wise dot product between a predicted sentiment distribution and the gold sentiment distribution. empirical results, unlike @xcite , suggest that our rl-based fine-tuning framework, tailored to optimize the harmonic reward, preserves both sentiment and semantics in a given nmt context. additionally, we also found that the above fine-tuning method in the ambit of curriculum learning achieves an additional performance gain of the mt system over a setting where curriculum based fine-tuning is not employed. the core of curriculum learning (cl) @xcite is to design a metric that scores the difficulty of training samples, which is then used to guide the order of presentation of samples to the learner (nmt) in an easy-to-hard fashion. to the best of our knowledge, this is the very first work that studies the curriculum learning (cl) for nmt from a new perspective, i.e. given a pre-trained mt model, the dataset to fine-tune, and the two tasks viz. sentiment and content preservation; we utilize a reward-based metric (i.e. harmonic score) to define the difficulty of the tasks and use it to score the data points. the use of harmonic reward based scoring/ranking function implicitly covers the tasks' overall difficulty through a single metric. moreover, understanding that obtaining a gold-standard polarity annotated data is costlier, the fine-tuning of pre-trained nmt model is performed by re-using only a small subset of the supervised training samples that we annotated with respect to (w.r.t) their sentiment. empirical results suggest that additionally enriching a random small subset of the training data with extra sentiment information, and later re-using them for the fine-tuning of the referenced model via our proposed framework (c.f. section 3) observes an additional gain in bleu and f 1 score over a supervised baseline. we summarize the main contributions and/or the key attributes of our current work as follows: (i). create a new domain-specific (i.e. product review) parallel corpus, a subset of which is annotated for their sentiment; @xcite . propose an ac-based fine-tuning framework that utilizes a novel harmonic mean-based reward function to meet our two-fold objectives, viz. enabling our nmt model to preserve; (a). the non-opinionated semantic content; and (b). the source sentiment during translation. (iii). additionally, we utilize the idea of cl during the rl fine-tuning of the pre-trained model and try to learn from easy to hard data, where hard corresponds to the instances with lower harmonic reward. to the best of our knowledge, this is the first work in nmt that studies cl in the ambit of rl fine-tuning. ## related work the use of translation-based solution for cross-lingual sentiment classification is successfully leveraged in the literature @xcite @xcite @xcite @xcite @xcite which suggest an inspiring use-case of the mt system, and brings motivation for this piece of work. given the context of this work, we look at the pieces of works that address the preservation of sentiment in the automatic translation. in one of the early works, chen and zhu (2014) used a lexicon-based consistency approach to design a list of sentiment-based features and used it to rank the candidates of t-table in a phrase based mt system. @xcite prepared the positive, negative and neutral sentiment-specific translation systems to ensure the cross-lingual sentiment consistency. recently, @xcite proposed machine-oriented (mo) reinforce, a policybased method to pursue a machine-oriented objective foot_1 in a sentiment classification task unlike the traditional human-oriented objective foot_2 . it gives a new perspective for a use-case of the mt system (i.e. machine translation for machine). to perform this task-specific adaption (i.e. produce output to feed a machine), @xcite adapted the reinforce of @xcite by incorporating an exploration-oriented sampling strategy. as opposed to one sampling of reinforce, mo reinforce samples k times, (@xmath0, and obtains a reward for each sample from the sentiment classifier. a final update of the model parameters are done w.r.t the highest rewarding sample. although they achieved a performance boost in the sentiment classification task, they had to greatly compromise with the translation quality. in contrast to @xcite , we focus on performing a task-specific customisation of a pre-trained mt system via a harmonic reward based deep reinforcement framework that uses an ac method in conjunction with the cl. the adapted nmt system, thus obtained, is expected to produce a more accurate (high-quality) translation as well as improve the performance of a sentiment analyser. @xcite ; @xcite , unlike us, used the popular ac method, and focused only on preserving the semantics (translation quality) of a text. additionally, we develop a cl based strategy to guide the training. recently, @xcite also studied ac method in the context of nmt. however, they used this method to learn the curriculum for re-selecting influential data samples from the existing training set that can further improve the performance (translation quality) of a pre-trained nmt system. ## methodology firstly, we perform the pre-training of a nmt model until the convergence using the standard log-likelihood (ll) training on the supervised dataset (c.f. s(gold sent) [ 1, 0, 0 ] t(auto sent) [ 0.2, 0.1, 0.7] ek bhayaanak utpaad mila found an awsome product reinforce nmt (pretrained with mle) encoder decoder policy sample episode and reward update policy parameter value critic encoder decoder value function sample translation classifier sentiment corpus t(auto sent) [ 0.2, 0.1, 0.7 ] harmonic mean sbleu element wise dot product reward module (r) ek bhayaanak utpaad mila. ek bahut badhiya utpaad mila. ek bhayaanak utpaad mila. 1. pos 2. neu 3. neg s(gold sent) [1, 0, 0 ] the critic feedback is used by the actor to identify those actions that bring it a better than the average reward. in the above context, a feedback of a random critic would be useless for training the actor. hence, similar to the actor we warm up the critic for one epoch by feeding it samples from the pre-trained actor, while the actor's parameters are frozen. we then fine-tune these models jointly so that -as the actor gets better w.r.t its action, the critic gets better at giving feedback (see section 4.2 for the dataset and reward used in the pre-training and fine-tuning stages). the details of the loss functions that the actor and critic minimizes are discussed in section 3.1. furthermore, to better utilize the data, we finally integrate cl into our ac framework (our proposed approach). empirical results (section 5.1) show that during fine-tuning, presenting the data in an easy-to-hard fashion yields a better learned actor model over the one obtained via vanilla (no-curriculum based) fine-tuning. our proposed framework brought improvements over several baselines without using any additional new training data in the two translation tasks, i.e. (i). english-hindi foot_3 and (ii). french-english foot_4 . since our proposed framework is a combination of rl via ac method and cl, we first present the details of the main components of the ac model alongside their training procedure in section 3.1. the details of the reward model are presented in section 3.2, and then introduce the plausibility of cl in section 3.3. finally, we describe our proposed cl-based ac framework in algorithm 1. ## datasets and experimental setup in this section, we first discuss the datasets used in different stages of experiments followed by the steps involved in the creation of datasets, the baselines used for comparison, and the model implementation details. dataset: our nmt adaptation experiments are performed across two language pairs from different families with different typological properties, i.e. english to hindi (henceforth, en-hi) and french-english (henceforth, fr-en). we use the following supervised datasets for the pre-training and validation of ll-based nmt in en-hi and fr-en tasks, (i). for en-hi task, we use a newly created domain-specific parallel corpus (see section 4.1) whose sources were selected from an e-commerce site. this corpus is released as a part of this work. statistics of the dataset is shown in ## results and analysis we first present the results of fine-tuning the pre-trained mt through different rl-based methods, i.e. (i). reinforce (ii). @xcite . vanilla ac (ours) in section 5.1. further, to better analyse the utility of harmonic reward (r 3 ) in sentiment and content preservation task over the previously studied rewards (i.e. sbleu: r 1 or bert: r 2 ) in the context of nmt @xcite @xcite @xcite , we additionally present the fine-tuning results of the vanilla ac method with the following two types of rewards: (i). only content, i.e. r 1 and (ii). only sentiment, i.e. r 2 as a reward. we choose the best performing reward model (i.e. r 3 ) among the ac-based nmt(s). at last, we discuss the results in the context of our curriculum-based ac framework. to evaluate the translation quality we record the bleu score of the rl testset when translated from the relevant models. to validate our claim that the translations obtained by our proposed mt system can further improve the performance of the sentiment classifier in a cross-lingual setup over the baselines, we do the following. we apply the target language sentiment classifier to the translations obtained by the ll-based nmt system vs. all the customised rl-based nmt systems, and record their f 1 scores. ## conclusion in this paper, we have proposed a curriculum-based deep re-inforcement learning framework that successfully encodes both the underlying sentiment and semantics of the text during translation. in contrast to the reinforce-based frameworks @xcite (actor only models), ours is a critic-based approach that helps the actor learns an efficient policy to select the actions, yielding a high return from the critic. besides, with the support of curriculum learning, it can be more efficient. this is also established (empirically) through the observed additional boost (significant at @xmath1.05) in bleu score over the baselines. further, we have manually created a domain-specific (product reviews) polarity-labelled balanced bilingual corpus for english-hindi, that could be a useful resource for research in the similar areas. we shall make the data and our codes available to the community.