Automatic Translation for Multiple NLP tasks: a Multi-task Approach to Machine-oriented NMT Adaptation (2020)

Abstract: Although machine translation (MT) traditionally pursues “human-oriented” objectives, humans are not the only possible consumers of MT output. For instance, when automatic translations are used to feed downstream Natural Language Processing (NLP) components in cross-lingual settings, they should ideally pursue “machine-oriented” objectives that maximize the performance of these components. Tebbifakhr et al. (2019) recently proposed a reinforcement learning approach to adapt a generic neural MT(NMT) system by exploiting the reward from a downstream sentiment classifier. But what if the downstream NLP tasks to serve are more than one? How to avoid the costs of adapting and maintaining one dedicated NMT system for each task? We address this problem by proposing a multi-task approach to machine-oriented NMT adaptation, which is capable to serve multiple downstream tasks with a single system. Through experiments with Spanish and Italian data covering three different tasks, we show that our approach can outperform a generic NMT system, and compete with single-task models in most of the settings.

## introduction neural machine translation (nmt) systems are typically developed considering humans as the c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. however, humans are not the only possible consumers of mt output. in a variety of application scenarios, mt can in fact act as a pre-processor to perform other natural language processing (nlp) tasks. for instance, this is the case of text classification tasks for which, in low-resource conditions, the paucity of training data provides a strong motivation for exploiting translation-based solutions. in tasks like sentiment classification, hate speech detection or document classification (the three application scenarios addressed in this paper) a translation-based approach would allow: i) translating the input text data from an underresourced language into a resource-rich target language for which high-performance nlp components are available, ii) run a classifier on the translated text and, finally, iii) project the results back to the original language. this approach represents a straightforward solution in low/medium-resource 1 language settings where reliable nlp components for specific tasks are not available, and represents a strong baseline in a variety of multilingual and cross-lingual nlp tasks @xcite . however, the nmt systems normally used are still optimized by pursuing human-oriented adequacy and fluency objectives, which are not necessarily the optimal ones for this pipelined solution. these models can indeed produce translations in which some properties of the input text are altered or even lost. for instance, as shown in @xcite , this happens in sentiment classification, where automatic translations can fail to properly project core traits of the input text into the target language. when this happens, the downstream linguistic processor will likely produce results of lower quality. in light of these considerations, @xcite argued that when the role of nmt is to feed a downstream nlp component instead of a human, translating into fluent and adequate sentences is not necessarily the main priority. rather, if the goal is producing translations that are "easy to process" by the downstream component, other optimization strategies might be more effective, even if they result in low-quality output from the point of view of human comprehension. back to the sentiment classification example: before meaning and style, a "machine-oriented" translation should prioritize the optimal projection of the sentiment traits of the input text, which are the key clues from the automatic sentiment classification standpoint. to pursue machine-oriented translation objectives, @xcite proposed machine-oriented reinforce (mo-reinforce), a method based on reinforce @xcite . while in reinforce the objective is to maximize the reward given by humans to nmt systems' output, in mo-reinforce the human feedback is replaced by the reward coming from a downstream nlp system. focusing on sentiment classification, where the classifier's output is a probability distribution over the classes for each input text, they define the reward as the probability of predicting the correct class. evaluation results computed on twitter data show that a downstream english sentiment classifier performs significantly better when it is fed with machine-oriented translations rather than the human-oriented ones produced by a general-purpose nmt system. despite its potential usefulness, mo-reinforce has a limitation that might reduce its general ap-plicability: it requires one nmt model for each downstream task. this represents a possible bottleneck in real industry scenarios, where training and maintaining multiple task-oriented nmt systems (one for each possible downstream task) would be costly and time-consuming, if not unfeasible. to overcome this limitation, in this paper we explore the possibility to simultaneously address multiple downstream tasks with a single nmt system. in this direction, we propose a multi-task learning approach that has two main potential strengths. one is the higher flexibility for industrial deployment due to its architectural simplicity. the other is the possibility to exploit knowledge transfer across similar tasks @xcite , eventually improving the results achieved by the single-task mo-reinforce approach. we test the viability of our multi-task approach on two source languages (spanish and italian foot_1 ) for which data covering different tasks (sentiment classification, hate speech detection and document classification) have to be translated into english and then processed by dedicated nlp components. our results show that translating with the proposed multi-task extension yields significant gains in classification performance with respect to both i) a generic nmt system and ii) the original singletask mo-reinforce by @xcite . besides exploring for the first time a multi-task approach to "machine-oriented" nmt, this paper provides two technical contributions that explain the reported performance gains, namely: i) a reward normalization strategy to weigh the importance of each sample in the course of training, and ii) the application of dropout while sampling the translation candidates, which makes the model more reactive and avoids local optima. on the experimental side, another contribution of this work is the first evaluation on multi-class classification data (i.e., those used for the document classification task), a more challenging scenario compared to the binary task considered by @xcite . by maximizing this objective, the model indirectly pursues the human-oriented objectives of adequacy and fluency embedded in the training parallel corpora. in addition to normal nmt training, these objectives can be directly addressed using reinforcement learning methods such as reinforce @xcite . this method maximizes the expected reward from the end-user: since collecting human rewards is costly, the process can be simulated by comparing the sampled translation candidates with the corresponding reference translations using automatic evaluation metrics like blue @xcite . the two learning strategies (supervised and reinforcement) have two main commonalities: i) the learning objectives are human-oriented, and ii) they both need parallel data, respectively for maximizing the probability of the translation pair in supervised learning and for simulating the human reward in reinforcement learning. ## multi-task machine-oriented nmt our multi-task extensions of mo-reinforce include: i) prepending task-specific tokens to the input for managing multiple domains and computing normalized rewards to avoid under/over-fitting (section 3.1), and ii) adding randomness to the sampling process to push for higher exploration of the probability space (section 3.2). ## experiments our multi-task extension of mo-reinforce is evaluated on two source languages: spanish and italian. for spanish, we consider the downstream tasks of document classification and hate speech detection. for italian, we select document classification and sentiment analysis. the evaluation is done by feeding dedicated english classifiers (one for each downstream task) with translations produced by different nmt models, namely: i) a general-purpose nmt system, ii) the original single-task mo-reinforce, and iii) different variants of our multi-task extension. the goal is to maximize the classification performance on each downstream task. as another term of comparison for the three translation-based solutions, we consider the results obtained by directly processing the input sentences with task-specific spanish and italian classifiers trained on the same small datasets used to adapt the general-purpose nmt system. in line with @xcite , the multi-task approach is expected to outperform the generic (human-oriented) nmt system, as well as the task/language-specific classifiers trained on few data points. ideally, thanks to the solutions proposed in section 3, it should also compete with the single-task (machine-oriented) models. this would indicate the viability of a single-model approach to simultaneously address multiple tasks. in the following, we describe the task-specific data used for model adaptation and evaluation, as well as the parallel corpora used for training the generic nmt system. their statistics are respectively reported in tables 1 and 2 . document classification. for this multi-class labelling task, we use the mldoc corpora @xcite , which cover 8 languages, including english, spanish and italian. they comprise news stories labeled with 4 different categories: ccat (corporate/industrial), ecat (economics), gcat (government/social), and mcat (markets). for each language, the training, de-velopment and test sets respectively contain 10k, 1k, and 4k documents uniformly distributed into the 4 classes. following @xcite , for training and evaluation we only consider the first sentence of each document, which usually provides enough information about the general content of the document. we use the whole english training set to build our downstream classifiers. to simulate an under-resourced setting, we randomly sample 100 documents for each class from the spanish and italian training sets. we use these samples to adapt the generic nmt system for the downstream task, while for development and test we use the whole sets. ## results and discussion our experimental results are shown in the f1 scores obtained by the generic nmt systems in document classification (mldoc) show that the simplest translation-based approach produces competitive results compared to those achieved by language-specific classifiers trained on small in-domain data. the situation is different for tasks whose data differ significantly from those used to train the general-purpose system. on the user-generated content used for hate speech detection and sentiment classification (i.e. twitter data), the generic results are indeed poor. this shows that nmt models trained by only pursuing humanoriented criteria might not fit to target downstream tasks, for which machine-oriented adaptation becomes necessary. machine-oriented adaptation with single-task mo-reinforce yields the expected benefits, with improvements (+3.25 f1 points for document classification, +18.38 for sentiment classification in italian) that allow to outperform the languagespecific (source) classifiers in three tasks out of four. these gains confirm and validate on multiple tasks (including multi-class classification) the findings of @xcite , showing that mo-reinforce can leverage the feedback from external linguistic processors to adapt the nmt model towards translations that maximize the performance in downstream applications. the middle part of the bottom part of the results are significantly better with the noisy sampling approach discussed in section 3.2. in both the languages and in all the tasks, the reported f1 scores approach those obtained by the singletask variant of mo-reinforce (which in two cases is even outperformed) and always improve over the language-specific source classifiers. this confirms that enabling dropout while generating the translation candidates avoids the model to get stuck in local optima, and promotes diversity in producing candidates that eventually receive higher rewards. combined, the two contributions of this paper (reward normalization and noisy sampling) yield mixed outcomes. for spanish, we observe a further improvement compared to noisy sampling in document classification (+2.40), which comes at the cost of a small drop in hate speech detection (-0.48). also for italian there is an improvement over noisy sampling alone in document classification (+0.59), but a larger drop in sentiment classification performance (-2.25). however, it's worth remarking that: i) the size of the italian sentiment analysis dataset is almost 10 times larger than the size of the document classification dataset, and ii) the data used to train the english classifiers are even more unbalanced. being able to harmonize the results of the two task hence becomes quite difficult. nevertheless, combining reward normalization and noisy sampling has a general positive effect, which allows the multi-task mo-reinforce system to approach and, in some tasks, even to outperform the single task models. in our final analysis, we investigate the effect of introducing dropout and reward normalization when mo-reinforce is used in the single-task scenario. as shown in looking at the output of the system, we noticed that the translations are shorter and are not adequate compared to the output of the generic system. for instance, in document classification, the samples belonging to the corporate class are usually translated to "the company.", or the positive samples in sentiment analysis are translated to "i'm very happy.", which are easier to be classified by the downstream classifiers. ## conclusion we proposed an extension of the mo-reinforce algorithm, targeting "machine-oriented" nmt adaptation in a multi-task scenario. in this scenario, different nlp components are fed with translations produced by a single nmt system, which is adapted to generate output that is "easy to process" by the downstream processing tools. to close the performance gap between the single and the multitask variants of mo-reinforce, we enhanced the latter with reward normalization and noisy sampling strategies. our experiments show that, with these two features, the multi-task mo-reinforce approach achieves significant gains in performance that make it competitive with the single-task solution (though, having one single model to build and maintain, at considerably lower deployment costs). furthermore, we show that reward normalization and noisy sampling can also help in the single-task setting, where our approach outperforms the original mo-reinforce in four tasks.