S pan A lign: Efficient Sequence Tagging Annotation Projection into Translated Data applied to Cross-Lingual Opinion Mining (2021)

Abstract: Following the increasing performance of neural machine translation systems, the paradigm of using automatically translated data for cross-lingual adaptation is now studied in several applicative domains. The capacity to accurately project annotations remains however an issue for sequence tagging tasks where annotation must be projected with correct spans. Additionally, when the task implies noisy user-generated text, the quality of translation and annotation projection can be affected. In this paper we propose to tackle multilingual sequence tagging with a new span alignment method and apply it to opinion target extraction from customer reviews. We show that provided suitable heuristics, translated data with automatic span-level annotation projection can yield improvements both for cross-lingual adaptation compared to zero-shot transfer, and data augmentation compared to a multilingual baseline.

## introduction large self-supervised pre-trained models which are fine-tuned on downstream tasks have become the de facto standard in nlp. however, monolingual pre-trained models are only available in some high resource languages due to data limitations. additionally, in multilingual settings, having a separate model for each language quickly becomes unpractical. multilingual models which are pretrained on monolingual corpora in multiple languages such as multilingual bert (mbert) and xlm-r provide an alternative. they have been shown to yield on par, if not better performance on various tasks compared to monolingual models and perform particularly well for low-resource languages provided similar languages are represented in the training data @xcite ). what's more, they have shown surprising capacities for generalization with zero-shot cross-lingual transfer, in which a model is fine-tuned for a task using annota-ted data in a specific language and then evaluated in an unseen language @xcite . considering that labeled data is more readily available for certain languages, this technique of cross-lingual transfer can be harnessed to process low-resource languages, therefore bypassing the need for a costly annotation process. some research effort has been done in grasping to what extent these models are language agnostic @xcite , though more probing is necessary in order to fully understand and measure how multilingual these models are. another approach to cross-lingual transfer consists in using machine translation (mt) to adapt the annotated training data available in a source language to the target language. the resulting translated data is used to train a model with supervision in the target language. producing a label for translated data is straightforward for sentence-level tasks. however, little work focuses on adapting datasets annotated at the word level. such adaptation requires an additional step of label projection which can be error-prone and introduce noise in the training data. a common way of projecting labels consists in obtaining word alignments between source and target utterances and then projecting the labels. in a recent paper, @xcite argued that zero-shot cross-lingual transfer surpassed translation-based adaptations for several sequence tagging tasks, and proposed to first warm up the model's weights on the translated data only and then fine-tuning on the original data. in contrast, we show that translationbased adaptation yields superior performance compared to zero-shot cross-lingual transfer, provided the right annotation projection method is used. we propose to apply this method in the context of opinion mining, and specifically to aspect-based sentiment analysis (absa) @xcite . nowadays, people increasingly rely on reviews and comments, e.g. on social media and review websites, to select which products to buy or which services to use. companies can also make use of these data which provide insightful feedback from their customers. absa is concerned with extracting finegrained information from customer feedback, e.g. which aspects of a product or a service are being referred to, which entities are associated with them, and what is the customer's attitude towards them. one challenge that this task poses is that reviews are user-generated and can thus be very noisy. online users tend to produce text that derives from the standard form of language with e.g. misspellings, internet slang, abbreviations, phonetic transcriptions, and missing or incorrect punctuation marks. for companies operating in several countries, another challenging aspect is the multiplicity of languages found in these reviews. accordingly, multilingual approaches to opinion mining are essential to efficiently analyze customer feedback across countries. we first review previous works related to crosslingual transfer and absa. we then present our translation-based adaptation approach, and show its relevance in two scenarios : cross-lingual adaptation where annotated data is only available in the source language, and data augmentation where annotated data is available for both source and target languages. lastly, we turn to an analysis of noisy user-generated text and propose a heuristic to filter out noisy utterances before translation. ## spanalign in the case of sentence-level tasks, the translated data can be used right away using the same labels as in the source language corpus. in contrast, adapting sequence tagging datasets annotated at the word-level requires an additional step of annotation projection. a word-to-word correspondence between source and target utterances can be obtained using word alignment tools. one simple approach to annotation projection is then to project the gold labels in the source utterance to their corresponding word in the target utterance using this mapping. however, as token-level annotations may span several words, this approach can be error-prone, as illustrated in figure 1 . consequently, we introduce a span-based alignment approach to project gold annotations at the span level. additionally we make the assumption that translated data with erroneous annotation alignment are more likely to be harmful for the subsequent training process, and propose to introduce a constraint on span alignment to filter out translated annotations and remove them from the candidate pool of translated training data. ## translation-based adaptation modern neural machine translation (nmt) provides satisfactory results thanks to transformer models and has become increasingly available. as a result, such models can be used effectively to create synthetic data by translating a source language reference corpus. we use marian nmt foot_0 to translate the source language corpora into the other languages, and viceversa. marian nmt is open source and allows the use of pre-trained nmt models from the opus-mt project (tiedemann and thottingal, 2020) available on this framework. a description of each model and its evaluation on benchmarks can be found online. 2 to obtain word alignments, we use both off-the-shelf fast_align @xcite and simalign (jalili sabet et al., 2020). for the latter, which doesn't need any parallel data for training but relies on multilingual word representations, we use the itermax variant, which was shown to outperform other word alignment approaches. note that we did not adapt the translation models nor the alignment approaches to our specific domains. ## experiments in this section, we describe the experiments that we conducted to explore the relevance of translation-based adaptation, both for cross-lingual adaptation, where annotated data is available in only one language, and for data augmentation, where multilingual datasets are available and we seek to improve performance from fine-tuning on the plain concatenation of data in all languages. ## analysis in line with the results from the previous section, we conduct a refined analysis to shed some light on the impact of noisy text. when applied to the semeval datasets, data augmentation in the resource-rich setting is beneficial and yields a significant improvement in performance, especially for english. for the in-house datasets however, data augmentation does not affect the performances as much compared to the multilingual baseline. our hypothesis is that the inhouse datasets are more noisy and contain a more specific lexicon, which is what could be perturbing the translations, and thus negatively impacting the rest of the pipeline. as we did not have gold data to explicitly evaluate the orthographic or grammatical deviation rate, nor the translation quality, nor the alignment quality, we propose an approximation to quantify noise and compare semeval and in-house datasets. we compute the out-of-dictionary (ood) rate using reference dictionaries recommended with hunspell 7 and available from firefox for each language. 8 the ood rate is computed as follows : considering the ood rate as an indicator of noise in text, we can filter out the most noisy utterances from the source dataset in french before translating it into the other languages. we compute the ood rate for each utterance and rank them from most noisy to least noisy. the top n percent is then filtered out from the dataset before translation. we experiment with several values for n. results can be seen in ## conclusion we have proposed an efficient yet simple way of generating training data for a sequence tagging task based on translation and label projection. when applied to noisy data such as customer reviews, we 9. note that the higher average ood rate in arabic can be attributed to dialectal variations.