Negation typology and general representation models for cross-lingual zero-shot negation scope resolution in R ussian, F rench, and S panish. (2021)

Abstract: Negation is a linguistic universal that poses difficulties for cognitive and computational processing. Despite many advances in text analytics, negation resolution remains an acute and continuously researched question in Natural Language Processing. Reliable negation parsing affects results in biomedical text mining, sentiment analysis, machine translation, and many other fields. The availability of multilingual pre-trained general representation models makes it possible to experiment with negation detection in languages that lack annotated data. In this work we test the performance of two state-of-the-art contextual representation models, Multilingual BERT and XLM-RoBERTa. We resolve negation scope by conducting zero-shot transfer between English, Spanish, French, and Russian. Our best result amounts to a token-level F1-score of 86.86% between Spanish and Russian. We correlate these results with a linguistic negation typology and lexical capacity of the models.

## introduction negation continues to occupy the minds of many researchers. it is a fascinating and complicated linguistic phenomenon that is still not entirely understood or conceptualized. moreover negation is an important thought process. the ability to negate is a deeply human trait that is also universal, therefore any given language is bound to have negation @xcite . negation has the power to change the truth value of a proposition. thus its identification in text is of utmost importance for the reliability of results since negated information should either be discarded or presented separately from the facts. this is particularly relevant for biomedical text mining and sentiment analysis but is also important for most natural language processing (nlp) tasks. the identification of negated textual spans, however, is far from trivial. negation exhibits great diversity in its syntactic and morphological representation. like many other nlp tasks, most work on negation detection has been done on the english language, though there is a growing amount of research on negation detection in spanish, chinese and some other languages. despite the need for quality text analytics around the world, annotated data is still sparse in many languages. this motivates the further exploration of approaches like transfer learning where models are trained on available resources and subsequently tested on a different target language. in this paper we use a cross-lingual transferlearning approach for negation scope detection using two state-of-the-art general purpose representation models: mbert (multilingual bert, @xcite and xlm-r (xlm-roberta, @xcite . we fine-tune the models on freely accessible annotated corpora in english, spanish, and french and test them cross-lingually. additionally we test the models on a small dataset in russian which was specially annotated for the experiment. our research is guided by three objectives: in section 2 we perform a brief typological analysis of the languages in relation to negation. additionally, we overview previous work on cross-lingual negation scope resolution. section 3 discusses the datasets and highlights their annotation differences. we describe the experiments and present the results in section 4, and in sections 5 and 6 we discuss the results and draw conclusions. when a sentence contains more than one negation trigger, negative concord (nc) languages treat them as one, letting relevant negative markers intensify one another. the majority of languages including french, spanish, and russian belong to the nc group. standard english, on the other hand, is a double negation (dn) language where each negative marker is interpreted separately. hossain et al. (2020) compared english to a number of languages in regards to negation features drawn from the world atlas of language structures (wals) 1 . they showed that the number of negation-related errors in machine translation corresponds to how close the languages are in a typology based on negation. inspired by their discoveries we construct a negation-based typology for our languages and merge it with the classification from @xcite . even though english, french, spanish, and russian are in the same linguistic family and feature the same subject-verb-object pattern, the typology based on negation assigns them to different categories ( we expect a negation-based linguistic typology to help us predict and interpret our results. according to our classification, russian is most similar to spanish and least similar to english. thus we hypothesise that zero-shot transfer from spanish into russian will be most successful. automated negation detection consists of two tasks: identification of negation cues, and detection of sentence parts that are affected by these cues. the latter is called negation scope resolution, the task that interests us most. negation detection began in the medical domain with the goal of improving information retrieval from electronic health records (ehrs). rulebased algorithms such as negexpander @xcite , negfinder @xcite , negex @xcite , and their adaptations were used in order to find medical concepts and then determine whether they are negated. the scope of negation was often understood as a distance between a negation cue and a medical term that it affects. these algorithms were successful and some are still in wide use due to their explainability, customizability, and independence from annotated data. negex is incorporated into various modern computational libraries 2 and is successfully used for biomedical texts @xcite . despite the aforementioned qualities, rulebased algorithms suffer from an inherent inability to generalize @xcite @xcite . the release of the bioscope corpus @xcite became a pivotal moment for negation detection by providing data for machine learning. negation scope resolution was formalized by @xcite ; @xcite , who established it as a problem of sequence classification. using gold-standard cues and an ensemble of three different classifiers, they achieved the best f1-score of 84.71% on the full papers subcorpus of bioscope and 90.67% on the abstracts subcorpus. the latter result was later surpassed by @xcite who employed neural networks and reached a score of 92.11%. the shared task on resolving the scope and focus of negation @xcite addressed the issue of negation scope resolution directly and released another annotated corpus (conandoyle-neg, @xcite . the best system @xcite used an enhanced hybrid model by @xcite and a semantic parser. they reached an f1-score of 88.2% using gold-standard cues. these results were surpassed by @xcite who used the conditional random fields classifier and reached an f1-score of 89.4%. additionally, @xcite secured an f1-score of 89.93% on the sfu review-neg corpus @xcite , another publicly available corpus annotated for negation scope. the results on these three corpora remained the benchmark for negation scope resolution until the bidirectional encoder representation from transformers (bert, @xcite became the new state of the art. moreover, bert became widely used for transfer learning due to its enhanced ability to generalize using attention and general purpose language representations. negbert @xcite set new records for negation scope resolution on all three publicly available corpora. cross-lingual negation scope work. many languages remain under-researched regarding negation detection and particularly scope resolution. one of the main problems is the lack of annotated data. there currently exist a handful of corpora in english, two in spanish, and one corpus each in swedish, german, dutch, chinese, italian, and portuguese which are not all publicly available @xcite . negation work on spanish has been growing in recent years but it has mostly concerned senti-ment analysis @xcite @xcite . rivera zavala and martinez (2020) are the first ones to work with sense embeddings to detect negation cues and scopes in the spanish biomedical and general domain texts. they also worked with mbert but in a monolingual setting. the research on negation in french is particularly limited. aside from a few papers describing rule-based approaches @xcite and the implementation of bilstms by @xcite , there is barely any other research available on the topic. cross-lingual work on negation detection is even more limited. @xcite developed a truly cross-lingual system that uses no language specific features. they worked with english and chinese and used universal dependencies to abstract away from the word order. their bidirectional dependency lstm model reached an f1score of 72.46%. finally, @xcite employed multilingual bert to perform zero-shot transfer for negation scope resolution and showed good preliminary results. we build on this work and compare mbert with a new multilingual general purpose representation model, xlm-r. unlike mbert, xlm-r was pre-trained on more than two terabytes of filtered data collected by commoncrawl. instead of wordpiece units it uses sentencepiece @xcite units and features a bigger size of shared vocabulary (250k). ## data in our experiments we work with a corpus of clinical texts in french @xcite , and sfu reviewsp-neg, a spanish corpus of online reviews @xcite . the english data includes the biological paper abstracts and full scientific articles in the domain of bioinformatics from bioscope @xcite , all available subcorpora of the conandoyle-neg corpus @xcite as well as sfu (sfu review-neg, @xcite , a large multidomain corpus of product reviews. we use the english corpora separately and also combine them into one training dataset. the three corpora belong to different domains and feature certain variations in scope annotation guidelines. despite these significant problems we combine the datasets based on the successful cross-corpora knowledge transfer described by @xcite . the bioscope annotators set the precedent by ultimately basing scope annotation on syntax. they employed a maximal scope size strategy and extended annotation to the biggest syntactic unit possible. the normal direction of scope was assumed to be to the right of the cue. the subject is not included in the scope, unless the sentence has a passive voice. @xcite argued that semantically the subject should be always annotated within the scope. thus, unlike the bioscope corpus, conandoyle-neg includes the subject yet excludes the cue. additionally, it features morphological negations. the sfu corpus mostly adheres to the bioscope's annotation guidelines but does not include cues into the scope of negation. the french data is described in dalloux et al. ( 2020 ) and is publicly available on request foot_1 . it combines two subcorpora of clinical narratives. its format and annotations are loosely modeled on the conandoyle-neg corpus. the data in the spanish sfu reviewsp-neg corpus can be requested from simon fraser university. its annotations reflect the guidelines of the english corpora but are also based on spanish grammar. in our experiments we only use sentences that contain at least one negation. we duplicate sentences with multiple negations into several copies containing a single negation. ## experiments and results we used negbert @xcite as the main architecture and employed bert-base-multilingual-cased and xlm-roberta-base-model models. we fine-tuned the two models on the three datasets: english (en), spanish (es), and french (fr). all the models were trained with the same set of hyperparameters. early stopping method with patience set to 9 was used to prevent overfitting. the maximum input length was adjusted to 250 to prevent truncated sentences. the word-level token class is determined by using the argmax function on the averaged softmax probabilities of all subword units. we use goldstandard negation cues and report token-level f1scores for negation scope resolution ( despite the fact that the english corpora are of different domains, models fine-tuned on the combined english data brought better cross-lingual results than models that were fine-tuned on each corpus individually. even the model fine-tuned on the conandoyle-neg corpus did not perform better on the russian version of the text. thus, we only discuss the results of the model trained on the entirety of the english data. ## discussion and error analysis there have been many debates on whether bertlike models truly "understand" negation. @xcite showed evidence for shallow encoding of this phenomenon in both bert and roberta. meanwhile @xcite demonstrated that these models lack linguistic abstraction abilities and fail when confronted with compositional semantic aspects of language. in our experiments, the xlm-r model performed significantly better than mbert for all language pairs. as an additional metric, we measured how well both models identified scopes with 100% precision. averaged across all languages, both models performed equally well, with mbert solving 46.23% of exact scopes, and xlm-r -46.66%. the best result for russian was produced by the xlm-r model fine-tuned on spanish (53% of exact scopes). in fact, russian benefited most from a transfer from spanish and least from french, irrespective of training data size or model type. we can assume that the success of the spanish-russian transfer is partially due to the commonalities described in we investigated several factors that might have negatively affected the french-russian knowledge transfer. for example, we examined the vocabularies of the models and calculated lexical overlap between the datasets based on a model-specific tokenization. the comparison in we found 19 sentences in the russian dataset that contain negation structures with negative indefinite pronouns. despite the fact that these pronouns are always marked as cues, the english xlm-r model included them into the scope 9 times. the english mbert model made that same mistake 3 times. on the other hand, neither spanish, nor french models had this problem. we can hypothe-sise that a model fine-tuned on english could not coordinate a negative particle with an indefinite negative pronoun in the same sentence since this does not occur in english. during the examination of these 19 sentences we stated that the models fine-tuned on the french data persistently omit the subject of a sentence in the annotation of scope. the english models also suffered from this problem but to a lesser extent. this can be traced to the difference in annotation. the subject is not annotated in the french corpus while only part of the english data features that annotation. figure 2 illustrates the issue of negative indefinite pronouns as well the annotation of a sentence's subject. additionally we investigated scope annotations which were precisely identified by one type of model but not the other. we chose to look at the highest scoring language pair spanish-russian where the models were trained on 1870 sentences. there are 15 sentences where the xlm-r model found scope with a perfect precision while mbert did not. in most cases mbert made a mistake in the leftward direction from a negation cue. we found only four cases where mbert scored perfectly while xlm-r made a mistake. the mistakes are rather random and do not seem to belong to a particular pattern. overall, we detected several situations where mistakes made by the models could be scrutinized due to questionable annotation. we acknowledge that the lack of additional annotators and an inter-annotator agreement is a weakness that should be addressed in further work. ## conclusion the short excursion into negation scope resolution in russian using zero-shot model transfer has shown good preliminary results. despite contro-versial previous findings, multilingual general purpose representation models perform rather well on negation scope resolution. xlm-roberta scored consistently better than mbert in all language pairs. we constructed a typology that classifies english, spanish, french, and russian according to their negation-based features. since indefinite negative pronouns play a role in several typological categories, we investigated their effect on zero-shot transfer. we found that fine-tuning models on english compromises their performance with this phenomenon when transferring to russian, which correlates with the negation typology. transferring syntactic negation knowledge from spanish brought the most benefit for russian. this result is fully in line with the negation typology of the four languages. despite the clear correlation between the negation typology and the results of the spanish-russian transfer, not all outcomes are easily explainable. the relatively poor performance of the french-russian transfer might be related to the domain mismatch and the difference in annotation schemes. a lower lexical overlap between the vocabularies could have had an effect as well. future work involves growing the russian corpus of negations, ideally benefiting from multiple annotators. it may prove beneficial to perform a systematic examination of all the categories constituting the negation typology and to expose their effects on knowledge transfer across languages.