D etecting F ine- G rained C ross- L ingual S emantic D ivergences without S upervision by L earning to R ank (2020)

Abstract: Detecting fine-grained differences in content conveyed in different languages matters for cross-lingual NLP and multilingual corpora analysis, but it is a challenging machine learning problem since annotation is expensive and hard to scale. This work improves the prediction and annotation of fine-grained semantic divergences. We introduce a training strategy for multilingual BERT models by learning to rank synthetic divergent examples of varying granularity. We evaluate our models on the Rationalized English-French Semantic Divergences, a new dataset released with this work, consisting of English-French sentence-pairs annotated with semantic divergence classes and token-level rationales. Learning to rank helps detect fine-grained sentence-level divergences more accurately than a strong sentence-level similarity model, while token-level predictions have the potential of further distinguishing between coarse and fine-grained divergences.

## introduction comparing and contrasting the meaning of text conveyed in different languages is a fundamental nlp task. it can be used to curate clean parallel corpora for downstream tasks such as machine translation @xcite , cross-lingual transfer learning, or semantic modeling @xcite , and it is also useful to directly analyze multilingual corpora. for instance, detecting the commonalities and divergences between sentences drawn from english and french wikipedia articles about the same topic would help analyze language bias @xcite , or mitigate differences in coverage and usage across languages @xcite @xcite . this requires not only detecting coarse content mismatches, but also fine-grained differences in sentences that overlap in content. consider the following english and french sentences, sampled from the wikimatrix parallel corpus. while they share important content, highlighted words convey meaning missing from the other language: en alexander muir's "the maple leaf forever" served for many years as an unofficial canadian national anthem. fr alexander muir compose the maple leaf forever (en) qui est un chant patriotique pro canadien anglais. ## unsupervised divergence detection we introduce a model based on multilingual bert (mbert) to distinguish divergent from equivalent sentence-pairs (section 3.1). in the absence of annotated training data, we derive synthetic divergent samples from parallel corpora (section 3.2) and train via learning to rank to exploit the diversity and varying granularity of the resulting samples (section 3.3). we also show how our model can be extended to label tokens within sentences (section 3.4). ## rationalized english-french semantic divergences we introduce the rationalized english-french semantic divergences (refresd) dataset, which consists of 1,039 english-french sentence-pairs annotated with sentence-level divergence judgments and token-level rationales. figure 2 shows an example drawn from our corpus. our annotation protocol is designed to encourage annotators' sensitivity to semantic divergences other than misalignments, without requiring expert knowledge beyond competence in the languages of interest. we use two strategies for this purpose: task description an annotation instance consists of an english-french sentence-pair. bilingual participants are asked to read them both and highlight tokens in each sentence that convey meaning not found in the other language. for each highlighted span, they pick whether this span conveys added information ("added"), information that is present in the other language but not an exact match ("changed"), or some other type ("other"). those fine-grained classes are added to improve consistency across annotators and encourage them to read and compare the text closely. finally, participants are asked to make a sentence-level judgment by selecting one of the following classes: "no meaning difference", "some meaning difference", "unrelated". participants are not given specific instructions on how to use span annotations to make sentence-level decisions. furthermore, participants have the option of using a text box to provide any comments or feedback on the example and their decisions. a summary of the different span and sentence labels along with the instructions given to participants can be found in a.3. curation rationale examples are drawn from the english-french section of the wikimatrix corpus @xcite . we choose this resource because (1) it is likely to contain diverse, interesting divergence types, since it consists of mined parallel sentences of diverse topics which are not necessarily generated by (human) translations, and (2) wikipedia and wikimatrix are widely used resources to train semantic representations and perform cross-lingual transfer in nlp. we exclude obviously noisy samples by filtering out sentence-pairs that a) are too short or too long, b) consist mostly of numbers, c) have a small tokenlevel edit difference. the filtered version of the corpus consists of 2,437,108 sentence-pairs. quality control we implement quality control strategies at every step. we build a dedicated task interface using the brat annotation toolkit @xcite (figure 2 ). we recruit participants from an educational institution and ensure they are proficient in both languages of interest. specifically, participants are either bilingual speakers or graduate students pursuing a translation studies degree. we run a pilot study were participants annotate a sample containing both duplicated and reference sentence-pairs previously annotated by one of the authors. all annotators are found to be internally consistent on duplicated instances and agree with the reference annotations more than 60% of the time. we solicit feedback from participants to finalize the instructions. inter-annotator agreement (iaa) we compute iaa for sentence-level annotations, as well as for the token and span-level rationales ( disagreements mainly occur between the "no meaning difference" and "some meaning difference" classes, which we expect as different annotators might draw the line between which differences matter differently. we only observed 3 examples where all 3 annotators disagreed (tridisagreements), which indicates that the "unrelated" and "no meaning difference" categories are more clearcut. the rare instances with tridisagreements and bidisagreements-where the disagreement spans the two extreme classes-were excluded from the final dataset. examples of refresd corresponding to different levels of iaa are included in a.5. ## experimental setup data we normalize english and french text in wikimatrix consistently using the moses toolkit @xcite , and tokenize into subword units using the "berttokenizer". specifically, our pre-processing pipeline consists of a) replacement of unicode punctuation, b) normalization of punctuation, c) removing of non-printing characters, and d) tokenization. 2 we align english to french bitext using the berkeley word aligner. 3 we filter out obviously noisy parallel sentences, as described in section 4, curation rationale. the top 5,500 samples ranked by laser similarity score are treated as (noisy) equivalent samples and seed the generation of synthetic divergent examples. 4 we split the seed set into 5,000 training instances and 500 development instances consistently across experiments. results on development sets for each experiment are included in a.7. models our models are based on the hugging-face transformer library @xcite . 5 we fine-tune the "bert-base multilingual cased" model (devlin et al., 2019), 6 and perform a grid search on the margin hyperparameter, using the synthetic development set. further details on model and training settings can be found in a.1. evaluation we evaluate all models on our new refresd dataset using precision, recall, f1 for each class, and weighted overall f1 score as computed by scikit-learn (pedregosa et al., 2011). 7 6 binary divergence detection we evaluate divergent mbert's ability to detect divergent sentence pairs in refresd. ## finer-grained divergence detection while we cast divergence detection as binary classification in section 6, human judges separated divergent samples into "unrelated" and "some meaning difference" classes in the refresd dataset. can we predict this distinction automatically? in the absence of annotated training data, we cannot cast this problem as a 3-way classification, since it is not clear how the synthetic divergence types map to the 3 classes of interest. instead, we test the hypothesis that token-level divergence predictions can help discriminate between divergence granularities at the sentence-level, inspired by humans' use of rationales to ground sentence-level judgments. ## conclusion we show that explicitly considering diverse semantic divergence types benefits both the annotation and prediction of divergences between texts in different languages. we contribute refresd, a new dataset of wiki-matrix sentences-pairs in english and french, annotated with semantic divergence classes and tokenlevel rationales that justify the sentence level annotation. 64% of samples are annotated as divergent, and 40% of samples contain fine-grained meaning divergences, confirming that divergences are too frequent to ignore even in parallel corpora. we show that these divergences can be detected by a mbert model fine-tuned without annotated samples, by learning to rank synthetic divergences of varying granularity. inspired by the rationale-based annotation process, we show that predicting token-level and sentence-level divergences jointly is a promising direction for further distinguishing between coarser and finer-grained divergences.