AMTA Best Thesis Award Abstract: Detecting Fine-Grained Semantic Divergences to Improve Translation Understanding Across Languages (2024)

Abstract: In this thesis, we focus on detecting fine-grained semantic divergences—subtle meaning differences in sentences that overlap in content—to improve machine and human translation understanding.

## introduction a widespread hypothesis adopted by machine translation research is that a source text and its (human) translation-or parallel text-are equivalent in meaning. in principle, this hypothesis drives the way we think about our models when designing our training losses and our evaluation metrics and protocols. yet when humans translate, they make lexical decisions influenced by cultural and situational aspects of language that break the hypothesis of meaning equivalence in nuanced ways @xcite . consider the english and french sentences above drawn from wikimatrix @xcite , a corpus that is routinely used to train translation systems and is perceived as highly parallel. while they share important content, highlighted words convey meaning missing from the other language (i.e., served for many years) or content reflecting fine-grained semantic divergences between concepts that, although related, are not equivalent (i.e., national anthem vs. patriotic song). regardless of why such subtle divergences exist in parallel texts, we hypothesize that they matter for machine translation systems-as they yield challenging training samples-and for humanswho might benefit from a nuanced understanding of the source. thus, in this line of work, we argue that quantifying fine-grained divergences is crucial to improve both machine and human translation understanding across languages. in what follows, we start by introducing methods for detecting fine-grained divergences in the wild @xcite . as we will see, such methods lay the foundation for studying their connection to machine translation models @xcite @xcite and human evaluations pipelines @xcite . ## detecting fine-grained divergences in our first piece of work, we start our exploration by asking: how frequent are semantic divergences in parallel texts? our goal is to address challenges in detection of fine-grained divergences within bitexts in two settings: human annotation and automatic prediction @xcite . starting with human annotation, we contribute the rationalized english-french semantic divergences corpus, based on a novel divergence annotation protocol that exploits rationales to improve annotator agreement. annotations on the collected dataset reveal that semantic divergences are surprisingly frequent, comprising 40% of samples in a cor-pus consisting of wikipedia-mined translations, and are perceived as highly parallel. after establishing that divergences exist, we explore computational methods for detecting them at scale, crucially, without assuming access to gold supervision. to that end, we introduce a contrastive loss designed to make a multilingual language model sensitive to subtle cross-lingual differences between linguistically motivated synthetic samples. despite being trained only on synthetic samples, we show that our model detects fine-grained divergences accurately, outperforming a strong sentencelevel similarity model @xcite . ## improving machine translation equipped with the tools that allow us to study divergences at scale we now ask: how do fine-grained divergences impact neural machine translation? we contribute a controlled empirical analysis on several aspect of nmt models that are exposed to different types and amounts of divergences at training time. our findings reveal that small divergences hurt translation accuracy and confidence of nmt models, and crucially are one of the root causes that lead to neural text degeneration, i.e., translation outputs that are incoherent or get stuck in repetitive loops @xcite . drawing from those findings, a natural question arises: how can we mitigate the negative impact of divergences on nmt? to this aim, we explore two orthogonal strategies. our first strategy intervenes in the training assumption of translation equivalence in parallel texts and aims to model divergences explicitly. drawing from our prior work on automatically detecting divergences, we propose a divergentaware framework-div-factorized-that incorporates token-level divergence signals into nmt training @xcite . our second strategy proposes an orthogonal mitigation direction: instead of altering training to model divergences closely, we aim to automatically re-write divergent samples to yield more equivalent translations. in this direction, we introduce two approaches to solve this problem in the lack of supervised data. our first approach-equiv sem-div-relies on synthetic translations, i.e., translations generated by mt, that selectively replace divergent references under a semantic equivalence con-dition @xcite . our extensive evaluations on both intrinsic and extrinsic tasks for two medium-resource languages show that this approach is capable of revising divergences in parallel texts, given synthetic translations of sufficient quality. in our subsequent work, we address this problem in low-resource conditions via introducing-bitextedit-an editing-based model that, given a parallel text, edits one of the two references to generate a refined version of the original as necessary. our editing model is trained solely on synthetic supervision via leveraging recent advances in bitext mining based on massively multilingual sentence embeddings @xcite and is shown to utilize divergences more effectively in heterogeneous data scenarios @xcite . ## assisting humans to detect translation differences in meaning after exploring how detecting semantic divergences helps us improve machine translation understanding, we finally turn to analyze ways of assisting humans in understanding and detecting translation differences. although detecting divergences in parallel texts as a binary classification task, i.e., equivalence vs. divergence, is found to be sufficient for augmenting and improving nmt, we hypothesize that other task framings that shed more light on the nature of divergences are needed to improve human translation understanding. in this direction, our last piece of work asks: how can we explain semantic divergences in a human-interpretable fashion? to that end, we equip divergence detectors with the ability to indicate not just whether divergences exist but also tell us where the translation differences reside @xcite . drawing on social science studies, we introduce a method to extract contrastive phrasal highlights that explain the predictions of our divergent detectors by explicitly modeling the relationships between the contrasted texts. we contribute evidence that contrastive phrasal highlights match human-provided rationales of divergence better than standard highlighting approaches, and more importantly, they assist bilingual speakers in annotating fine-grained divergences, easing the need to ask for human rationales. finally, we show that contrastive highlights could help humans detect critical errors due to local mistranslations in machine-translated texts.