Quality Scoring of Source Words in Neural Translation Models (2022)

Abstract: Word-level quality scores on input source sentences can provide useful feedback to an end-user when translating into an unfamiliar target language. Recent approaches either require training special word-scoring models based on synthetic data or require repeated invocation of the translation model. We propose a simple approach based on comparing the difference of probabilities from two language models. The basic premise of our method is to reason how well each source word is explained by the target sentence as against the source language model. Our approach provides up to five points higher F1 scores and is significantly faster than the state of the art methods on three language pairs. Also, our method does not require training any new model. We release a public dataset on word omissions and mistranslations on a new language pair.

## introduction neural machine translations exhibit humanlike fluency and accuracy over many language pairs @xcite , and are increasingly getting embedded in several end-user applications. a hindrance to their deployment in safety-critical applications such as healthcare and diplomacy, is that unlike human translators, a machine translation system does not provide reliable feedback when parts of a source sentence are misrepresented in the generated translation. recent studies have discovered that nmt sentences while fluent are often inconsistent with the source @xcite . on the target side, calibrated confidence with the generated output is a standard way to give quality feedback to the user @xcite . however, when a user is unfamiliar with the target language, word-level confidence in the generated translation is not actionable. we instead propose that words in the source sentence be assigned indicators of the loss of fidelity, so end-users could reformulate the input for possibly better translation, much like the way a human translator could ask for clarification on misunderstood words. as a step in this direction, we attempt to solve the following problem: given a source sentence x with words x 1 , . . . x n and its translation ŷ from a translation model, assign each word x i in x a score q(x i ) that when low indicates that the word x i is either omitted or mistranslated in ŷ. figure 1 shows an example with a source english sentence and its translation to german. the words in the source sentence that are omitted or mis-translated are assigned relatively small scores. two categories of approaches have been explored so far. the first category @xcite train a supervised model to score words using synthetically inserted errors in parallel translation datasets. @xcite show that such synthetic training fails to identify real cases of omissions. they instead propose a contrastive conditioning approach on the premise that the likelihood of the target ŷ will be largest when the source is stripped off the words not well translated in ŷ. in addition to requiring that the stripped sentence be well-formed (enforced in the paper via a constituent parser), we show that their scoring is not as good at detecting omission and mistranslations compared to a more direct approach that we propose here. ## experiments we evaluate our method on three datasets and compare with two existing methods. ## conclusions and limitations we presented a simple method of assigning quality scores to source words in a translation model as the lift of the conditional probability from a reverse translation model, over the unconditional probability from the language model. in spite of its simplicity, our method is more accurate and faster than state-of-the-art methods, and does not require training any special models. we also release a new dataset of word-level omissions and mistranslations on a new language pair.