S im QA : Detecting Simultaneous MT Errors through Word-by-Word Question Answering (2022)

Abstract: Detractors of neural machine translation admit that while its translations are fluent, it sometimes gets key facts wrong. This is particularly important in simultaneous interpretation where translations have to be provided as fast as possible: before a sentence is complete. Yet, evaluations of simultaneous machine translation (SimulMT) fail to capture if systems correctly translate the most salient elements of a question: people, places, and dates. To address this problem, we introduce a downstream word-by-word question answering evaluation task (SimQA): given a source language question, translate the question word by word into the target language, and answer as soon as possible. SimQA jointly measures whether the SimulMT models translate the question quickly and accurately, and can reveal shortcomings in existing neural systems—hallucinating or omitting facts.

## introduction recent advances in simultaneous machine translation (simulmt) hold the promise of breaking language barriers by democratizing simultaneous interpretation, a demanding form of real-time translation which currently requires trained human experts @xcite . however, progress in this field is hampered by evaluation challenges. since simulmt systems must output translations in real time before the input sentence is complete, evaluation methods should account for the timeliness of outputs, in addition to capturing the dimensions of translation quality that also matter in traditional mt settings @xcite . adequate evaluation on simulmt is critical to build models that help a user decide when to take the right action quickly and correctly in a multilingual situation. since human evaluation is too costly (and slow) to guide system development, evaluation is figure 1 : overview of the question answering evaluation for simulmt (simqa). given a word-by-word qa task, the simulmt system produces translations of questions word by word. a good simulmt system will allow the downstream qa system to answer correctly as quickly as possible. to address these issues, we propose to evaluate simulmt by measuring how well it helps execute a cross-lingual word-by-word question answering task (simqa). this task measures the quality of timely adequacy, or immediacy of adequacy, of simulmt more directly than existing evaluations. given a source language question, the task consists of translating the question word-by-word into the target language, and answering it as soon as possible. this evaluation is inspired by the use of question answering (qa) to evaluate mt systems in full-input settings, where a correct answer is considered an indication that the salient source content was adequately translated @xcite @xcite . unlike prior work, we evaluate translations on a word-by-word basis rather than waiting for a complete translation of the source, thereby evaluating whether source content is immediately conveyed adequately enough for a qa system to answer correctly. we introduce a novel cross-lingual quizbowl dataset xqb, 1 to systematically evaluate polish-english and spanish-english simulmt systems of varying latency, and compare them against fullinput mt that translates complete inputs. by jointly accounting for timeliness and translation quality, the simqa evaluation reveals different trends that complement full-input mt and qa evaluations based on complete inputs, and also it can reveal critical simulmt errors by monitoring qa quality over time. ## simultaneous question answering evaluation framework in this section, we describe the two components of our framework, simultaneous translation (section 3.1) and the word-by-word qa task (section 3.2), and how they are integrated (section 3.3). ## main results figure 3 shows mt and qa metrics across mt systems on the polish-english task. mt metrics. as expected, traditional metrics of mt quality all increase monotonically as the simulmt system waits longer before translating in figure 3b . translation quality increases steeply by 10 bleu points from wait-3 to wait-9 and plateaus around 48 bleu with longer wait times, 4 bleu points below the translation quality of the full-input system that uses the same architecture and training data as simulmt ("transf"). bertscore and comet show consistent trends. the quality of simulmt models falls roughly between that of full-input smt, which is close to or slightly above wait-3 depending on the metric, and that of full-input nmt systems, with our transformer followed by google translate. simqa results. by jointly capturing how mt quality and latency impact the probability of winning, the qa evaluation in figure 3a paints a different picture. while ew and ewo initially improve with longer latency, the translation quality gains have diminishing returns on qa as latency increases. the ew scores shows that the wait-9 models have better timely adequacy than wait-12, even though wait-12 models have access to three additional source tokens, and ew scores decrease further with higher latency. the ewo scores follow the same trend but peak with wait-6, indicating that timely adequacy could be obtained with a lower latency given an oracle buzzer. full-input mt. we compare simulmt systems with full-input mt. the scores of full-input mt systems are represented by horizontal lines in figure 3 . first, we compare the qa accuracy of these models based on the complete translation of the input question, thus ignoring the consideration of timeliness (figure 3a ). in this setting, simulmt systems are less accurate than full-input mt systems by at most 6 points, and the wait-15 setting is on par with full-input mt systems. while nmt outperforms smt by a wide margin according to mt evaluation metrics, their accuracy is close on the qa task, falling roughly 2 points below the upper-bound achieved with human translation. next, we return to the simqa evaluation of timely adequacy via ew and ewo scores. we compute these scores for full-input mt models by assuming that they are wait-k models where k is ∞, and calculate the source buzzing position and corresponding ew score accordingly. in this setting, the best simulmt system (wait-9) outperforms full-input mt systems according to ew. given an oracle buzzer (ewo), all wait-k models improve over full-input mt, and four of them even improve over human translation. while the accuracy of transformer and smt lag slightly behind that of google translate on full input accuracy, the transformer and smt systems are the best and second best of full-input mt according to ewo and ew scores, indicating that our local models' timely adequacy is higher than that of a commercial system. taken together, these results confirm the potential of simqa to evaluate the joint impact of adequacy and timeliness. this is further augmented by the results of the experiment with xqbes which shows consistent trends in appendix e. design choices for simulmt systems, namely the wait-k policy, have a clear impact on simqa metrics. the discrepancy between the simqa and mt results confirms that traditional simulmt evaluation does not suffice to assess systems' ability to convey core source information correctly and quickly. questions and report ewo for the wait-k models. while the rnn guesser outperforms bert and elas-ticsearch, ewo follows consistent trends with all guessers as dal increases, with peak timely adequacy for wait-6. this confirms the soundness of our evaluation approach. the details of the experiment and full results of each qa metric are in appendix c and figure 7 . ## step-wise visualization of simulmt errors this section moves away from aggregate evaluations and analyzes the behavior of the simqa system step-by-step on a single question at a time. we show that monitoring the quality of question answering over time can reveal translation errors. method. figures 5a and 5b show how the answers of the qa system and their quality change as the simqa system consumes the input question. the x-axis represents the relative position of a guess based on the question on the source side. the target word where the qb system generates a guess can be mapped to the location of the part of the source question that has been consumed by simqa. we normalize by the length of the source question to obtain the relative character position used on the x-axis of our plots. the y-axis uses mean reciprocal rank (mrr) to represent the goodness of the current answer. reciprocal rank is the inverse of the position of the highest ranked answer, and if there are no correct answers within the top n then the reciprocal rank is 0. (higher is better) we use log mrr for more even plots. here n is set to 50, and the lowest value of log mrr is -4. mrr can also be used for overall system evaluation, which results in similar trends as those described above with the ew and ewo metrics (refer to the appendix d for details). hallucination. in the first example (figure 5a and at position (1), marked in figure 5a and bleu does reveal translation errors, but it treats every word the same and is calculated after the full output, while simqa focuses on essential words that change the answer prediction in real-time. for example, when "półpłaszczyzna" is erroneously translated as "southern" instead of "half-plane" in figure 5a , qa is misled and guesses wrong, but ignores minor errors such as excluding "the" or "a", while both cases have same impact on bleu. under translation. in figure 5b , there is a flat span where log mrr dosen not change for more than half the question. this is because even though the wait-3 models consume source words continuously, it does not output any new words due to an under-translation error. as can be seen in these two examples illustrate that qa provides useful signals to pinpoint critical translation errors in simulmt outputs, and suggest that simqa might provide a strategy for more systematic error detection on the fly in future work. in 1899, halford mackinder claimed that he was the first person to climb this mountain, and the people of kikuyu named this mountain "mount kirinyaga", meaning "the one with the ostrich", while the people of maasai believe that their ancestors descended from this mountain at the beginning of time. @xcite , while @xcite use qa generation to generate questions from references and extract an answer. while these papers show the promise of qa for evaluating mt, they focus on the full-input setting. qa has also been used as an extrinsic evaluation for text summarization systems @xcite @xcite to complement existing evaluation metrics, which are notoriously insensitive to factual inaccuracies and inconsistencies. ## conclusion the main motivation of simultaneous translation (simulmt) is to convey the core meaning in a source sentence as quickly as possible. however, current simulmt research measures the quality of output on "finalized" translations and separately considers latency measures, which cannot fully reflect the quality of timely adequacy. this paper introduces a cross-lingual word-byword question answering task (simqa) to quantify the timely adequacy of simulmt more directly. our experiments on a cross-lingual quizbowl task show that some wait-k simulmt systems can win more often than full-input mt systems with higher bleu score, and can outperform full-input human translations with an oracle buzzer. overall our simqa results complement intrinsic qa and mt metrics by jointly accounting for timeliness and translation quality, and suggest that simqa can diagnose critical simulmt errors on the fly. these results represent a first step toward broadening the scope of simulmt evaluation to more directly assess its usefulness in specific scenarios. in future work, we would like to evaluate on more languages and investigate the impact of varying each component of the simqa pipeline, including using alternate simulmt architectures, and providing simulmt outputs to human contestants in addition to automatic qa systems. finally, we would like to expand the modality to speech for more realistic simulmt and qa.