Pulling Out All The Full Stops: Punctuation Sensitivity in Neural Machine Translation and Evaluation (2023)

Abstract: Much of the work testing machine translation systems for robustness and sensitivity has been adversarial or tended towards testing noisy input such as spelling errors, or non-standard input such as dialects. In this work, we take a step back to investigate a sensitivity problem that can seem trivial and is often overlooked: punctuation. We perform basic sentence-final insertion and deletion perturbation tests with full stops, exclamation and questions marks across source languages and demonstrate a concerning finding: commercial, production-level machine translation systems are vulnerable to mere single punctuation insertion or deletion, resulting in unreliable translations. Moreover, we demonstrate that both string-based and model-based evaluation metrics also suffer from this vulnerability, producing significantly different scores when translations only differ in a single punctuation, with model-based metrics penalizing each punctuation differently. Our work calls into question the reliability of machine translation systems and their evaluation metrics, particularly for real-world use cases, where inconsistent punctuation is often the most common and the least disruptive noise.

## introduction and related work since the advent of the transformer models @xcite , machine translation (mt) has seen tremendous improvement in performance, with several claims of parity with human translations @xcite @xcite . however, one issue that is common to most deep learning models but does not hinder humans is sensitivity to small changes in the input, or a lack of robustness. robustness in machine translation refers to the ability of the models to produce consistent translations that preserve the meaning of the source sentence regardless of any noise in the input (heigold tan et al. (2020) when is the suspended team scheduled to return? when are the suspended team schedule to returned? wallace et al. (2020) did you know that adversarial examples can transfer to production models did you know that adversarial examples can transfer to production models siehe siehe siehe siehe siehe siehe siehe there has also been some work on mt evaluation metric robustness that has included similar pertur-bations at the character and word-level, and other linguistic phenomena such as synonyms, named entities, negation, numbers, and others @xcite @xcite . however, @xcite argue that many of these perturbations do not preserve the meaning on the source side. they propose that "meaningpreserving" perturbations should be limited to nearest neighbours in the embedding space and out-ofvocabulary word-internal character swaps. in this work, we take a further step back from meaning-preserving spelling and grammatical perturbations, and ask: are machine translation models robust to trivial changes in sentence-final punctuation? are the metrics used to evaluate machine translation robust to the same changes? to investigate this, we test basic punctuation variation for which robustness may have been taken for granted. we perform simple sentence-final punctuation perturbations, restricting the experiments to two settings: insertion and deletion. mimicking a very common form of natural noise, we insert or delete full stops, exclamation marks and question marks at the end of the input sentence ( §2; see we test commercial mt systems from google, deepl and microsoft on 3 language pairs from across resource levels and scripts: german (de), japanese (ja) and ukrainian (uk) to english (en). these systems are intended for real-world use, and can therefore be expected to already be robust to common noise in real-world data. we first investigate whether commonly used evaluation metrics are robust to our perturbations, in order to ensure that our subsequent evaluation of the mt systems is fair ( §3). we find that both stringbased and model-based evaluation metrics are not robust to trivial sentence-final punctuation perturbations, significantly penalizing text with mismatched full stops, question marks or exclamations, sometimes more than text with more severe perturbations such as insertion or deletion of random characters. based on these results, we deviate from the stan-dard robustness testing regime of perturbing the inputs and expecting the translations of both the original and the perturbed source text to match exactly. in the mt setting, adding a punctuation to the source text can naturally induce the model to also produce the corresponding punctuation in the translation. we therefore reset the punctuation changes in the translations in order to perform evaluation, and call for a review of standard mt robustness evaluation in such settings. more importantly, we show that even commercial machine translation systems are extremely sensitive to trivial punctuation changes, particularly in languages such as japanese and ukrainian ( §4). we show that both insertion and deletion of punctuation causes performance drops, which indicates that models may be biased to expect (or not expect) punctuation in certain types of sentences. we conduct a manual analysis and find that in more severe cases, a mere punctuation change can cause complete changes in the meaning of the translation or introduce hallucinations such as negation, with less severe changes including pronouns, named entities, tense, number, and others ( §5). @xcite provide some common examples of punctuation variation in real-world data and demonstrate how dependency parsers are sensitive to such punctuation differences. @xcite demonstrate the sensitivity of neural models to punctuation in natural language inference tasks. though there has also been some work on punctuation-based perturbation for machine translation @xcite , the tendency has been to make more extreme perturbations than we adopt. unlike previous work, we do not combine all punctuation changes into one bucket, and instead analyse each punctuation separately. we find that models are more sensitive to some punctuation than others. we also unify the usually independent work on machine translation robustness and evaluation metric robustness, and adjust our evaluation based on our observations. our work exposes serious real-world use-case implications and serves to show that while great strides have been made in both machine translation and its evaluation, we are a long way from building systems that are reliable for real-world use. ## test set creation in this section, we describe the original test sets and perturbation operations we perform to build our test sets. our perturbations reflect natural noise in punctuation occurrence: we only insert or delete punctuation such as full stops, exclamation marks and question marks from the ends of sentences. ## evaluation metrics before we evaluate the machine translation systems on our punctuation perturbation test sets, we first evaluate the evaluation metrics themselves to see if they are robust to these variations. this metaevaluation is crucial; if the metrics are not reliable, we cannot be sure if changes in the scores are due to changes in translation content. we include the string-based metric bleu @xcite for convention, and based on the recommendations from @xcite , we use chrf @xcite , which is another string-based metric, and comet @xcite , which is a model-based metric shown to have high correlations with human judgements, and also include bleurt-20 @xcite and bertscore @xcite . metric versions can be found in appendix a.2. ## machine translation experiments we now test the publicly available commercial machine translation systems of google, deepl and microsoft through their paid apis on our test sets. 4 some of these commercial systems have previously been claimed to have reached human parity @xcite . commercial systems are generally expected to deal with nonstandard inputs as they are targeted for real-world use cases. we therefore expect that these systems have already been trained to be somewhat robust to various kinds of input noise. for the insertion tests, we compare the translation of the original source text without punctuation against the translation of the perturbed source with sentence-final punctuation. for deletion tests, we compare the translation of the original source text with punctuation against the translation of the perturbed source without sentence-final punctuation. ## analysis and discussion some examples of translation changes caused by the perturbations are given in reordering. often, inserting or deleting punctuation leads to a reordering of the words in the sentence. in many cases the reordering leads to mostly similar but slightly off translations (example 4), with some cases causing significant differences in meaning (example 8). while we might expect punctuation perturbation to ideally cause no other changes in translation apart from the difference in punctuation itself, there could be cases of valid translation changes caused by the perturbation. for example, while "1) heben sie die autorisierung des lesegeräts auf" is originally translated as "1) deauthorize the reader", adding a question mark does not produce "1) deauthorize the reader?" but instead "1) are you deauthorizing how many times a day should i send line? -8.5 +14.1 25.9 the reader?". this word reordering for an interrogative sentence, typical particularly for english, can be considered a valid change even though the chrf (65.5 -→ 52.7) and comet (28.2 -→ -25.8) scores drop. there are also cases when the resultant reordering actually improves the scores of the translation despite being wrong e.g., adding a question mark to "und ich muss nochmal versandkosten zahlen" changes the translation from "and i have to pay shipping again" to "and do i have to pay shipping costs again?" (instead of "and i have to pay shipping again?") and improves both chrf (17.5 -→ 23.4) and comet (26.3 -→ 45.6) scores, presumably due to the presence of the word "costs" that now matches the reference ("and i still need to pay the delivery costs"). similarly for question mark deletion, removing the question mark from "заняття в понедiлок i середу вiдрiзняються?" changes the translation from "are monday and wednesday classes different?" to "monday and wednesday classes are different", dropping the chrf (76.2 -→ 74.9) and comet (91.0 -→ 83.7) scores. expecting translations of both original and perturbed source texts to match is a standard evaluation setting for robustness tests, even for more severe perturbations resulting in drastic changes and out-of-vocabulary inputs (see however, there are several cases where the interrogative nature of the source is not dependent on the question mark and the model correctly produces a translation that is also interrogative but different. for example, deleting the question mark from "ありますか？" changes the translation from "is there?" to "do you have". example 12 shows another case where the model correctly recognizes the perturbed source as a question, but produces a significantly different translation. example 5 and example 6 are also cases of translation differences that are more severe than reordering. ## conclusions in this work, we unite the robustness evaluation of both machine translation systems and their evaluation metrics, and discuss ways in which both fail to be adequately robust to trivial punctuation change. this shows that models and metrics are in fact far more sensitive and a lot less reliable in real-world use cases than is commonly expected. we show that both metrics and machine translation systems treat each punctuation differently, with machine translation systems showing associations between punctuation and sentence styles. we also highlight the implications of these sensitivities for robustness research and evaluation for machine translation. although it may not necessarily be a hard task to train systems that are robust to punctuation, our goal is to highlight one of the issues that has possibly been overlooked due to its triviality. we hope that future research in robustness, evaluation metrics and machine translation accounts for these sensitivities while performing evaluation and model training.