T e CS : A Dataset and Benchmark for Tense Consistency of Machine Translation (2023)

Abstract: Tense inconsistency frequently occurs in machine translation. However, there are few criteria to assess the model’s mastery of tense prediction from a linguistic perspective. In this paper, we present a parallel tense test set, containing French-English 552 utterances. We also introduce a corresponding benchmark, tense prediction accuracy. With the tense test set and the benchmark, researchers are able to measure the tense consistency performance of machine translation systems for the first time.

## introduction translation tools are often found in a variety of social situations to enable cross-linguistic communication. tenses are used to express time relative to the moment of speaking. human translators frequently pay close attention to tense correspondence @xcite . similarly, machine translation (mt) systems are supposed to maintain temporal consistency between the original text and the predicted text to avoid misunderstandings by users. however, accurately keeping the tense consistency is undoubtedly difficult. taking french-english (one of the most classic language pairs for mt) as an example in in fact, this is not an isolated case. you can also find several examples in appendix b. besides. the translation mechanics may not the only reason leading to tense inconsistency. the corpora matter as well. for example, we have extracted 20,000 pairs english-french parellel sentences from the widely used dataset europarl @xcite , and ## annotation rules and tools as the first work of the mt tense study, we choose english-french, one of the most classic language pairs of mt, to construct the dataset foot_0 . tense, the dominant topic of our research, is a combination of tense and aspect. in the modern grammar system of english, "a tense system is a system associated with the verb where the basic contrasts in meaning have to do with the location in time of the situation, or the part of it under consideration" @xcite . the modern grammatical system divides tense into present and preterit based on the inflections added to the end of verbs, and the aspect into perfective and progressive on the state where an action is @xcite . while this tense classification system is too crude for daily life, we therefore apply the following classification methods. on the one hand, we classify the tenses according to the macro-temporal interval of the action into three major time intervals, namely present, past and future tenses; on the other hand, we classify the tenses according to the state of the action into general, progressive and perfect aspects. hence, 9 kinds of tenses are born through combining the three tenses and the three aspects. french and english belong to the same indo-european language family and share many similarities in various respects. the main difference is that in french there is another grammatical point called mode, part of which is like the aspect in english. in terms of tenses, we will generally discuss the tenses in the indicative mode of french and will describe the others later in this section. in the following, if there is no mode qualifier before a tense, it is by default in the indicative mode. careful identification and comparison of the subdivided tenses in the three main tense intervals, english and french, reveals a very similar usage of the tenses, as sum-marised in when discussing tenses from a semantic point of view, the modes also need to be taken into account. the grammatical correlations between french and english modes are quite complicated. considering the corresponding grammatical expressions of 2 modes strongly related to tense, conditionnel and subjonctif, in french rely on the usage of modal verbs, we introduce modal verbs to simplify the distinguishment of the modes. based on these grammatical rules, we merge the nine common tenses in english into seven categories that correspond reasonably and rigorously to french, namely the 6 tense categories of past/present/future + simple/perfect and statements containing modal verbs that correspond to the french subjonctif and conditionnel tenses. we construct an automatic annotation method based on the spacy package @xcite . first, we label the grammatical components of each word in the sentence based on the spacy package, and then we define and compare the grammatical structures of the verb phrases with the structures of each tense classification to derive the sentence tense labels. during this process, to simplify the annotation process and better correspond with french futur proche tense, we classify the expression 'be going to do', grammatically in future tense, into the present tense, just like expressions 'be about to do' and 'be + verb progressive', whose stucture are in present tense but the real meaning is about the close future. also, a sentence may have several tense structures, in this case, the tense label consists several tenses. for example, the label of the sentence 'so it is in that spirit that we have made this change.' is 'present+preperfect'. ## experimental results to measure the tense consistency performance of different systems, we introduce a benchmark called tense (prediction) accuracy, as shown in eq. ( 1 ). to verify the validity of our tense corpus, the following approach was adopted: to begin with, 100, 000 parallel utterance pairs from the eu-roparltr (containing 201, 374 pairs) mentioned in section 3.1 were extracted as the tense-rich train set, and 100, 000 parallel utterance pairs from the europarl corpus @xcite were extracted as the tense-poor train set. there were no overlapping utterances between the latter and the former. we performed the same preprocessing procedure, including data cleaning, tokenization and bpe coding. we then trained four pairs of french-english nmt systems with different architectures based on fairseq @xcite , where two systems in each pair differed only in the train set. after this, we summarized the scores evaluated by sacre-bleu @xcite and comet @xcite and tense prediction accuracies of the eight systems on different test sets. we have applied three types of test sets: our tense set, the europarl test set and the wmt15 test set. the europarl test set contains 3,000 parallel utterance pairs drawn from the europarl corpus, the exact same field of train set, while the wmt15 is a test set for the wmt15 @xcite , deriving from data in the different field of train set. besides, we also apply our approach to mesure the tense consistency performance of several business translators, includ-ing bing translator, deepl translator and google translator. the results are listed in ## conclusion we presented the french-english parallel tense test set and introduced the corresponding benchmark tense prediction accuracy, providing a brand-new approach to measure the tense consistency performance of machine translation systems. this test set firstly focuses on the tense prediction ability, posing a new dimension to improve the mt quality. in the future, we will endeavour to generalize the test set to other languages. considering there are statements like "the use of tense a in language x is equivalent or similar to the use of tense b in english" in grammar books of other languages @xcite , even across language families(gadalla, 2017) and human translators also apply such rules @xcite , we are confident in taking this forward.