Long to reign over us: A Case Study of Machine Translation and a New Monarch (2023)

Abstract: Novel terminology and changes in terminology are often a challenge for machine translation systems. The passing of Queen Elizabeth II and the accession of King Charles III provide a striking example of translation shift in the real world, particularly in translation contexts that have ambiguity. Examining translation between French and English, we present a focused case-study of translations about King Charles III as produced both by publicly-available MT systems and by a neural machine translation system trained specifically on Canadian parliamentary text. We find that even in cases where human translators would have adequate context to disambiguate terms from the source language, machine translation systems do not always produce the expected output. Where we are able to analyze the training data, we note that this may represent artifacts in the data, raising important questions about machine translation updates in light of real world events.

## introduction with the passing of queen elizabeth ii on september 8, 2022, king charles iii became the first king of canada in over 70 years. given official bilingualism (english and french) in canada, this raised a natural question of how machine translation (mt) systems -particularly those trained on data collected from canadian government sources, which forms a disproportionately large amount of publicly available data for this language pair @xcite -might perform on terminology related to the new sovereign. we hypothesized that systems trained on relatively recent parliamentary text might produce errors due to both linguistic features of french and english as well as the paucity of references to kings in the training data. we expand on this, showing that not only is this the case for mt systems trained solely on canadian parliamentary data; these errors also appear (albeit less frequently) in the output of large publicly available mt systems. in this work we will distinguish between errors, where context (and world knowledge of the two sovereigns in question) would be sufficient for a human translator to translate correctly, and other potential artifacts of the data where there is insufficient context at the sentence level to translate unambiguously. this work can be viewed as a narrowly-focused miniature challenge set @xcite , aiming to examine a specific intersection of mt challenges through a recent known example: world knowledge (or lack thereof) and changes in the state of the world, dataset imbalances, a subset of the different ways in which grammatical gender and the pronouns and inflections used for the referent affect translation for this language pair, and asymmetries in translation ambiguity. foot_0 by keeping this tight focus, we are able to point out some areas in which mt is not yet "solved," even for this highly-resourced language pair. on the other hand, this tight focus on both the language pair and the specific case of text about these two monarchs limits the scope of what this work addresses; we provide a brief discussion of more general related work in the following section and additional notes in the limitations section. ## related work how to incorporate (new or updated) terminology into mt has long been an area of interest, from compound noun splitting and subword models @xcite to rapidly incorporating terminology from external sources like lexicons or dictionaries @xcite . recently, there has been a focus on handling novel terminology resulting from the covid-19 pandemic, including a shared task @xcite , the release of targeted datasets @xcite , and evaluations of mt performance on related terminology @xcite . there has been work on bias, imbalance, and gender-inclusivity in coreference resolution @xcite @xcite , on linguistic gender in mt ( @xcite , on incorporating coreference into mt to improve pronoun and gender inflection translation @xcite , and on benchmarks for and analysis of gender in mt @xcite . there has also been analysis of and attempts to mitigate language pair asymmetries in linguistically conveyed information, such as by incorporating factors @xcite @xcite . here, while some of our examples might benefit from such approaches, many would require additional context beyond the sentence. the topic of additional context in mt and its evaluation remains an open area @xcite @xcite , and within this realm there has been work specifically done on anaphora resolution @xcite . ## linguistic and grammatical notes in french, nouns are grammatically classed as masculine or feminine, and adjectives, articles, and determiners take inflected forms that agree with the nouns in terms of number and grammatical gender. the noun majesté (majesty) is feminine (f). the form of address sa majesté on its own is ambiguous to translate into english, as the feminine form of the third person singular possessive determiner sa agrees with the feminine noun majesté, without regard to the specific referent. depending on the referent's pronouns, sa could be correctly translated as various singular third person pronouns such as her, his, or their (singular; for plural their, the french source would be leurs majestés). without additional context, like the sovereign's title and name, we expect current mt systems to almost always produce her majesty as a translation, due to the preponderance of that translation in the data. the question arises: will mt systems use information about words like king/roi or the frequency with which the name charles is associated with masculine pronouns to produce translations like his majesty king charles iii? we anticipate more translation errors in the french-english translation direction, but examine both translation directions. ## experiments we collect a small amount of existing parallel text from several sources: the text of the prime minister's statement regarding king charles iii's accession to the throne, text from the canadian hansard (proceedings of the house of commons), and the royal anthem (god save the queen/king). 5 from these, we manually extract terms that vary in at least one language based on whether they would refer to queen elizabeth ii or king charles iii. this includes pronouns/determiners, adjectives and nouns that are grammatically marked for gender, and their names and titles. after translation, an author of this paper annotated each term in context to mark if it had been translated as expected. this was done via first automatically checking for string matches, followed by a manual check of all examples and notes on the cases where the expected translation was not found. ## discussion and conclusions perhaps unlike the introduction of covid-19 terminology (where an entire new topic or domain is rapidly introduced to the translation landscape), the accession of a new monarch may cause a shift in terminology in an existing domain, in this case one with 70 years of history. 7 as we expected, ambiguous terms tend to be translated in a way that likely corresponds to the imbalance in the training data (i.e., in the feminine, as referencing queen elizabeth ii); this also highlights the need for context (whether document-level or external) that is often required for accurate translation when there is an asymmetry in what information is (un)marked across a language pair. though they likely contain many canadian translations (see @xcite ), we cannot examine the public system training data, only the internal system data. while there are thousands of mentions of the queen in the hansard training data, there are only hundreds of references to kings, and only 36 instances of the term "his majesty" as compared to 882 instances of "her majesty". in our internal system, an additional consequence of this is subword segmentation of words like roi: the word was fully segmented into its three characters, rather than appearing as a single token in the vocabulary, likely contributing to observed errors. we also found that even in sentences that would have adequate context for a human translator (with knowledge of the forms of address for the two monarchs), the mt systems sometimes made errors. without examining the inner workings of the systems, the fact that this occurred primarily in sentences with references to both monarchs leaves open the question of whether this is a problem of erroneous implicit coreference resolution, imbalance in the training data around these particular terms, or a combination of the two. nevertheless, while accuracy in term translation is high overall, these striking errors where context ought to be sufficient serve as a warning that even in high-resource language pairs, history and data maintain a strong influence.