Up to Par? MT Systems Take a Shot at Sports Terminology (2025)

Abstract: We present a submission to the WMT25 test suite subtask, focusing on the capabilities of MT systems to translate sports-related language. Although many sports attract extensive media attention and feature a rich, polysemous language, often shaped by active neologism and community-driven translations, the sports domain has received relatively little focus in MT research. In English-Icelandic automatic translations, sports-specific vocabulary often appears to be mistranslated. Our test suite is designed to test whether this observation holds merit. We evaluate 34 systems, both automatically and manually, and find that sports language poses challenges to a varying degree for all the systems.

## introduction with the advent of large language models (llms), significant advances have been made in machine translation (mt) @xcite . while general translation capabilities are impressive for many high-resource and even some less-resourced languages, many mt systems commonly fail when dealing with specialized vocabulary or other rare peculiarities. we have noticed that some commonly used systems seem more prone to making errors when the topic is sports than for other common topics in the media. to investigate this further we built a test suite and submitted it to the wmt 2025 test suite subtask @xcite . we compile a list of segments discussing five different sports: basketball, football, golf, gymnastics and chess. these are international sports that are popular and have been played in iceland for decades. as a result, these sports often have a well-established and diverse vocabulary that many speakers need to agree upon. it is therefore important that different translation systems apply vocabulary that is known and in actual use by speakers in order to translate sports texts successfully. additionally, while "new" sports known from abroad start being played in iceland, the vocabulary may consist of a somewhat high proportion of loanwords adapted to icelandic (using, e.g., english-oriented stems with icelandic inflection). vocabulary for sports that have a long history in iceland, however, strikes a balance between using loanwords and new (and old) words applying other methods. our test suite is made available in plain text format with term annotations as well as evaluation code on github. 1 ## related work there is a long tradition of terminology work in iceland (see, e.g., @xcite . the icelandic terminology bank (íðorðabankinn) foot_1 at the árni magnússon institute for icelandic studies is the center of terminology work in iceland. as of august 2025, it hosts more than 70 terminologies and glossaries. most of these give translations of the terms in one or more languages, and english is usually one of them. however, only one of said terminologies and glossaries deal specifically with sports -that is the terminology on climbing. furthermore, even if a given translation system may contain, e.g., material from published dictionaries in its training data, its usefulness may be limited by the fact that some well-known words from sports may be missing. also, various words generally known by speakers of english are used in a very specific sense in sports, that may differ from the general use. such terms may have a different translation in icelandic altogether. an example is the noun paint, as in the paint is wet. in basketball, this refers to a specific place on a basketball court. whereas paint in a general sense could be translated as 'málning' in icelandic, in the basketball sense it can be translated as 'teigur, vítateigur', meaning something like 'free-throw lane', or even as 'undir körfunni', meaning 'below/under the basket'. we therefore find it interesting -and challenging -to look into vocabulary that may not be very well documented in lexicographic works. however, at least some of the vocabulary can be found in texts online, e.g., in corpora such as tímarit.is 3 (e.g., @xcite and the icelandic gigaword corpus 4 @xcite . @xcite study the performance of nmt systems at handling terminology and find that the systems are twice as likely as humans to commit terminology errors in the hansard corpus. they measure the accuracy by counting the occurrences and translations of unique terms. in the shared task on machine translation with terminologies at wmt 2023, three evaluation approaches were used @xcite : standard metrics to evaluate general translation quality, comet @xcite and chrf @xcite ; term success rate, or accuracy, was calculated by comparing machine-translated terms with their dictionary equivalents; term consistency was measured to investigate whether technical terms were translated uniformly across the entire corpus. at wmt 2024, two submissions specifically looked at problems in translating between english and icelandic. the genderqueer test suite @xcite ## test suite and translations the test suite consists of 300 segments in total, divided into five documents. the segments can be sentences or short paragraphs containing a few sentences. an example of a segment from gymnastics is shown in ## results we report on the results of our automatic evaluation approach and manual evaluation and compare the outcomes. in section 3.3 we discuss how the outputs of some systems were problematic. in the results tables, we tag the systems that may be at a disadvantage due to other reasons than just their capability in translating from english into icelandic. we use two tags: ■ for translations that had missing lines or repetitions, possibly because all segments for a given sport were being translated as one document, but the model failed to handle such long documents, and ⋆ for translations where we had to run sentence alignment to match translations to the original segments. these tagged systems may score lower than expected, when compared to the results in the general translation task @xcite , in most cases likely due to inability to handle long documents. two of the three systems that we realigned using sentalign, online-b and llama-4-maverick, seem to score similarly to the systems in the general translation task, so splitting up and realigning may have had minimal effects in these cases. ## limitations there are various limitations to the current work, some of which are discussed below. in various segments, the english terms are a part of a larger compound. we have tried to avoid including such compounds as listed terms in some places but where we think it is appropriate, we try to make the translated terms capture this fact. an example is the term point guard, which is translated as 'leikstjórnandi, ás'. however, in one segment, point guard is part of the compound star point guard. 5 in addition to the translations 'leikstjórnandi, ás', we include 'stjörnuleikstjórnandi, stjörnuás'. when translating to icelandic, english compounds can sometimes be reworded as a phrase rather than a single compound. an example is liverpool supporter which appears in one of the segments in our football set. supporter is a term in the set and one of the translations we give for it is 'stuðningsmaður'. by rephrasing we could translate liverpool supporter as 'stuðningsmaður liverpool', meaning 'a supporter of liverpool', but if we would insist on translating it as a compound, we could translate it as 'liverpool-stuðningsmaður' -and we give that as a possibility in our test suite in that particular segment. in some cases, however, we may have failed to notice that a term is a part of a compound but when a translation system translates the whole term as "one" word (without spaces or without rephrasing) the automatic evaluation would mark it as incorrect. a dataset with more synthetic data could more easily avoid such compounds. the manual evaluation makes it clear that we miss out various translation possibilities we did not think of, showing that the automatic evaluation is limited. furthermore, only one annotator evaluated each translation. if multiple annotators would have evaluated the same translations they would possibly have had some disagreements. that would also have allowed us to calculate inter-annotator agreement, which could give us an indication of whether a manual evaluation such as the one we carried out is straightforward or whether it demands multiple annotators for each segment. we try to reflect actual use in our translations. that is not an easy task, especially when term use in written texts, such as in fairly formal news coverage, differs from spoken language use. an example might be layup in basketball which we only translate as 'sniðskot'. however, searching for a string starting with "layup" in the icelandic gigaword corpus gives 219 results. 'sniðskot' has a formal feel to it but it captures the english term. sometimes, however, we were not sure whether the icelandic translation captures it completely or is well-known enough. an example is the basketball term screen, which occurs as a noun in the test suite and is translated as 'hindrun' in the icelandic version of the fiba basketball rule book; foot_3 we made the decision to give an icelandic spelling version of screen, i.e., 'skrín', in addition to 'hindrun' which goes back centuries in the language and can be translated as 'hindrance, obstacle', even though we translated layup as 'sniðskot' only (a relatively new compound in the language whose head is skot, meaning 'shot'). further work that takes a closer look at the actual usage would be interesting and working with professionals in each sport on the translations of the terms would be ideal. which words should be given as terms in a work like this can be debated. we have the verb and the noun win in various places listed as terms but as winning is not specifically found in one sport but not the other we might want to exclude some such words when we are exploring a translation system's ability to translate the vocabulary in a certain branch of sport. the same goes for, e.g., parts of the body: a word like shin is probably used more in a sport like football than golf but, nevertheless, one could certainly disagree with our decision of leaving that in as a football term in our dataset. as natural data are the bulk of segments in the test suite, it is not always clear from the segment's context alone what the sport in question is. in a few places, we added information in the segments. instead of using the original unchanged in the segment a screen is the legal action of a player who, without causing undue contact, delays or prevents an opponent from reaching a desired position we added "in basketball" to make it clear what the sport is: in basketball, a screen is the legal action of a player who, without causing undue contact, delays or prevents an opponent from reaching a desired position. however, we did not generally focus on this when finalizing the dataset and what impact this has on the translation scores merits further study. moreover, a synthetic dataset could control for this. ## conclusions and future work overall, the results indicate that more attention needs to be paid to the language of sports in llms and mt and the generally low scores confirmed our suspicion that mt systems have a hard time at translating sports texts adequately. even the highest scoring systems do not do a very good job at translating the language of sports, and in all cases consistency is lacking. categorizing the errors could be useful for analyzing comparative differences between system types. such categorization could also be useful for building mt systems better suited for translating sports language. we intend to use our test suite to automatically evaluate new systems and keep track of their competence in translating sports terminology. while the sports we selected are some of the most popular ones that have a specialized vocabulary in icelandic, expanding the test suite by adding more sports could be useful. covering the terminology of each sport more thoroughly could help us get more accurate results and having at least two occurrences of each term could make our consistency evaluation more precise. it would also be interesting to translate the keyword list into more languages than icelandic to investigate whether this problem is specific to icelandic.