Low-resource Machine Translation: what for? who for? An observational study on a dedicated Tetun language translation service (2025)

Abstract: Low-resource machine translation (MT) presents a diversity of community needs and application challenges that remain poorly understood. To complement surveys and focus groups, which tend to rely on small samples of respondents, we propose an observational study on actual usage patterns of a specialized MT service for the Tetun language, which is the lingua franca in Timor-Leste. Our analysis of 100,000 translation requests reveals patterns that challenge assumptions based on existing corpora. We find that users, many of them students on mobile devices, typically translate text from a high-resource language into Tetun across diverse domains including science, healthcare, and daily life. This contrasts sharply with available Tetun corpora, which are dominated by news articles covering government and social issues.Our results suggest that MT systems for institutionalized minority languages like Tetun should prioritize accuracy on domains relevant to educational contexts, in the high-resource to low-resource direction. More broadly, this study demonstrates how observational analysis can inform low-resource language technology development, by grounding research in practical community needs.

## introduction while machine translation (mt) is often considered a single area for natural language processing (nlp), it covers a large range of applications, enduser requirements, and language directions. mt can be leveraged for everyday communication, specialized sectors (e.g., health or education), learning, or translation of official communication @xcite @xcite @xcite . it can be used by students in the classroom @xcite , workers in an office environment @xcite , or by foreign travelers in public figure 1 : overview of our approach and research questions spaces @xcite . its reliability varies significantly depending on the size and domain coverage of available corpora @xcite , and depending on similarity between source and target languages @xcite . this disparity of potential applications and accuracy across domains or language pairs is particularly salient when translating to or from lowresource languages @xcite . in low-resource scenarios, the combination of high variability both in mt accuracy across domains @xcite and of what native speakers value in nlp applications @xcite , has prompted nlp researchers to better anchor their work in local community perception, including through surveys, focus groups, and product design @xcite @xcite . these approaches, however, could be complemented by observational studies that rely on inferring needs or lack thereof for mt across domains or demographics from end-user behaviors. in this paper on low-resource mt, we work with the tetun language, 1 which is the most spoken language in timor-leste @xcite . we rely on a sample of server logs from tetun.org, a volunteer-run mt service for the tetun language, available in both website and app 2 format. until google translate added tetun support in june 2024, this service was the only widely available mt tool for tetun. tetun.org has over 70, 000 monthly active users, and translates over one million documents per month. from a sample of 100, 000 end-user translation text inputs, and their associated machine-translated output, collected between march and august 2024, we aim to respond to the following research questions (figure 1 ): tetun: low-resource, but institutional. following nigatu et al. (2024), we define why tetun is low-resource in our view: (1) tetun has very limited digital resources, with less than one million monolingual sentences available in open corpora (de jesus and nunes, 2024c) (2) tetun only has around one million native speakers (general directorate of statistics timor-leste, 2015) and (3) tetun has very little nlp research coverage, apart from work by gabriel de jesus during his phd candidature at the university of porto (de jesus and nunes, 2024a; de jesus, 2024; de jesus and nunes 2024b ). however, tetun is an institutional language, with both official status in timor-leste, and active usage in education and the media @xcite . our findings, while focused on tetun, may extend to other institutionalized lowresource languages, but are less likely to apply to non-institutionalized or endangered ones. community-driven approaches to nlp for lowresource languages. recent years have witnessed a growing focus on developing nlp technologies for low-resource and endangered languages. researchers have identified that academic 2 play.google.com/store/apps/details?@xmath0.tetun.tetunorg ## methodology we work with data from tetun.org, a specialized service for automated translation between tetun dili (tdt) and any of english (eng), indonesian (ind), or portuguese (por). the service also includes bilingual word dictionaries and a spell checker for the tetun language, but we focus on the mt part. the mt service uses a transformer model, and inference is run on the server side. the system was initially developed by fine-tuning a m2m100 multilingual model @xcite on a scraped parallel tetun-english corpus. to optimize inference speed, this model was then distilled into a smaller transformer-base architecture. for portuguese and indonesian translation, the system uses english as a pivot language. the model's performance was evaluated using bleu scores on a curated test dataset assembled by tetun.org volunteers. we get access to the following data sources: ## recommendations anchoring mt evaluation in user demand. our results highlight the importance of: (1) broad domain coverage that includes education material, especially about science and health (2) accuracy when translating into, rather than from, the lowresource language, and (3) accuracy when translating very short documents (including individual words). this could start with an evaluation framework that finds a balance between these criteria. in particular, the sharp imbalance between the domain of available tetun corpora and mt inputs leads us to recommend the systematic evaluation of mt accuracy in a wide variety of domains, and advise against measuring mt accuracy on domains that are over-represented in available corpora (e.g., news or religion), as this would risk giving a poor representation of end-user experience. meeting end-users where they are. we recommend a focus on meeting the needs of students who use mt on their mobile device. given that students seem to represent a large portion of mt users, and that learning outcomes are an important social good, future study of mt perception and improvement for students in low-resource contexts would be beneficial, both to the research community and end users. similarly, given the high cost and low reliability of internet connections in timor-leste (the asia foundation, 2023), we recommend a heightened focus on mt inference on mobile devices, for example through further research into efficient mt, or through the creation of an open source framework for on-device mt that targets mobile devices. 11 ## conclusion we present an analysis of mt usage for tetun, a low-resource language, based on server logs from tetun.org, a widely-used mt service. our study reveals key insights into user demographics, device preferences, translation directions, and domain coverage. we discover that a large share of users are likely students in timor-leste using mobile devices, translating education material from highresource languages into tetun. our analysis also uncovers a significant mismatch between the domains of user-translated text and existing tetun 11 similar to the bergamot / browser mt project corpora. our recommendations include a stronger emphasis on domains that are not well covered by existing corpora (in particular health and science education material), accuracy for the high-@xmath1, performance on very short texts, and the development of mobile-optimized mt solutions. for human-computer interaction (hci), we suggest focusing on mt perception and improvement for students in low-resource contexts. this study provides the first observational analysis of real-world mt usage for a low-resource language. it contributes to a more nuanced understanding of the challenges and opportunities in this field. we hope these insights will guide future research efforts, ultimately leading to more effective and accessible mt solutions for speakers of low-resource languages.