Machine Translation to Inform Asylum Seekers: Intermediate Findings from the M a TIAS Project (2025)

Abstract: We present key interim findings from the ongoing MaTIAS project, which focuses on developing a multilingual notification system for asylum reception centres in Belgium. This system integrates machine translation (MT) to enable staff to provide practical information to residents in their native language, thus fostering more effective communication. Our discussion focuses on three key aspects: the development of the multilingual messaging platform, the types of messages the system is designed to handle, and the evaluation of potential MT systems for integration.

## introduction the matias project aims to develop a multilingual notification tool for asylum reception centres in belgium. the prototype will consist of a web platform that allows staff to send practical messages via whatsapp in the residents' preferred language. the project started in july 2023 and will finish in december 2025. the project is carried out by two research groups from ghent university 1 in collaboration with fedasil, the federal agency responsible for the reception of asylum seekers in belgium. it has been funded by the eu asylum, migration and integration fund (amif). ## the multilingual messaging platform the web platform is based on django (a pythonbased web framework). the platform's interface will be available in dutch, french and english, which are also the three source languages for writing messages. residents will receive messages and their translations via whatsapp. the main functionalities of the platform are (1) the registration of © 2025 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. 1 multiples ( https://research.flw.ugent.be/en/ multiples ) and lt3 ( https://research.flw.ugent.be/ en/lt3 ) users (i.e. residents), (2) the writing and sending of messages, and (3) the viewing of previously sent messages. a link to a fedasil database containing information on residents will make it possible to send messages to specific groups of residents (e.g. only residents on the 2nd floor, only residents with children). for user registration, staff can enter the resident's unique fedasil identification number, the centre in which the resident lives, the preferred language for receiving messages and the resident's telephone number. to send a message, staff use an interface similar to traditional e-mail programs. fields include subject, source language (dutch, english or french), department (e.g. social services, reception), recipient type (entire centre or groups of residents), resident centre, resident groups (if applicable) and scheduled delivery. for viewing previously sent messages, centres can use settings options to determine which staff members have access to this functionality. ## inventory of messages based on observations in four asylum reception centres @xcite , an inventory of about 400 dutch messages was compiled. the inventory includes content on house rules, hygiene and safety, administration and services, opening hours and public holidays, appointments, work and classes, etc. this list of messages was narrowed down to 200 based on the criteria of variation, frequency, and length 2 . the messages were then manually translated into english and french by staff at ghent university. the english messages (6711 words) were then sent to a translation agency to obtain human translations into 14 languages 3 (albanian, ara-bic, armenian, farsi, georgian, german, pashto, portuguese, romanian, russian, somali, spanish, tigrinya, and turkish). the set of 200 translated messages was used for mt evaluation (see the section below) and will also serve as a context-specific translation memory to customize the selected mt system prior to its integration into the messaging platform. ## mt evaluation the mt evaluation was conducted in two phases: a preliminary evaluation from may to august 2024 and a subsequent evaluation in december 2024 to january 2025. these evaluations aimed to determine the usefulness of existing automatic evaluation metrics for low-resourced languages, assess the translation quality of different mt systems, and investigate the impact of source language (english versus french or dutch) on translation quality. the first evaluation was based on a fully parallel test set of 577 sentences (6226 english words) in 14 languages extracted from the fedasil website @xcite . the source languages tested were english, french and dutch, while the target languages included a diverse set of 11 languages (the languages listed in section 3, with the exception of armenian, georgian and romanian). we evaluated three commercial systems (google translate, microsoft translator and modernmt), and one open-source model (meta ai's no language left behind model foot_2 ). in the second evaluation, we selected the 100 odd-numbered english messages (3518 english words) from the message inventory described above and translated them into the 14 target languages using the three commercial systems. as modernmt is an adaptive system that can be easily customised by uploading a translation memory, we saved the remaining 100 even-numbered messages (3193 english words) in a translation memory to adapt mod-ernmt to our domain. we looked at all the automatic evaluation metrics available in mateo @xcite , but quickly ruled out the neural metrics as they either lacked support for certain target languages foot_3 or had not been sufficiently tested on them. we faced tokenization issues with word-based metrics (bleu and ter) in several languages. for instance, in tigrinya, the ge'ez punctuation mark isn't properly stripped during preprocessing. thus, the characterbased metric chrf is the only robust metric across all target languages. the results of the first test indicate that, across all language pairs, the three commercial systems consistently outperformed the open-source model. among the commercial systems, google translate ranked first, followed by microsoft translator and modernmt, although the rankings varied depending on the language pair. in the first dataset, translations from english consistently achieved higher scores compared to those from dutch or french. in the second test, modernmt with translation memory performed better than its counterpart without translation memory, except for georgian. for 9 language pairs the customised version of modernmt achieved the highest chrf scores; google translate achieved the highest scores for 3 language pairs; microsofttranslator scored best for one language. modernmt's use of translation memory for adaptation demonstrated a positive impact on translation quality. based on the evaluation results and modernmt's adaptability, this system was selected for integration into the matias project. its ability to efficiently incorporate domain-specific translations meets the project's objectives. in a follow-up study, we will manually evaluate the usability of automated translations for all target languages and correlate available automated metrics with manual scores.