ITALERT : Assessing the Quality of LLM s and NMT in Translating I talian Emergency Response Text (2025)

Abstract: This paper presents the outcomes of an initial investigation into the performance of Large Language Models (LLMs) and Neural Machine Translation (NMT) systems in translating high-stakes messages. The research employed a novel bilingual corpus, ITALERT (Italian Emergency Response Text) and applied a human-centric post-editing based metric (HOPE) to assess translation quality systematically. The initial dataset contains eleven texts in Italian and their corresponding English translations, both extracted from the national communication campaign website of the Italian Civil Protection Department. The texts deal with eight crisis scenarios: flooding, earthquake, forest fire, volcanic eruption, tsunami, industrial accident, nuclear risk, and dam failure. The dataset has been carefully compiled to ensure usability and clarity for evaluating machine translation (MT) systems in crisis settings. Our findings show that current LLMs and NMT models, such as ChatGPT (OpenAI’s GPT-4o model) and Google MT, face limitations in translating emergency texts, particularly in maintaining the appropriate register, resolving context ambiguities, and managing domain-specific terminology.

## introduction llms have shown remarkable advancements in generating fluent and coherent translations. they are trained on large-scale multilingual datasets and can improve translation quality, efficiency and domain adaptation. the interest in llms also stems from the fact that they can provide valid translations for high-resource languages, producing competitive results with respect to traditional mt systems @xcite . given their potential, this preliminary evaluation aims to assess the effectiveness of state-ofthe-art language models during the preparedness and response phases of an emergency @xcite for the language pair @xmath0. the rationale for selecting this language pair lies in the need to communicate effectively with the non-italian-speaking population, including immigrants, refugees, and tourists who are in italy. the study focuses on identifying which translation system performs better under emergency conditions for specific natural disaster scenarios, providing critical insights to improve multilingual crisis communication in these phases. the current italian-english bilingual corpus consists of 13,218 words in total: 6,622 in italian and 6,596 in english. it includes 440 segments across eight subdomains (flooding, earthquake, forest fire, volcanic eruption, tsunami, industrial accident, nuclear risk, and dam failure), extracted from the communication campaign website io non rischio. 1 ## background and related work previous studies of crisis translation have demonstrated that mt systems can be fast, reliable tools for emergency response @xcite . the past decade has seen a renewed focus on the possibility of using mt in preventing and mitigating disasters @xcite @xcite mt systems in the crisis management workflow. during crises, it is crucial that messages are spread quickly and effectively to the population @xcite . but what impact can these messages have if they are communicated in a language that's foreign to the recipients or only partially understood by them? based on the latest tourism report provided by istat @xcite , in 2023 italy recorded 234.2 million overnight stays by foreign tourists, with non-residents making up 52.4% of total hospitality demand. at the same time, the country experienced a significant influx of immigrants and asylum seekers, representing more than 15 nationalities (istat, 2024). in these multilingual and multicultural contexts, english is frequently used as a lingua franca, especially in interactions between migrants, institutions, and interpreters @xcite . this highlights the importance of using english as a vehicular language to foster mutual intelligibility in critical legal, medical, and social settings. for these reasons, we chose the language pair @xmath1. our goal is to answer the following research question: how accurate are current language models in translating crisis-related texts from italian to english? in line with this direction, we started researching existing corpora on the topic of crisis translation in italy, and we identified one dataset containing humanitarian response documents: humset @xcite . however, this dataset was not selected for our study due to its lack of data in italian, a critical requirement for our research objectives. while humset offers valuable multilingual resources (english, spanish, french), the absence of italian significantly limits its relevance to our analysis, which focuses on evaluating the quality of translations in crisis communication scenarios, specifically involving the @xmath2. regarding mt evaluations and interpretability, previous work by @xcite has examined both human and automatic evaluation methods. recent research has also explored explainable mt evaluation @xcite , with a focus on providing detailed, interpretable error analyses. additionally, the human-centric postediting based metric hope offers both explainable feedback and supports the creation of a post-edited gold-standard corpus. as outlined by @xcite , hope adopts a simplified and practical approach to human evaluation, specifically designed for machine translation outputs. given that our study also aims to develop such a bilingual corpus, hope is a suitable choice, as it offers both detailed error analysis and the option to produce a post-edited corpus. the original hope metric has eight predefined error categories and severity levels. the eight error types in hope are: impact (imp), required adaptation missing (ram), terminology (trm), ungrammatical (ugr), mistranslation (mis), style (stl), proofreading error (prf), and proper name (prn). the error severity levels and corresponding point values are: minor (1), medium (2), major (4), severe (8), and critical (16). for our annotation, we decided to use only 7 of the 8 error categories in hope, which will be explained in later sections (4.3). ## investigation methodology the italert methodology for our investigation is shown in figure 1 . source (it) segments: flooding earthquake volcanic eruption, etc. nmt-output gpt-output reference (en) dcc corpus error types + statistics + iaa post-editing evaluation: quantitative + qualitative hope metric mt systems genai (chatgpt) source (it) nmt (googlemt) figure 1 : the italert investigation methodology design framework. @xcite , asking the model to perform a translation task. the prompt was as follows: "this is a document in italian including crisis or emergency text for eight subdomains, specified in the heading part of each table. can you please translate the content into english and keep the original document format in the tables?" ## conclusion and future work in this study, we investigated the performance of llms and nmt systems in translating crisisrelated texts. the evaluation was conducted using 440 segments from eight subdomains, with data sourced from the national communication campaign website io non rischio. chatgpt-4o and google translate were selected as representatives of generative ai and stand-alone nmt systems respectively, and were evaluated using a humancentric evaluation framework. errors from each system were categorised using the default 7 error types (merged from 8) from the hope metric, with a revised severity mapping, adjusted to account for the sensitivity of the crisis domain. the findings reveal that both systems share common error types but differ in their rankings. chatgpt showed a high incidence of style and terminology errors, while google mt was characterised by a greater presence of mistranslation, impact, terminology, and style issues. importantly, both systems produced a non-negligible amount of severe and major errors, despite the predominance of minor and medium-level issues. the number of segments with severity ratings above 4 was slightly higher in google mt outputs than in those of chatgpt, indicating a greater incidence of critical errors. as seen in the qualitative analy-sis, several of these high-severity errors in google mt translations had the potential to significantly distort the intended meaning and undermine the actionability of the messages. interestingly, automatic evaluation metrics appear to diverge from human error analysis findings. while bleu scores show a clear advantage for google mt, indicating stronger surface-level fidelity, comet scores are only marginally higher for the gpt system, suggesting comparable semantic adequacy. this trend aligns more closely with human judgments: hope-based error annotation reveals that google mt's surface-level advantage does not correspond to improved quality in critical cases, as human annotators identified 728 errors in google mt outputs, compared to 611 in those produced by chat-gpt. this discrepancy reflects limitations of using merely quantitative metrics in capturing contextsensitive, high-impact errors, and highlights the importance of complementary human-centric evaluations, especially in high-stakes scenarios @xcite , as well as the need of developing domain specific automatic metrics in the future, e.g. for crisis translation. in addition to the error analysis of current llms and nmt systems, using the hope framework, we also produced the diamond crisis corpus (dcc), a new post-edited reference set derived from the italert dataset. since italert represents the first mt corpus on italian crisis translation, we plan to add new subdomains, such as public health emergencies from the healthcare domain @xcite . other authoritative sources from which to extract the texts include médecins sans frontières foot_3 , the international red cross foot_4 , and the united nations office for disaster risk reduction foot_5 . we also plan to calculate the levenshtein distance between the systems' outputs and the dcc corpus, to investigate differences in findings compared to those obtained using the original "gold" corpus. this analysis will help determine whether the post-edited corpus enhances the evaluation of translation performance. future work will explore in-domain training to address the challenges of context disambiguation and terminology management @xcite and raise awareness of a responsible use of translation technology in high-stakes settings. finally, while we acknowledge that resources comparable to italert are not yet widely available, this work represents an initial step toward addressing the underexplored area of multilingual crisis communication @xcite , hoping it will serve as a foundation for future research and resource development in this emerging field.