Leveraging LLM s for MT in Crisis Scenarios: a blueprint for low-resource languages (2024)

Abstract: In an evolving landscape of crisis communication, the need for robust and adaptable Machine Translation (MT) systems is more pressing than ever, particularly for low-resource languages. This study presents a comprehensive exploration of leveraging Large Language Models (LLMs) and Multilingual LLMs (MLLMs) to enhance MT capabilities in such scenarios. By focusing on the unique challenges posed by crisis situations where speed, accuracy, and the ability to handle a wide range of languages are paramount, this research outlines a novel approach that combines the cutting-edge capabilities of LLMs with fine-tuning techniques and community-driven corpus development strategies. At the core of this study is the development and empirical evaluation of MT systems tailored for two low-resource language pairs, illustrating the process from initial model selection and fine-tuning through to deployment. Bespoke systems are developed and modelled on the recent Covid-19 pandemic. The research highlights the importance of community involvement in creating highly specialised, crisis-specific datasets and compares custom GPTs with NLLB-adapted MLLM models. It identifies fine-tuned MLLM models as offering superior performance compared with their LLM counterparts. A scalable and replicable model for rapid MT system development in crisis scenarios is outlined. Our approach enhances the field of humanitarian technology by offering a blueprint for developing multilingual communication systems during emergencies.

## introduction the excitement surrounding llms stems from their potential to revolutionise many fields, from language translation @xcite and content generation @xcite to chatbots foot_0 and virtual assistants. @xcite observes that from the very outset, machine translation (mt) capability has been overhyped at each paradigm shift over the past 75 years, but with their ability to 'understand' language and generate complex responses, llms do appear capable of enhancing human communication and productivity in ways that were unimaginable with previous approaches, especially given that llms are not restricted to text-based use-cases, and can be used in creative applications such as generating music foot_1 or art. when building llms, the focus is on designing and training the model architecture. this involves selecting the appropriate neural network architecture and hyperparameters, as well as deciding on the training data and optimisation techniques to use. tuning an mllm or llm, in contrast, involves adjusting the parameters of a pre-trained model to improve its performance on a specific task. in neural networks such as mllms, the weights and biases are parameters that the network adjusts through training to minimise a cost function. this is performed by training the model on a task-specific dataset and adjusting the model's hyperparameters to optimise its performance. tuning an mllm can be a challenging task, as the model is often very complex and the training process can take a long time, but @xcite offer an opensource solution to fine-tuning pre-built mllms, with a particular focus on low-resource language pairs, thus overcoming much of this complexity. in contrast to predictions of their imminent demise (van der meer, 2021), @xcite predicts that tools such as adaptmllm will instead allow translators to gain a competitive edge, by building and tuning their own models with their own high-quality data, "while retaining full control over the process, leading to self-empowerment and an improved sense of well-being". given their potential, this paper investigates whether tools such as adaptmllm can be used to rapidly build good-quality mllm-based mt systems for deployment in crisis scenarios, where speed of development is crucial, but not at the expense of quality altogether. these deployments are contrasted with the development of custom gpts and fine-tuned llms. for two language pairs and four language directions, each featuring a minority language, we present and evaluate a pipeline that we hope can be used as a blueprint for rapid deployment in crisis scenarios to improve multilingual communication. this remarkable effort led to the writing of a cookbook for mt in crisis scenarios @xcite , so that the lessons learned from the exercise could be put into practice when other crises arose, as they do all too commonly, regrettably. importantly, @xcite note that "if done right, mt can dramatically increase the speed by which relief can be provided". in any such scenario, translation is almost always needed, and despite its importance, it is often overlooked. in response to the need for better preparation for translation readiness in crises, sharon o'brien coordinated the interact project foot_2 featuring partners from academia, industry, as well as ngos. @xcite provide a set of recommendations within that project which apply mainly to human translation provision in crisis scenarios. ## datasets 4.1 language pairs to benchmark the translation performance of adaptmllm in fine-tuning mllms for lowresource languages, we had to choose suitable language pairs for which appropriate datasets existed. the english-to-irish (en↔ga) and english-to-marathi (en↔mr) language pairs were selected since they fulfilled the criteria of low-resource languages, and data was freely available from shared tasks featuring these language pairs in crisis scenarios. therefore, these language pairs were very suitable for evaluating our proposed pipeline for rapidly generating high-quality translations in crisis situations by fine-tuning mllms. irish is the first official language of the republic of ireland, and is also recognised as a minority language in northern ireland. irish is an official language of the european union and a recognised minority language in northern ireland with an iso code of "ga". 10 the dominant language spoken in india's maharashtra state is marathi, with an iso code of "mr". it has over 83 million speakers, and it is a member of the indo-aryan language family. despite being spoken by a significant number of people, marathi is considered to be relatively underresourced when compared to other languages used in the region. ## empirical evaluation after outlining the details of our approach, the quality of the models developed is evaluated by training models for the en↔ga and the en↔mr language pairs. ## discussion a significant finding of this research is the demonstrated capability to substantially improve translation quality for low-resource languages through fine-tuning with crisis-specific datasets. the adaptability and speed of deployment offered by llms and mllms hold the promise of making such rapid response a standard practice in future crises, ensuring that linguistic barriers do not impede vital aid and information flow. however, this potential comes with its share of challenges, particularly concerning the assembly and quality of training datasets. this study's proposed solution, leveraging community input through custom gpts to crowd-source and refine translation data, presents a scalable model for corpus development in crisis scenarios. looking ahead, this re-search lays the groundwork for expanding the application of llms and mllms beyond mt to address a wider range of nlp challenges in crisis situations. the blueprint provided for rapid mt system deployment in emergencies, emphasising community involvement and model fine-tuning, offers valuable insights for future endeavours aiming to harness ai for humanitarian purposes. ## conclusion in this paper, we outlined how the advent of llms has transformed our ability to rapidly develop mt systems for low-resource languages in crisis scenarios. a system for rapid corpus development was presented which adopts a collaborative approach, emphasising community involvement and open-source methodologies. the appropriate response to developing mt systems at different phases of a crisis were highlighted. using the recent covid pandemic as a reference crisis, mt systems were developed using custom gpts, fine-tuned models from openai and fine-tuned mllm models. we demonstrated that a custom gpt delivers a functioning mt system rapidly whereas a fine-tuned mllm delivers a higher-quality solution given a longer time horizon. by highlighting how a fine-tuned mllm can provide sota accuracy during a crisis, our work demonstrates how llms and mllms can provide more inclusive communication. language barriers in crisis communication will be diminished with the help of this approach which in turn helps minority communities in times of real need. our paper introduces a pipeline which is applicable to a broader range of nlp problems. as part of future work, the methodologies and insights derived from our research could extend beyond the scope of mt to other domains within nlp. consequently, a versatile framework for addressing a variety of language processing challenges in crisis scenarios has been put forth in this study.