Entity-aware Multi-task Training Helps Rare Word Machine Translation (2024)

Abstract: Named entities (NE) are integral for preserving context and conveying accurate information in the machine translation (MT) task. Challenges often lie in handling NE diversity, ambiguity, rarity, and ensuring alignment and consistency. In this paper, we explore the effect of NE-aware model fine-tuning to improve handling of NEs in MT. We generate data for NE recognition (NER) and NE-aware MT using common NER tools from Spacy, and align entities in parallel data. Experiments with fine-tuning variations of pre-trained T5 models on NE-related generation tasks between English and German show promising results with increasing amounts of NEs in the output and BLEU score improvements compared to the non-tuned baselines.

## introduction machine translation (mt) of named entities (nes) such as person or place names remains a significant challenge even for modern modelling architectures simply because they appear less frequently in training data than other words or phrases. furthermore, new and unseen nes get created every day like organization or product names, and even common nouns in certain contexts can become nes. meanwhile, the task of ne recognition (ner) has reached a fairly acceptable level for many languages with precision values of around 80-90%. since most conventional mt models are trained to perform translation based only on the parallel training data and context provided, they still often struggle with rare nes appearing less often during training or never at all. in such cases, the models tend to hallucinate by generating output comprised of tokens or subword units which are statistically close in the embedding space to the rare ne, but this can lead to the generation of a novel word or phrase instead of the proper acceptable translation. in this work, we look into improving how the model handles nes by highlighting them in the training data and training not only to translate but also to recognize nes in plain input text. the motivation for this approach is for the model to form a more defined understanding of what certain nes look like thus enabling it to handle them better when performing the mt task. we experiment with multi-task training and fine-tuning the t5 model @xcite for translation between english and german, as well as its multilingual counterpart mt5 @xcite and the updated 1.1 version of t5. we compare the results with the non-modified versions of t5, mt5, and the instruction-tuned flan-t5 @xcite . our contributions are 1) a novel, easily reproducible and further extensible method for finetuning transformer models in a multi-task fashion on named entity recognition and machine translation tasks; 2) empirical evaluation of the method on a recent shard task benchmark data set; 3) opensourcing of data preparation and training scripts, and model checkpoints for reproducibility. ## proposed approach since the existing pre-trained t5 model versions have already been pre-trained on large multilingual corpora, the quality of the data used for fine-tuning on the resource-rich languages plays a more significant role than the quantity @xcite . we start with filtering out any critical noisy data from the wmt23 foot_1 general translation shared task training set before tagging named entities in the form of xml boundary tags. next, we prepend instructions to the source side of the training data as shown in ## data preparation we use the english-german parallel data from the wmt 2023 shared task on general text translation for experimentation. to develop our models, we use the general test set from wmt22 and for evaluation and result reporting -general test set of wmt23. we first filter the data by removing noisy parallel segments. then we populate the data with ne tags in either the source or target side, depending on the task. finally, we prepend taskspecific instructions to all source-side inputs. for the ner task training data, we use both source and target mt parallel sentences, essentially doubling the amount when compared to mt task data. ## results we evaluate mt performance by computing bleu @xcite scores using sacre-bleu @xcite the f 1 score. an overview of the main automatic evaluation results is shown in tables 3 and 4 . by looking only at the bleu scores, it does seem like de-en translation improves compared to baseline results in tables 5 and 6 while en-de seems to be degraded. however, the amounts of recognized nes in the generated translations are overall higher for the ne-aware models. performance on the ner task is relatively low, aside from the t5 large model, but that is not our main focus. ## conclusion in this paper, we introduced a simple approach for fine-tuning sequence-to-sequence models that is effective at mitigating one of the commonly known drawbacks of mt -the translation of rare words and named entities. with a small training data modification, we were able to increase the amount of generated named entities in translations, and even achieve a higher bleu score than the baselines when translating from english into german. in future work, we plan to evaluate the approach on more languages and alternative ner taggers for training data generation. we are also eager to explore the applicability of the back-translation approach for incremental ner improvements, as well as an extension of our method to summarisation and question-answering tasks.