Conditioning LLM s with Emotion in Neural Machine Translation (2024)

Abstract: Large Language Models (LLMs) have shown remarkable performance in Natural Language Processing tasks, including Machine Translation (MT). In this work, we propose a novel MT pipeline that integrates emotion information extracted from a Speech Emotion Recognition (SER) model into LLMs to enhance translation quality. We first fine-tune five existing LLMs on the Libri-trans dataset and select the most performant model. Subsequently, we augment LLM prompts with different dimensional emotions and train the selected LLM under these different configurations. Our experiments reveal that integrating emotion information, especially arousal, into LLM prompts leads to notable improvements in translation quality.

## introduction large language models (llms) are transformerbased @xcite deep learning models designed to understand and generate natural language text by predicting the probability of the next token in a sequence. llms excel across various natural language processing (nlp) tasks, such as information retrieval @xcite , instruction following @xcite , or engaging in chatbot discussions @xcite . among nlp tasks, llms have shown great capacities in machine translation (mt) @xcite , the task of translating a text from one language to another. previous research has enhanced llm performance in mt through various strategies, including optimized prompting techniques @xcite , in-context learning features @xcite to improve translation quality over time @xcite , and a two-stage fine-tuning method composed of a first fine-tuning on monolingual data to learn general linguistic knowledge followed by a second fine-tuning on parallel data @xcite ) that establishes the current state-of-the-art method in mt. apart from llms, previous works in mt have demonstrated the possibility of controlling the translation by adding extra information to the model that is not explicitly specified in the source sentence to be translated, and that can influence the translation. existing works in that direction focused on the control of politeness @xcite , gender @xcite , or emotion @xcite of the translation and showed that this extra information helps improve translation quality. in this work, we propose to improve translation performances of an llm-based model by adding emotion as extra information in the prompt of the model to condition the translation. this work relies on the fact that words can be classified into emotion categories, leading to affective word lists @xcite . thus, conditioning the translation with a specific emotion would use a suitable vocabulary in the translation. in @xcite , authors showed that adding arousal information, reflecting the level of stimulation (ranging from calm to excited), extracted from the voice and added at the start of each input text sentence, helps improve translation performances. in the following, we study the behavior of several llms for the task of mt when emotion dimensions are added to input prompts. to address this problem, we first fine-tune several existing llms for the task of english-to-french text-to-text translation. then, after selecting the best model as baseline for our experiments, we compute for each input sentence its emotional dimensions with the help of a state-of-the-art speech emotion recognition (ser) model applied to audio recordings. finally, we compare translation performance with and without the addition of each emotional dimension as extra information added to each input prompt. we show that emotion improves translation (bleu and comet), especially in the case of arousal. ## related works in this work, we aim at combining an llm-based mt model with emotion information to improve translation performances. in the following, we first describe a close work that performs this combination without the use of an llm. then, we list several existing llms that can be used as a baseline for our mt task. ## experiments and results in this section, we describe our experiments for the task of english-to-french text-to-text translation. we conduct two successive experiments. firstly, we fine-tune five existing llms on the libri-trans dataset @xcite and consider the best model as a foundation for our second experiment. secondly, we fine-tune the selected llm on the same task but under different configurations. henceforth, prompts used for translation include each emotion dimension that is automatically estimated from the ser model. ## conclusion we proposed a new mt pipeline that combines an llm-based model and emotion information extracted from a ser model to improve translation performances. we obtain the best performances when the arousal value is added to the llm prompt. as future work, we will apply our method to other multilingual datasets including must-c @xcite . unlike the libri-trans dataset, which consists of literary text read by speakers, must-c encompasses various speech types, such as ted talks, which can offer more emotional variability and therefore further enhance translation performance. we also plan to extend our method to the speech-to-text task, also known as speech translation.