Analysing the Correlation between Lexical Ambiguity and Translation Quality in a Multimodal Setting using W ord N et (2022)

Abstract: Multimodal Neural Machine Translation is focusing on using visual information to translate sentences in the source language into the target language. The main idea is to utilise information from visual modalities to promote the output quality of the text-based translation model. Although the recent multimodal strategies extract the most relevant visual information in images, the effectiveness of using visual information on translation quality changes based on the text dataset. Due to this, this work studies the impact of leveraging visual information in multimodal translation models of ambiguous sentences. Our experiments analyse the Multi30k evaluation dataset and calculate ambiguity scores of sentences based on the WordNet hierarchical structure. To calculate the ambiguity of a sentence, we extract the ambiguity scores for all nouns based on the number of senses in WordNet. The main goal is to find in which sentences, visual content can improve the text-based translation model. We report the correlation between the ambiguity scores and translation quality extracted for all sentences in the English-German dataset.

## introduction in recent years, neural machine translation (nmt) model is widely used in translation tasks and represents remarkable performance in terms of fluency and precision compared with the previous generations of machine translation. recurrent neural network (rnn)-based nmt with attention mechanism has found broad application in different fields of nlp tasks such as machine translation. the transformer model as a self-attention based model has been introduced by google in 2017 as a new architecture for nmt @xcite . the self-attention mechanism uses cross-lingual attention that allows the input words to interact with each other (self) and find out which one should pay more attention to (attention). in addition to the mechanism of cross-lingual attention, the transformer model uses a stacked self-attention layer that follows with a point-wise feed-forward component. recently many studies in machine translation have been increasingly focusing on using visual content well as textual to improve the translation quality. therefore, multimodal neural machine translation (mnmt) as a subarea of nmt has been introduced to use visual information extracted from other modalities such as speech, image or video to translate a sentence in a source language into the target language. mnmt is an area of research that plays an important role in machine translation tasks since multimodal resources have been increasingly used in deep learning techniques. mnmt tries to extend the ability of the nmt models by taking visual context such as images as an additional input to better translate the source text. the main idea behind this is that the textual context does not provide sufficient information for the text-based nmt model in some situations to translate ambiguous sentences (ambiguous terms or grammatical gender). due to this, visual information can enrich text-best nmt systems by adding extra information to disambiguate the input words and provide correct translations on the target side. one of the main ideas of using multimodality in machine translation is that visual information can help the textual context to find the correct sense of ambiguous words in the translation process of the source sentence. for example, the word "track" in the english sentence "a man is performing a trick on a track" is an ambiguous word and could have at least two different translations in german -(1) "ein mann führt einen trick auf einer strecke aus", and (2) "ein mann führt einen trick auf einem bahngleis aus". given the word "track", the context does not provide enough information to disambiguate and translate it correctly. therefore, multimodal resources such as images can guide the translation system to select the correct sense based on the visual information. word sense disambiguation (wsd) is widely studied in different natural language processing tasks. wsd analyses given the context of an ambiguous word to assign the correct sense based on a pre-defined sense net for words. visual sense disambiguation (vsd) as a modified version of wsd use visual context instead of textual to disambiguate words. although disambiguation of word sense can be done directly by machine translation models, research on multimodal machine translation more focuses on analysing of contributions of each modality to disambiguate words in the translation process. in this work, we focus on identifying ambiguous sentences and leverage therefore the wordnet hierarchical structure to calculate an ambiguity score for each sentence. this is then used to study a correlation between ambiguity and translation evaluation scores. analysing the lexical ambiguity and translation quality allowed us to identify sentences that are more challenging in the translation process and most likely visual content can help the textbased nmt to translate sentences more accurate. ## related work multimodal machine translation is a new trend in machine translation tasks that aims to create multimodal frameworks to use information from visual modality as well as text context @xcite . different practices were used for the visual part of the mmt framework. the common approach is to extract visual information by using convolutional neural networks (cnn) and then integrate this information with textual features @xcite . many mmt models were developed based on the transformer approach. the transformer approach extracts the relationships between words in the source and target sentences by using a multihead self-attention mechanism @xcite in some studies, the global image features are used in the encoder beside word sequences to use both types of features in the decoding stage @xcite or used to initialise the hidden parameters of the encoder and decoder in rnn @xcite . @xcite use elementwise multiplication to initialise hidden states of encoder/decoder in the attention-based model. @xcite links visual and corresponding text semantically by using a visual attention mechanism. despite successfully using multimodal information in mmt, recent studies show that most of the information in the image is not related to the text while the translation process and when there is limited textual information, visual content plays more important for the translation model @xcite . the studies use visual features by focusing on relative importance among different modalities. @xcite introduced a multimodal cross-lingual word sense disambiguation model based on multimodal lexical translation dataset (mltd) @xcite to generate contextually correct translations for the ambiguous words. mltd includes a list of words of the source language with multiple translations in the training set of multi30k. @xcite introduced a translate-and-refine mechanism by using images in a second stage decoder to refine the text-based nmt model in the ambiguous words listed in mlt dataset. @xcite use a latent variable model to extract the multimodal relationships between modalities. recent methods try to reduce the noise of visual information and select visual features related to the text. @xcite ) use a multimodal transformer-based self-attention to encode relevant information in images. to capture various relationships, @xcite propose a graph-based multimodal fusion encoder. ## experimental setup this section provides insights on the dataset used in this work, neural architectures and the translation evaluation metric bleu. ## methodology in this section, we explain our methodology to calculate the ambiguity scores for each sentence based on the hierarchical structure of wordnet. to find a meaningful relationship between ambiguity and translation quality, we analyse the correlation functions between different ambiguity scores and the translation evaluation metric bleu. our focus in this work is on the inherited structure of english nouns in wordnet. each noun in wordnet can be defined as a set w of pairs (w,s) where w is a word in that language and a sense s is possible set of meanings (synonyms or synsets) for the word w. ## results this section provides the results of our experiments. after calculating ambiguity and bleu scores (nmt, mnmt) for each sentence in the test set, we analysed the correlation coefficients between ambiguity and translation quality scores to find a meaningful relationship between them. to better analyse the correlation between the sentence ambiguity and translation quality, we grouped them into sets of 50 sentences (resulting in 20 groups) after ranking them by the ambiguity score. the corpus bleu scores for nmt and mnmt on the evaluation dataset in en-de are 30.66 and 35.80 respectively. as seen in figure 2 the ambiguity score calculated by the wordnet hierarchy correlates with the translation quality, i.e., if the ambiguity of a sentence is high, the translation quality in terms of bleu is low. on the other hand, if the ambiguity of a sentence is low, the translation quality in terms of the bleu metric improves. this can be seen for all methods used to calculate the ambiguity, i.e. synsets, min_length, min_length-1. in addition to that, the graphs also illustrate the better performance of the mnmt system (orange points) compared to the text-based nmt system (blue points). ## conclusion recent studies in multimodal machine translation focused on using visual information to improve the quality of translation tasks. one of the main challenges for the translation systems is to find a correct translation in terms of the context used. despite the progress of research in this area, the performance of multimodal translation systems is more related to the quality of visual content which is used along with textual dataset. in this study, we analysed different approaches to calculate the ambiguity of the sentence to find a correlation between sentence ambiguity and the translation quality in terms of the bleu metric. we tested different approaches to calculate the ambiguity and observed that multiplying the number of entries at the minimum length level of the wordnet hierarchy for each noun provided the best correlation to the evaluation metric for each sentence. within our future work, we plan to consider the frequency and further linguistic features of wordnet synsets. in addition to that, we plan to leverage the polylingual wordnet @xcite , a large multilingual wordnet in more than 20 european languages, to calculate the lexical ambiguity beyond english. furthermore, we plan the incorporation of imagenet @xcite , which has an image dataset organised according to the wordnet hierarchy.