Detecting and Translating Language Ambiguity with Multilingual LLM s (2024)

Abstract: Most languages could be ambiguous, which means the same conveyed text or speech, results in different actions by different readers or listeners. In this project, we propose a method to detect the ambiguity of a sentence using translation by multilingual LLMs. In particular, we hypothesize that a good machine translator should preserve the ambiguity of sentences in all target languages. Therefore, we investigate whether ambiguity is encoded in the hidden representation of a translation model or, instead, if only a single meaning is encoded. In our experiments, we have been able to predict ambiguity of sentences with high accuracy using machine translation without direct use of semantics and only based on the reconstruction error of a function that maps the forward and backward translation hidden representations to each other. The potential applications of the proposed approach span i) detecting ambiguous sentences, ii) fine-tuning existing multilingual LLMs to preserve ambiguous information, and iii) developing AI systems that can generate ambiguity-free languages when needed.

## introduction language ambiguity is defined as the potential of different actions as a response to a single text by different people, based on their interpretations @xcite . this definition aligns with the semantic, syntactic, pragmatic tests along with identity tests defined in @xcite to identify ambiguous sentences. several research studies have been focusing on the ambiguity of language. for a comprehensive review on resolving ambiguities in nlp, refer to @xcite . @xcite have studied lexical and syntactic ambiguity in the korean language. they proposed adding new words as a solution for lexical and syntactic ambiguities. @xcite proposed a prototype for an ambiguity ambiguous disambiguation "give me the bat!" (lexical) "give me the baton!" "the professor said on monday he would give an exam" (syntactic) "the professor said that on coming monday he would give an exam" "jane saw the man with a telescope" (semantic) "jane saw the man by using a telescope" "i like you too!" (pragmatic) "i like you too like others do!" "the prof said she would give us all a's." (vagueness) "the prof said the ta would give us all a's." "proposal" to "voorstel" and "aanzoek" (translational) "research proposal" language ambiguity brings up misunderstandings and conflicts in real-world interactions such as political, commercial, and cultural interactions @xcite , @xcite ). this misunderstanding can lead to either wasting of huge amount of time in negotiation between the parties for conflict resolution or even in the worst case results in conflicting actions @xcite ). by using the powerful tools in nlu and nlp using language models, it could be possible to solve these issues. the main research questions being investigated in this project are: question 1: do state-of-the-art transformerbased mt models properly encode whether a sentence in the source language is (non-) ambiguous? question 2: are both semantic validity and ambiguity preserved by the translation of these models, when the sentence is translated into a target language, and then translated back? question 3: can we predict the ambiguity of a sentence by translating it into another language looking at the learned hidden representations? the main contribution of this work is proposing a solution that detects ambiguous sentences in different typos, without direct use of semantics. furthermore, through our experiments, we conclude that ambiguity of the sentences are preserved in the hidden representation of the multilingual llm translation model. ## related work before explaining the proposed approach, we review the related literature, consisting of ambiguity in nlp, ambiguity in machine translation, and an overview of multilingual llms. ## proposed method in this project, we aim at testing how language ambiguity is represented in multilingual llms. we propose language translation as an action performed by llm agents. accordingly, we propose a four-step approach in detecting language ambiguity, as illustrated in figure 1: 1. translation: translate the input text from the source language into the target languages using a multilingual llm. then extract the hidden representation from the llm. 2. back-translation: translate back the output texts of the first step from the target language into the source language using the same llm. then extract the hidden representation of the state of the llm as a vector. considering n different meanings for input text t a and m different interpretations of the output text t b , in the worst case we would have n×m different translation meaning pairs, which complicates the problem of ambiguity in translation. as it has been noted in section 4.3, the translation process by itself can be a source of ambiguity. the llm works as a function f (.) defined in equation ( 1 ): the hidden representation r consists of a distributed representation of multiple factors, not only including the semantics (bau (2022), zhang (2024)) and it is not easy to simply disentangle these factors and manually extract the representation of the input text t from the representation r. also as the representation r contains factors such as the information about the source and target language, the translation task, etc., we can not directly compare the two representations r a and r b to detect ambiguity in the text. therefore we propose a different approach in detecting ambiguity. in the first step, we define a function g(.) that maps the two representations to each other as illustrated in equation ( 3 to find the function g(.), we learn a simple autoencoder with a single hidden layer of size s h , input size of s a and output size of s b . note that as the translation in steps 1 and 2 are both performed using the same llm, we have s @xmath0. the auto-encoder maps the input translation representation r a to the output translation representation r b . the error of the network implementing g(.) is defined as the normalized mean squared error (nmse) of the elements of the two representations r a and r b (the actual equations can be found in the appendix a). we define the function c(.) as complexity of the function g(.) as follows: by learning function g(.), for each text t a in the input dataset, we can evaluate the translation error e(.) for each setting of the network complexity c(.) with different hidden layer sizes. figure 2 reports the error of the function against its complexity. the main idea for using an auto-encoder is based on the assumption that: (1) we expect the autoencoder will behave differently for ambiguous vs unambiguous sentences; (2) in particular, we conjecture that model size and the target language will affect differently the model when dealing with ambiguous vs unambiguous sentences. we propose using a simple neural network model to predict ambiguity using the data points in the elbow chart in figure 2 as input in a supervised manner. ## results in this section, we provide the results of our experiments. ## discussion based on the results of our classification experiments shown in learning more complex and nonlinear features actually helped the classification. moving back to our initial research questions, based on the results in the main contribution of the project is predicting ambiguity of the sentences, without direct use of semantics. as explained in section 3 this feature is achieved by classifying the ambiguity based on the shape of the mapping function. as a consequence, the algorithm does not require extensive training data to cover the whole semantic. furthermore, the approach is potentially much more generalizable to unseen sentences with unseen semantics. also, the model would be robust to changes to the input distribution as it is independent of the semantics. ## future work one future direction method is to investigate in more details the source of misclassification for all fourteen target languages other than italian and persian. other than that, detecting the source of ambiguity in sentences in terms of words could be an interesting direction. furthermore, extending the method to different source languages other than english could also be considered as future work. one of the potential applications of an ambiguity detection method could be in automatic translation of critical documents e.g. legal, political, commercial, where the user is asked to clarify the ambiguity of the source language manually, to prevent misunderstanding and potential conflicts. fine-tuning existing multilingual large language models to preserve ambiguity in sentences could be another potential application of the proposed method. finally, the trained classifier model can potentially be used as a partial loss function for designing and optimizing ambiguity-free ai-generated human languages investigated at synaptosearch foot_3 . in order to do so, for each input sentence generated by the ai, the ambiguity is measured using the model and the gradient with respect to the input is calculated and used to optimize the loss function term related to ambiguity. ambiguity can be considered of a strength of the language in cases such as providing efficient means of communication or when it is used as amphibology in literature. however, in critical political, commercial and cultural cases and social media, unintended ambiguity results in misunderstandings and conflicts. the outcome of the misunderstanding could lead to spending a lot of time in negotiation to elaborate the meaning, or in worse case conflicting actions. one major organization that can benefit from the proposed research is the united nations (un) where different countries with different languages interact with each other. considering automatic translation in such organizations where a speech/text is translated into many languages, detecting and informing the potential ambiguities to both the speaker/writer and the listener/reader, would prevent potential misunderstandings, tedious negotiations, and conflicting actions between the nations and parties in the long term @xcite ).