D eco MT : Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models (2023)

Abstract: This study investigates machine translation between related languages i.e., languages within the same family that share linguistic characteristics such as word order and lexical similarity. Machine translation through few-shot prompting leverages a small set of translation pair examples to generate translations for test sentences. This procedure requires the model to learn how to generate translations while simultaneously ensuring that token ordering is maintained to produce a fluent and accurate translation. We propose that for related languages, the task of machine translation can be simplified by leveraging the monotonic alignment characteristic of such languages. We introduce DecoMT, a novel approach of few-shot prompting that decomposes the translation process into a sequence of word chunk translations. Through automatic and human evaluation conducted on multiple related language pairs across various language families, we demonstrate that our proposed approach of decomposed prompting surpasses multiple established few-shot baseline approaches. For example, DecoMT outperforms the strong few-shot prompting BLOOM model with an average improvement of 8 chrF++ scores across the examined languages.

## introduction in this work, we focus on the translation between related languages, a vital aspect from both economic and social perspectives. a considerable amount of commercial activity and social interaction occur between neighboring regions speaking two related languages. in these situations, pivot translation via a third language, such as english, can prove inefficient due to two inference steps which can also cause cascading errors @xcite . instead, direct translation between related languages could significantly streamline trade and enhance social connections. related languages, often from the same family, share word order and lexical characteristics, leading to predominantly monotonic translations where word order is largely preserved. this is seen in languages like hindi, marathi, malayalam, tamil, bengali, etc. from the indian subcontinent, which follow a subject-object-verb (sov) structure. similar monotonic translation relationships are also observed among other language pairs, such as indonesian and malay or ukrainian and russian. recent work has shown the power of few-shot prompting with large language models (llms) for tasks like machine translation, summarization, and question answering @xcite . in machine translation, this approach prompts an llm with a handful of example pairs and a test example. this requires the model to generate translations while ensuring a fluent word ordering, a process that fails to account for any unique characteristics intrinsic to the languages involved. for instance, it neglects the monotonic alignment-an integral trait evident in translations between related languages. llms are often biased towards english in their training data. for example, in mt5 @xcite , hindi and malayalam tokens represent just 0.8% and 0.07% respectively. this imbalance hinders llm performance in tasks involving non-english languages and english to non-english translations @xcite . in particular, for fewshot translation tasks between related languages, these models may not have encountered sufficient data in these languages. overcoming these limitations can be achieved by incorporating inductive biases about related languages. recently, @xcite introduced an approach known as decomposed prompting. this the source text (h) is divided into several chunks (h 1 , h 2 ,..,h i ,h i+1 ,h i+2 ,..,h β ). each chunk is translated independently using few-shot prompting, yielding corresponding target chunks (m 1 , m 2 ,..,m i ,m i+1 ,m i+2 ,..,m β ). the decomt process leverages the source chunks, their respective translations, and the previously predicted contextual translation to incrementally predict the contextually appropriate translation of the subsequent chunk. while much of the existing research on prompting focuses on decoder-only llms, recent studies @xcite show the potential of encoderdecoder models like mt5 @xcite for such tasks. our decomt approach builds upon this premise, utilizing the mt5 encoder-decoder llm. the following are our contributions: we release code and model outputs on github foot_0 . ## related work few-shot prompting for mt few-shot prompting for mt leverages an autoregressive llm, which is prompted with a small number of sentence pairs alongside their translations. the llm then predicts the translation when provided with a test sentence. examples of such llms include xglm @xcite and bloom @xcite . we interchangeably refer to this approach as standard prompting. @xcite have shown the effectiveness of few-shot prompting in machine translation. yet, their method necessitates training a decoderonly llm from scratch. in comparison, we use an off-the-shelf llm, mt5, for decomt. a series of recent research delves into example selection for prompt construction @xcite @xcite . in our method, we rely on a fixed set of examples for prompting. @xcite analyzed machine translation using chatgpt and found that chatgpt's performance aligns closely with commercial translation systems when utilizing gpt-4. in the interest of reproducibility, our emphasis lies on publicly accessible llms like bloom and mt5. @xcite introduced an approach for prompting bidirectional llms, such as mt5 @xcite . their sequential autoregressive prompting (sap) method generates a token autoregressively, appends it back to the input, and predicts the subsequent token. they demonstrated that sap outperforms traditional few-shot prompting for llms. ## decomt in this section, we present the decomt approach, our technique for decomposed prompting in machine translation. our method involves a two-stage translation process for word chunks: firstly, an independent translation stage where each chunk is translated in isolation; and secondly, a contextual translation stage where translation occurs while considering the surrounding context. ## experimental setup we conduct a comparative study of our decomt approach, which is based on mt5 @xcite with 3.7b parameters, against various established approaches. these include the standard prompting technique applied to 7.1b parameters variant of bloom @xcite , and 7.5b parameters variant of xglm @xcite . we also compare our method with the standard prompting technique applied to the mt5 model. in this case, as mt5 generates only a few tokens at a time, we append the generated text back to the input to prompt further text generation. furthermore, we compare our approach with sap @xcite , a technique that also utilizes mt5 with 3.7b parameters. ## conclusion in this study, we introduced decomt, a novel approach using decomposed prompting for machine translation of related languages. decomt demonstrated superior performance over established fewshot prompting baselines in translating between low-resource related languages, as evidenced by our experiments with the flores dataset. additionally, decomt showed robust performance even in high-resource scenarios.