Assessing Translation Capabilities of Large Language Models involving E nglish and I ndian Languages (2024)

Abstract: Generative Large Language Models (LLMs) have achieved remarkable advances in various NLP tasks. In this work, our aim is to explore the multilingual capabilities of large language models by using machine translation as a task involving English and 22 Indian languages. We first investigate the translation capabilities of raw large-language models, followed by exploring the in-context learning capabilities of the same raw models. We fine-tune these large language models using parameter-efficient fine-tuning methods such as LoRA and additionally with full fine-tuning. Through our study, we have identified the model that performs best among the large language models available for the translation task.Our results demonstrate significant progress, with average BLEU scores of 13.42, 15.93, 12.13, 12.30, and 12.07, as well as chrF scores of 43.98, 46.99, 42.55, 42.42, and 45.39, respectively, using two-stage fine-tuned LLaMA-13b for English to Indian languages on IN22 (conversational), IN22 (general), flores200-dev, flores200-devtest, and newstest2019 testsets. Similarly, for Indian languages to English, we achieved average BLEU scores of 14.03, 16.65, 16.17, 15.35 and 12.55 along with chrF scores of 36.71, 40.44, 40.26, 39.51, and 36.20, respectively, using fine-tuned LLaMA-13b on IN22 (conversational), IN22 (general), flores200-dev, flores200-devtest and newstest2019 testsets. Overall, our findings highlight the potential and strength of large language models for machine translation capabilities, including languages that are currently underrepresented in LLMs.

## introduction generative large language models (llms) have made significant performance improvements in various natural language processing (nlp) tasks, showcasing exceptional progress in a wide range of applications @xcite . these tasks range from open domain question answering, where llms excel in providing accurate and coherent responses, to instructionbased tasks such as code completion, where llms can generate code snippets based on given prompts @xcite . llms have also demonstrated proficiency in tasks such as writing essays, grammar checks @xcite , and text summarization, where they can produce high quality results @xcite . these advances have been observed mainly in tasks centered on english. the popular llms support several natural languages. the performance of some languages other than english is not yet on par or yet to be evaluated @xcite . a multilingual country like india, where over 364+ languages and dialects foot_0 are spoken across its vast territory, presents a multitude of challenges across various domains due to language barriers @xcite , such as day-today communication, education @xcite , business, healthcare @xcite , tourism, governance, and more. recent advancements in the field of large language models may offer solutions to these challenges tailored to indian languages. to test whether decoder-based llms can effectively overcome language barriers, it is crucial to assess the proficiency of large language models in handling indian languages. machine translation, as a critical multilingual task, could be an ideal option to explore the multilingual capabilities of existing models. hence, we can formulate the question to assess the proficiency of large language models in handling indian languages as follows: how effectively do large language models perform in multilingual tasks like machine translation, particularly when dealing with indian languages? in this work, our main contribution is to address the following points in response to the above question. to address the above questions, we assess the translation capabilities of popular large language models (opt, bloom, llama-1, mpt, falcon, llama-2, and mistral ( §b)) that involve english and 22 scheduled indian languages (assamese, bangla, bodo, dogri, konkani, gujarati, hindi, kannada, kashmiri, maithili, malayalam, marathi, meitei, nepali, odia, punjabi, sanskrit, santali, sindhi, tamil, telugu, and urdu). we initially examine the translation capabilities of these raw large language models mentioned above ( §4.1). subsequently, we explore their in-context learning abilities ( §4.1). additionally, we fine-tune the base models using parameter-efficient fine-tuning methods, specifically lora ( §5). furthermore, we investigate the potential of two-stage fine-tuning for large language models, which involves full parameter finetuning in the first stage, followed by lora-based adapter fine-tuning ( §5). the key findings of our work, summarized in figure 1 , highlight the performance of our llmbased machine translation fine-tuned models compared to various known translation engines. these engines range from commercial (google foot_1 , gpt-3.5 foot_2 ) to open source (indictrans-2 foot_3 , ltrc-iiit-h foot_4 , seamlessm4t 6 ),our trained encoder-decoder bpcc (h) model (appendix a), traditional supervised encoder-decoder translation models (google, indictrans-2, ltrc-iiit-h), and decoder-driven causal large language model-based translation systems (gpt-3.5). our findings underscore the significant potential of large language models for translation tasks involving english and indian languages. although raw llms (llama-2-7b and llama-2-13b) do not perform well in translation tasks, our two-stage mt fine-tuned models (llama-2-13b + ff + lora (multi)) yield comparative results even with minimal parallel corpora. this suggests that llms have the potential to possess multilingual capabilities for translating into underrepresented languages, which can be further enhanced by fine-tuning. this work will be a crucial and pioneering milestone in evaluating llms for language representation and assessing their translation capabilities for a diverse range of indian languages, especially those with limited available resources. ## related work recent advancements in machine translation have shown that neural machine translation (nmt) has made significant strides in terms of output fluency and translation quality, especially when ample parallel data are available @xcite . however, the scarcity or absence of parallel data poses a challenge for most language pairs. in the case of indian languages, recent developments have tried to address this issue by introducing a new state-ofthe-art approach: multilingual machine translation involving indian languages and english @xcite @xcite . this approach uses a single script for machine translation, taking advantage of the lexical and syntactic similarities that arise from genetic and contact relatedness among indian languages @xcite @xcite . in the field of llm driven machine translation, in-context learning has gained significant attention @xcite . the use of large language models (llms) for multilingual machine translation has been a topic of interest @xcite . recent studies have evaluated the translation capabilities of llms for different language directions, with a focus on models like @xcite . in particular, @xcite proposed a twostage fine-tuning approach for machine translation using llm, involving fine-tuning in monolingual data followed by fine-tuning on a small set of highquality parallel data. our work represents the first study to specifically explore machine translation involving indian languages using large language models. the details on large language models are presented in the appendix b. ## indian languages representation in llms pre-trained (or base/raw) large language models are trained on a huge amount of language data, and some of these models are trained on multiple languages @xcite . however, their training mainly focuses on the english text @xcite . the emphasis on english is due to its substantial presence on the internet and its widespread usage in business contexts. for the purpose of this work, our objective is to assess the effectiveness of these models in machine translation tasks that involve both english and indian languages. consequently, it becomes crucial to investigate the representation of indian languages within these large language models. an approach to investigating the representation of indian languages within a large language model can involve analyzing the frequency of languagespecific words and sentences used during the training of these models. unfortunately, it is not possible to perform this analysis as the training data used for these models are not publicly accessible. llama-2, in particular, has mentioned that its pretraining corpus consists mainly of english and may not be optimal for other languages @xcite . however, it is worth mentioning that approximately language family indo-aryan dravidian sino-tibetan austroasiatic language asm ban kas snd urd doi hin gom mai mar nep san guj odi pan kan mal tam tel mni brx sat language script bangla perso-arabic devanagari gujarati odia gurmukhi kannada malayalam tamil telugu meitei devanagari ol chik no of letters in unicode 96 256 128 91 91 80 91 118 72 100 56 96 48 models (vocab) bloom (250680) (48,48) (49,207) (67,61) (57,34) (56,35) (55,25) (62,29) (66,52) (46,26) (61,39) (00,56) (67,29) (00,48) falcon (65024) (00,96) (12,244) (2,126) (00,91) (00,91) (00,72) (0,100) (00,56) (02,70) (04,96) (00,56) (02,94) (00,48) llama-1,2 (32024) (24,72) (45,211) (38,90) (01,90) (00,91) (04,76) (02,89) (33,155) (19,53) (01,99) (00,56) (38,58) (00,48) mistral (32052) (34,62) (47,209) (43,85) (05,86) (00,91) (02,78) (18,73) (04,116) (22,50) (11,89) (00,56) (43,53) (00,48) mpt (50277) (05,91) (35,221) (22,106) (02,89) (00,91) (00,80) (00,91) (01,117) (05,67) (03,97) (00,56) (22,74) (00,48) opt (50265) (00,96) (13,243) (1,127) (00,91) (00,91) (00,80) (00,91) (0,118) (00,72) (0,100) (00,56) (01,95) (00,48) for this work, we include a total of 22 scheduled indian languages for translation, which can be categorized into four main language families: indo-aryan, dravidian, sino-tibetan, and austroasiatic. these 22 indian languages are written using 13 major scripts. it is interesting to note that most of these scripts can be traced back to the brahmi script 7 , which served as the basis for the development of several indian scripts @xcite . each of these 13 writing systems has its own unique set of letters and characters 8 , reflecting the phonetic and linguistic characteristics of the respective languages they represent. ## experiment setup: machine translation using llms to evaluate the performance of large language models (llms) in machine translation tasks involving english and 22 indian languages, we mainly conducted two experiments. the first experiment focused on assessing the performance of the pretrained (raw) llm, and example-based in-context learning for the machine translation. in the second experiment, we explore the fine-tuning of the bestperforming large language models for the translation task. both directions of translation were explored, including english to 22 indian languages and 22 indian languages to english. all experiments were carried out using translation benchmark data, as discussed in section 5. as part of our experimental setup, we use the prompt pipeline shown in figure 2 . this pipeline involved using a prompt generator to generate specific prompts for the source and target language along with the source text. subsequently, an llm call is triggered to generate a response, which was then processed by a translation parser to obtain the actual translation. to ensure high-throughput and memory-efficient inference and serving of llm, we utilized the vllm library 10 (kwon et al., 2023). we conducted all experiments using a temperature parameter of 0, which ensures that the model behaves deterministically. when setting the temperature to 0, the model is restricted to selecting the word with the highest probability, effectively limiting its choice to the most likely option (aksitov et al., 2023). all of our experiments are conducted using the vllm library on a100, 40gb gpus. ## machine translation benchmark data we evaluated the performance of multilingual translation using three different benchmark datasets, as outlined in ## performance evaluation we evaluated the performance of the translation outputs using the bleu @xcite and chrf @xcite evaluation methods on benchmark data described in section 5. however, we did not include comet @xcite as an evaluation method due to the absence of support for many low-resource indian languages at the time of evaluation. we used sacrebleu library @xcite for bleu 13 and chrf foot_9 calculation. to mitigate the impact of randomness in scores, we present our findings as the average of two runs for all our results. ## conclusion our experiments and results have provided promising insights into the use of llms for translation tasks. we have found that llms have the potential to perform translations involving english and indian languages without the need for an extensive collection of parallel data, which distinguishes them from traditional translation models. furthermore, our findings indicate that the models based on llama-2 outperform other models in the zero-shot and in-context example-based learning. in particular, the llama-2-13b-based model demonstrates superior performance compared to its counterparts. to enhance the llm's understanding of english and indian languages, we have introduced a twostage fine-tuning process. this process begins with an initial full fine-tuning, followed by lora-based fine-tuning. through this approach, we have significantly improved the llm's comprehension of content in both languages. however, our experiments suggest that further work is required on llms to surpass the performance of traditional encoder-decoder-based translation models. this work could involve the development of llms specific to indian languages, which would improve vocabulary and alphabet coverage, resulting in a better representation of indian languages. however, in the future, we plan to incorporate indian-to-indian language translation using llm by exploring vocabulary expansion approaches. furthermore, our objective is to develop a single llm model capable of translating all indian languages, as well as english, in both directions. by doing so, we aim to push the boundaries of language capabilities within llms and further advance the field.