Generative Models For I ndic Languages: Evaluating Content Generation Capabilities (2023)

Abstract: Large language models (LLMs) and generative AI have emerged as the most important areas in the field of natural language processing (NLP). LLMs are considered to be a key component in several NLP tasks, such as summarization, question-answering, sentiment classification, and translation. Newer LLMs, such as ChatGPT, BLOOMZ, and several such variants, are known to train on multilingual training data and hence are expected to process and generate text in multiple languages. Considering the widespread use of LLMs, evaluating their efficacy in multilingual settings is imperative. In this work, we evaluate the newest generative models (ChatGPT, mT0, and BLOOMZ) in the context of Indic languages. Specifically, we consider natural language generation (NLG) applications such as summarization and question-answering in monolingual and cross-lingual settings. We observe that current generative models have limited capability for generating text in Indic languages in a zero-shot setting. In contrast, generative models perform consistently better on manual quality-based evaluation in both Indic languages and English language generation. Considering limited generation performance, we argue that these LLMs are not intended to use in zero-shot fashion in downstream applications.

## introduction since the release of instruction-based chatgpt, large language models (llm) have taken the language generation research landscape by storm. recent transformations in natural language processing (nlp) are largely enabled by pretrained llms such as t5 @xcite , gpt3 @xcite , and llama @xcite to name a few foot_0 . these models demonstrate impressive results in various nlp tasks, including language generation (nlg). accordingly, the use of such off-theshelf llms in solving downstream applications, such as conversational agents and creative copywriting, is rising. secondly, although the performance has reached a near-human level, most of these works focus on european languages. specifically, the latest generative models, such as chat-gpt and bard, generate near-perfect content in english and other high-resource languages. however, english is not the native language for most of the world's population. one prime example of this is india, where people interact in one of their native languages daily. considering india is the most populated country in the world foot_1 , it is imperative to evaluate the latest progress in nlp @xcite and the potential of llms to be used as-is in the downstream tasks with a focus on indic languages. so far, the llms have shown considerable prowess in tackling monolingual applications. but with increasing globalization and demand for information, research in cross-lingual approaches is gaining attention. this upcoming field consists of methods to enable information access across multiple languages. with this work, we provide an initial performance evaluation in monolingual and cross-lingual settings for indic languages. there has been a spurt of research in the direction of evaluating generative models. recent works include llm evaluation in multilingual learning @xcite , cross-lingual summarization @xcite , and multi-task, multimodal, and multilingual setting @xcite . evaluating llms as an alternative to human annotators and evaluators is also explored in @xcite @xcite . as a part of our analysis, we report preliminary observations on evaluating and anno-tating powers of generative models. this work focuses on nlg tasks such as summarization and question-answering. we use llms such as chatgpt variant (gpt-3.5), mt0, and bloomz to evaluate zero-shot (monolingual and cross-lingual) settings. we compare the zero-shot performance of these llms with state-of-the-art (sota) baselines for the above tasks. we manually evaluate the results on quality metrics such as relevance, correctness, and fluency. we present our observations on the various generative models and generation tasks, such as summarization and question-answering in monolingual and crosslingual settings. the main findings of this work are as follows: ## related work with rapid advancements in generative models, there has been a lot of interest in understanding and evaluating the performance of these models. since many of these models have not completely disclosed their technical and data specifications (e.g., bard and chatgpt), experimenting in different settings is one way to test their behavior. recently, targeted efforts have been observed to evaluate the performance of these llms in the context of multiple languages, modalities, and tasks. lai et al. (2023) perform a thorough evaluation of chatgpt for its performance in multiple languages across multiple tasks. similarly, bang et al. (2023) extensively investigate chatgpt in multilingual, multimodal, and multitask setting with a focus on reasoning and hallucination. liu et al. (2023) documents experiments evaluating chat-gpt's text-to-sql performance to explore its capability of generating structured sql text for given natural language text. wang et al. (2023a) document the performance of chatgpt-like llms for cross-lingual summarization. they consider english and chinese languages as a part of their study. in contrast, we focus solely on nlg tasks for indic languages. we consider english as a part of the cross-lingual generation setup. using generative models for annotation or evaluation is an interesting application, and many works have been reported to explore the same. @xcite explore the possibility of using chatgpt to evaluate the quality of natural language. @xcite extensively investigate chatgpt for its closeness to human experts. on similar lines, @xcite reports that chatgpt outperforms experts and crowd-workers in annotating for certain tasks. these works consider high-resource languages such as english and chinese. several other works, such as @xcite @xcite , explore using generative models as an alternative to human annotators and evaluators. our work focuses on low-resource indic languages to evaluate generative models for their ability to annotate and evaluate linguistic content. ## methodology this work aims to evaluate the performance of generative models for nlg tasks in indic languages in mono-lingual and cross-lingual settings. by definition, a monolingual setup considers a single language for input and output, whereas, in a cross-lingual setting, input and output content are in different languages. for example, generating an english summary from an english article is a monolingual task, while generating a tamil summary from an english article or vice versa is a cross-lingual task. considering continuing developments in generative models, we restrict this work to popular llms and selective tasks. but we are cognizant of the fact that continuous effort is required for exhaustive experimentation. in this work, we consider two broad nlg areas, viz. summarization (sum) and question-answering (qa). we use the indicnlg benchmark dataset @xcite covering 11 indic languages. these languages belong to indo-aryan and dravidian language families, the main difference between them being the agglutinative nature of dravidian languages. we also manually evaluate the generated content for quality metrics such as relevance, fluency, and correctness. ## experimental setup datasets as mentioned earlier, we primarily use task-specific datasets from indicnlg benchmark @xcite . to evaluate summarization performance, we choose headline generation benchmark dataset from indicnlg @xcite . this dataset consists of news articles and corresponding headlines in indic languages, with a total of 1,316,268 samples distributed across 11 indic languages. we randomly select 50 samples from every 11 languages for the monolingual summarization task. to evaluate cross-lingual capabilities, we generate the output summary in english and compare it with the english translation of the corresponding reference summary. we consider the question generation benchmark dataset from indicnlg to evaluate qa capabilities. this dataset is repurposed from squad question-answering dataset @xcite . the question, corresponding answer, and the sentence containing the answer is extracted and translated into indic languages. the dataset consists of around 98k samples each for 11 languages. for monolingual question generation and answer generation, we randomly select 50 samples from each language-specific data samples. for crosslingual experiments, we generate the questions in english and compare them with english translations of reference questions. following a similar translation methodology for cross-lingual experiments, we use llms to generate answers in english for comparison with english translations of answers in the ground truth. baselines we compare the performance of zeroshot llms with fine-tuned indicbart and mt5 models. indicbart @xcite is a pretrained model that focuses on all 11 indic languages considered in this work. similarly, mt5 @xcite is a pre-trained multilingual model covering 101 languages, including indic languages, in focus for this work. we consider results reported in @xcite for comparative analysis. metric since baseline results are reported in rouge metric, we rouge-1/2/l @xcite for our evaluation. the rouge score considers the lexical overlap between generated content and given reference text based on unigram (r-1), bigram (r-2), and the longest common subsequence (r-l). for rouge score computation, we use multi-lingual rouge toolkit foot_2 . ## results & analysis in this section, we present results and analysis for summarization, question generation, and answer generation tasks. ## concluding remarks with recent remarkable progress in generative models, it is essential to see no one is left behind. advancements in low-resource languages, such as indic languages, are lagging due to the shortage of quality data sources and technological thrust. understanding and evaluating current progress for such under-represented languages is extremely important to identify gaps for future research. with this work, we hope to assess the generative capabilities of recent generative models in the context of indic languages. we note that the generative models have limited capability in indic languages in their zero-shot setting. in contrast, these models are known to perform relatively well in generating relevant english qa content highlighting their superior understanding and reasoning abilities. off-the-shelf use of these llms in a zero-shot manner is observed to be suboptimal, underscoring the need for fine-tuning and task-relevant data sources. in comparison with a human evaluation of quality metrics, these models perform far better than actual reference content. it is observed that generative models may be useful as an alternative to manual annotation and evaluation efforts. we plan to continue this evaluation work by including gpt-4 and bard. we also hope to compile more human evaluations to better understand the efficacy of generative models as an annotator or an evaluator.