Large Language Models As Annotators: A Preliminary Evaluation For Annotating Low-Resource Language Content (2023)

Abstract: The process of collecting human-generated annotations is time-consuming and resource-hungry. In the case of low-resource (LR) languages such as Indic languages, these efforts are more expensive due to the dearth of data and human experts. Considering their importance in solving downstream applications, there have been concentrated efforts exploring alternatives for human-generated annotations. To that extent, we seek to evaluate multilingual large language models (LLMs) for their potential to substitute or aid human-generated annotation efforts. We use LLMs to re-label publicly available datasets in LR languages for the tasks of natural language inference, sentiment analysis, and news classification. We compare these annotations with existing ground truth labels to analyze the efficacy of using LLMs for annotation tasks. We observe that the performance of these LLMs varies substantially across different tasks and languages. The results show that off-the-shelf use of multilingual LLMs is not appropriate and results in poor performance in two of the three tasks.

## introduction traditionally, compiling annotations using human experts has been the primary step in formulating a supervised solution 1 for various tasks such as sentiment analysis @xcite , bot detection @xcite , and inference @xcite . the process of collecting human-generated annotations is often timeintensive and resource-hungry. specifically, in the case of lr languages, these efforts are more expensive due to a lack of quality data and human experts. therefore, alternatives to human-generated labels are being actively explored (cruz and cheng, 2020; @xcite . recent llms foot_1 , such as chatgpt, demonstrate impressive performance in various nlp applications such as summarization, classification, and text generation @xcite . furthermore, interesting use cases and applications using these generative models have been explored and reported @xcite . the research community is curious to know how close llms are to human experts and annotators. accordingly, @xcite conduct extensive evaluations in a questionanswering setup. in @xcite , chatgpt is evaluated in the context of reproducing humangenerated label annotations in social computing tasks. similar studies for misinformation in @xcite and hate speech in @xcite have considered chatgpt for annotations. additionally, several works @xcite @xcite ) compare chatgpt's annotation and evaluation performance with human experts. the point to note is that most of these efforts focus on high-resource (hr) languages like english. in reality, these hr languages are not recognized as the native languages for most of the world's population. for example, people in india prefer to interact in one of the indic languages despite of being literate in english. these indic languages are generally categorized as low-resource (lr) languages because of the unavailability of quality data sources @xcite . considering india as the most populated country foot_2 in the world, it is essential to evaluate current multilingual llms in the context of lr languages like indic languages. secondly, besides chatgpt, other multilingual llms like mt0 and bloomz must also be evaluated for such use cases. to this extent, we primarily explore the possibility of using multilingual llms as a substitute for human annotators. specifically, we focus on lowresource languages such as indic languages and compare the llm-generated annotations with the ground truth human-generated labels. to the best of our knowledge, this is the first work to evaluate the efficacy of llms as annotators for lr indic languages. we examine three llms-chatgpt, mt0, and bloomz, for three tasks-document classification, sentiment analysis, and natural language inference. the main observations from our experiments are as follows: ## methodology we follow a comparative approach to study the differences between human-generated and llmgenerated annotations for indic languages. under this premise, we consider three broad categories of tasks and relevant datasets: 1) wnli -winograd inference task involving inference based on a given context, 2) sa -identifying sentiment for a given text, and 3) newscls -categorizing given news text. we consider appropriate prompting strategies to simulate the manual annotation process. in the following subsections, we describe the multilingual llms used for annotations (section 2.1), datasets used for the three tasks (section 2.2), and our approach for the annotation process (section 2.3). ## experimental setup as mentioned earlier, we primarily use three tasks and corresponding datasets to evaluate if llms can replace or to some extent, aid the manual annotation efforts. we formulate the annotation task as a zero-shot inference task. we compare llm annotations with the ground truth labels. we consider 'test' split from all the datasets to ensure no data leakage. the 'gpt-3.5-turbo' api for chat-gpt is paid and under a constrained usage policy. hence, we use a subset of samples for the chat-gpt experiments. for mt0 and bloomz, we use the entire split whenever possible. the dataset distributions are as follows: we use the entire 'test' split distributed across various indic languages for wnli and newscls tasks, totaling to 284 and 5986 data samples, respectively. for sentiment analysis, we randomly select a total of 2862 samples spread across 11 languages with approximately 250 samples each, considering the budget for the paid experiments with the 'gpt-3.5-turbo' api. we use the following abbreviations for languages: as (assamese), bn (bengali), gu (gujarati), hi (hindi), kn (kannada), ml (malayalam), mr (marathi), or (odia), pa (punjabi), ta (tamil), and te (telugu). for chatgpt, we use the official openai api (gpt-3.5-turbo) with default settings to annotate the samples. similarly, we use hugging face models and tokenizers for mt0 and bloomz llms for annotations. due to infrastructure constraints, we use the 'mt0-large' model for mt0 and the 'bloomz-1b1' model for bloomz experiments. no training is involved since we consider zeroshot inferencing with the off-the-shelf model, i.e., a zero-shot setting. for comparison, we consider state-of-the-art baselines finetuned for these specific tasks. for sentiment analysis, we use results reported in @xcite , while results from @xcite are considered as baseline for wnli and newscls tasks. we use weighted-precision, weighted-recall, and weighted-f1 metrics from sklearn library for evaluation. we also report macro-average calculated across all languages to indicate the correctness of labels for a specific task. ## results & analysis this section presents the overview of the annotation experiments for three tasks and three llms. representative detailed language-wise performance results (f1 measure) for each task are listed in sa -superior performance in zero-shot inference all three llms perform well in identifying sentiment for a given textual content. it is interesting to see that chatgpt is ranked last amongst llms in most cases. in 9 out of 11 languages, bloomz shows superior or at-par performance as compared to baseline models. it is encouraging to see good zero-shot inference with just a single instruction. we expect even better results with more informed and aligned prompting strategies. ## concluding remarks remarkable progress in llms has opened up interesting possibilities in diverse domains. accordingly, we evaluate a novel way of using llms as annotators. we explore the efficacy of these llms as a substitute or as an aid for human annotators in the context of low-resource languages, specifically indic languages. despite the presence of multilingual training data, including data from indic languages data, the llms struggle to provide correct responses in indic languages. we report that anno-tations for simpler tasks, such as sentiment analysis, can be readily recreated by the current set of llms. we observe that these llms still have a long way to go before they can be used as annotators in lr language tasks where linguistic understanding and reasoning are essential, e.g., natural language inferencing and news classification. even though recent works have documented the feasibility of enabling annotations using these models in a positive light, these works are focused on high-resource languages. with this work, we wanted to highlight that additional efforts are needed for similar undertaking in low-resource indic languages. in the future, we intend to employ advanced prompting strategies to aid annotations, such as using linguistic markers as knowledge prompts and in-context learning to guide the evaluations. we also hope to use backtranslation to aid llms' understanding. we intend to experiment with these llm annotations as weak labels to assist improvements to data collection exercises for low-resource languages. we also plan to explore the possibility of using llms as evaluators for quality metrics such as relevance, coherence, and fluency in the future. furthermore, we note that the justifications provided by chatgpt, along with answers, are helpful and can be further exploited for annotators' training. we plan to use these justifications to improve the prompt guidelines for llm annotations. limitations we evaluate the performance of llms as annotators for certain tasks. there are a few limitations to note: 1) llm performance heavily depends on the prompts. currently, we use heuristically identified prompts, but exploring better prompts may give even better annotations in the future. 2) we agree that the experiments need more rigor. due to restrictions on api usage, we use only a subset of available datasets. 3) we believe that quality data is also an area of concern. we use translated data in some cases, which may adversely affect the performance.