GPT s Are Multilingual Annotators for Sequence Generation Tasks (2024)

Abstract: Data annotation is an essential step for constructing new datasets. However, the conventional approach of data annotation through crowdsourcing is both time-consuming and expensive. In addition, the complexity of this process increases when dealing with low-resource languages owing to the difference in the language pool of crowdworkers. To address these issues, this study proposes an autonomous annotation method by utilizing large language models, which have been recently demonstrated to exhibit remarkable performance. Through our experiments, we demonstrate that the proposed method is not just cost-efficient but also applicable for low-resource language annotation. Additionally, we constructed an image captioning dataset using our approach and are committed to open this dataset for future study. We have opened our source code for further study and reproducibility.

## introduction with the evolution of deep learning methods, various tasks in the nlp domain have demonstrated remarkable performance. however, training deep learning models requires a substantial amount of labeled data. data annotation, a process of gathering unlabeled data and labeling them, plays a crucial role in fulfilling this data demand. however, as the conventional procedure of data annotation is mainly conducted manually using human annotators, it cannot meet the growing demand for labeled data with an increase in the size of deep learning models @xcite . moreover, it is significantly challenging to recruit annotators for low-resource languages @xcite . to address the lack of labeled data and improve the performance of the model, the concept of pretrained language model (plm) was introduced. these plms have been trained on a large amount of text corpus to acquire a general knowledge of languages @xcite . by fine-tuning these models to specific downstream task, it was able to achieve performance improvement without the need for additional labeled data. with the evolution of plms via the enlargement of their sizes owing to increased training data, the development of a large language model (llm) with massive parameter size enabled few-shot learning from the context of the given prompt @xcite . accordingly, the diverse capabilities of llms have been investigated @xcite . however, despite their impressive abilities and adaptability, these llms cannot be actively exploited for downstream tasks because of the cost constraints and demand for hardware resources caused by their extensive model size. additionally, fine-tuning these models for specific purposes remains challenging due to their massive parameter size. consequently, training models for downstream tasks through labeled data is still the dominant approach for practical applications @xcite . data annotation refers to the creation of labeled data by assigning gold labels to unlabeled data. traditionally, data annotation was mainly conducted by human labelers using crowdsourcing platforms, such as amazon mechanical turk (mturk), and these platforms have aided the creation of modern, large-scale datasets. recently, to address these limitations of crowdsourcing-based data annotation and achieve a cost-efficient means to collect labeled data, several studies have proposed the utilization of llms as alternative annotators in place of human labelers @xcite @xcite @xcite @xcite . these studies have shown the possibility of cost-efficient and automatic data annotation through llms, such as gpt-3. however, as these existing studies mainly focused on simple tasks, such as text classification, additional investigation is required to apply these approaches to numerous subtasks of natural language processing. moreover, the potential of automatic data annotation via llms has not been explored for languages other than english. as previously highlighted, projects in low-resource languages may suffer from the high cost of data annotation, necessitating the need for automatic annotators for languages beyond english. in this study, we proposed a strategy that leverages llms as an assistant annotator to aid human annotators in image captioning task and text style transfer task. as depicted in figure 1 , the conventional process of establishing datasets for image captioning task required a considerable number of human annotators to generate five gold annotations for each image, resulting in a high cost for dataset construction in languages beyond english. moreover, the quality of the annotated data varies depending on the proficiency of the human annotators @xcite . similarly, the annotation process for text style transfer required significant human effort, including quality control @xcite . this study demonstrated the ability of llms to serve as assistant annotators for human annotators at a reasonable cost by generating multiple silver sentences for each gold annotation written by one single human annotator. specifically, we proposed a cost-efficient process to construct multilingual language datasets by exploiting the gpt annotator. particularly, we utilized gpt-4, which exhibits enhanced multilingual capabilities (openai, 2023), to autonomously produce diverse sentences in another language from a single english sentence, even if the human annotator is not familiar with the target language. moreover, the cost of the gpt annotator is constant as the cost is determined by the length of the processed token, regardless of the language. this highlights the efficiency of the proposed gpt annotator as an annotation method for low-resource language, which is more expensive and time-consuming compared to english. employing this method, we developed an image captioning dataset in latvian, estonian, and finnish -which are well-known low-resource languages -by employing the gpt annotator. in this scenario, a single human annotator, who lacks knowledge of the target language, provides one english gold caption for each image. through the experiment, we demonstrated that the proposed method achieves better performance compared to machine translation method. we open these datasets to support future studies. additionally, we release software to easily perform data annotation process described in this paper. our contributions are summarized as follows: ## related work gpt-3 has demonstrated that llms can conduct incontext learning from few-shot prompts. accordingly, various llms with different characteristics have been proposed @xcite . for example, based on the findings that llms can be further enhanced via human instruction and feedback @xcite , chatgpt foot_1 and its backbone gpt-3.5 with various abilities have emerged @xcite @xcite . in addition, the cutting-edge gpt-4 (openai, 2023) is a progressed version of gpt-3.5, with a longer input sequence, improved multilingual ability, and image inception ability. with the advancement of llms, studies have been conducted to augment given human-annotated data @xcite , or to annotate unlabeled data and train models for downstream tasks. one of the early studies in this field @xcite proposed an automatic annotation method that demonstrated the ability of gpt-3 to annotate a greater amount of data compared to human annotators at a lower labeling cost, resulting in higher performance at the same cost, and this strategy was observed to outperform gpt-3 itself. in addition, the study investigated the possibility of a collaboration between human and gpt annotators by leveraging the confidence of the automatic annotation of gpt to perform active labeling by human annotators. following this approach, subsequent studies expanded the annotation capabilities of gpt-3 to not just label unlabeled data but also create labeled data leveraging external knowledge, or even from scratch @xcite . meanwhile, a methodology was proposed to transfer the abilities of llms into a smaller model by generating a rationale for the labeled data, enhancing the performance of the small model @xcite . with the emergence of chatgpt, an improved version of gpt-3 that enables enhanced flexibility across diverse tasks, researchers have proposed its application for data annotation. chatgpt has been reported to outperform crowdworkers in text classification tasks in certain cases with the same instructions @xcite . additionally, studies observed that chatgpt even surpassed expert labelers in the annotations of political texts @xcite . these results have led researchers to examine the annotation abilities of chatgpt across various domains @xcite . recent studies have expanded the application of llms as annotators, from language understanding tasks, such as text classification or inference, to text generation tasks. for example, a previous study reported improved performance in query-focused summarization by reducing the noise of @xcite . additionally, dialogue generated by chatgpt has been observed to demonstrate comparable quality to reference dialogues written by human annotators @xcite . these studies indicate the capability of llms, including chatgpt, to perform as an effective annotator for not just text understanding tasks but also text generation tasks, which are more complex and challenging to annotate. however, the application of these abilities of llms to various natural language processing tasks is still limited and underexplored. in this study, we proposed an llm-based annotation method for image captioning task and text style transfer task, which has not been investigated in previous studies. furthermore, we validated the feasibility of llms as an autonomous multilingual annotator, which has not been explored in previous works. ## conclusion in this study, we have demonstrated the possibility of exploiting llm as a multilingual assistant annotator by generating multiple silver data from a single gold data in different languages. the experimental results showcased that the proposed method is cost-efficient compared to entirely human annotation, and can be effectively employed to construct datasets in various languages and tasks. the approach described in this work can be widely adapted to various languages, as it utilizes the multilingual fluency and flexibility of llms. we constructed an image captioning in latvian as a practical application of our gpt annotator. furthermore, the cost-efficiency of the gpt annotator suggested in this paper will be improved in the future, as the price of llms is expected to decline as recent cost reductions of gpt-3.5 and gpt-4 have shown. future studies will focus on improving the proposed method by utilizing the image inception ability and expanding this method to other tasks.