Human- LLM Hybrid Text Answer Aggregation for Crowd Annotations (2024)

Abstract: The quality is a crucial issue for crowd annotations. Answer aggregation is an important type of solution. The aggregated answers estimated from multiple crowd answers to the same instance are the eventually collected annotations, rather than the individual crowd answers themselves. Recently, the capability of Large Language Models (LLMs) on data annotation tasks has attracted interest from researchers. Most of the existing studies mainly focus on the average performance of individual crowd workers; several recent works studied the scenarios of aggregation on categorical labels and LLMs used as label creators. However, the scenario of aggregation on text answers and the role of LLMs as aggregators are not yet well-studied. In this paper, we investigate the capability of LLMs as aggregators in the scenario of close-ended crowd text answer aggregation. We propose a human-LLM hybrid text answer aggregation method with a Creator-Aggregator Multi-Stage (CAMS) crowdsourcing framework. We make the experiments based on public crowdsourcing datasets. The results show the effectiveness of our approach based on the collaboration of crowd workers and LLMs.

## introduction because of the ability or diligence of the workers, the quality of crowdsourced annotations is a crucial issue. one solution of quality control is collecting redundant annotations by assigning multiple crowd workers to one instance and aggregating the multiple answers to this instance into an estimated answer which has high reliability. there are many existing works in the crowdsourcing area on answer aggregation for various types of annotations. for example, for the classification tasks, crowd workers provide categorical labels to the instances @xcite @xcite . for some ranking tasks, workers provide pairwise preference labels, recently, the capability of large language models (llms) on data annotation tasks has attracted interest from researchers. because llms are cheaper than crowd workers for annotating the instances (e.g., @xcite reported that chatgpt is about twenty times cheaper than mturk in their experiments), whether llms can outperform crowdsourcing is one of the issues con-cerned. @xcite found that the crowd workers on mturk have been recently using llms to complete the crowdsourcing tasks. some works verified this issue with the average performance of individual crowd workers and llms on some specific tasks by collecting new datasets for their target tasks @xcite @xcite @xcite . these llm studies mainly concentrate on the average performance of individual crowd workers. however, the aggregated answers are the eventually collected annotations, rather than the crowd answers themselves. therefore, the scenarios involving crowd answer aggregation need further study. several recent works @xcite ) studied the scenario of answer aggregation on the crowd categorical labels in the classification tasks and llms are used as creators of the labels. the scenario of aggregation on other types of crowd annotations such as text answers and other types of llm roles such as aggregators are not yet wellstudied. we focus on investigating the capability of llms as aggregators on crowd text answer aggregation. llms can naturally utilize multiple text answers as the input, aggregate them, and generate the estimated answer. in other words, we can utilize a llm as a llm aggregator. in crowdsourcing area, because single-stage frameworks only collect all annotations by crowd workers at once, which is limited for the requesters to flexibly set diverse additional mechanisms for improving the data quality, multi-stage frameworks are proposed, e.g., creation-review two stages @xcite for image description and logo design, find-fix-verify three stages @xcite for shortening long texts, and so on. @xcite utilized llms in some multi-stage crowdsourcing frameworks. in this paper, we propose a human-llm hybrid text answer aggregation method based on a creator-aggregator multi-stage (cams) crowdsourcing framework. in the first stage, crowd workers (crowd creators) provide the raw text answers. in the next stage, we ask crowd workers (crowd aggregators) and llms (llm aggregators) to aggregate the raw crowd answers to each instance. in the final stage, we utilize the model aggregators on the combinations of three resources of answers with the workers, including raw crowd answers, the estimated answers by crowd aggregators, and the estimated answers by llm aggregators, to estimate the true answers. we construct experiments with public crowdsourcing datasets. the contributions are as follows. ## related work there are many existing works in the crowdsourcing area on answer aggregation for various types of annotations. for the categorical labels, some approaches jointly estimate worker ability and true answers using the maximum entropy principle @xcite , and bayesian inference @xcite . more sophisticated models incorporate task difficulty @xcite and their bayesian treatments @xcite . there are also other methods based on diverse techniques such as autoencoders @xcite , graph mining @xcite , and so on @xcite @xcite ). besides answer aggregation approaches, there is another type of approach that directly trains the classification models with the noisy crowd labels @xcite . in addition, some approaches consider reducing the budget while preserving the utility of the aggregated labels @xcite . for pairwise preference comparison labels (cattelan, 2012), a typical solution is the bradley-terry model and its various extensions or generalizations to diverse settings @xcite @xcite @xcite @xcite @xcite . there are also other types of methods, e.g., matrix completion @xcite . there are also methods for pairwise similarity labels @xcite @xcite , triplet similarity labels (van der @xcite @xcite , numerical data @xcite ) and text data @xcite . we share our crowdsourcing datasets, including various data types, which can be utilized for research on topics such as answer aggregation for crowd annotations at https://github.com/garfieldpigljy/ ljycrowd . in this paper, we mainly consider the close-ended text answer annotations. in crowdsourcing area, because single-stage frameworks only collect all annotations by crowd workers at once, which is limited for the requesters to flexibly set diverse additional mechanisms for improving the data quality, multi-stage frameworks are proposed, e.g., partition-map-reduce @xcite for writing essays, price-divide-solve @xcite for writing essays, creation-review @xcite for image description and logo design, find-fix-verify @xcite for shortening long texts, pair-compare-sort @xcite for sorting, and so on. @xcite utilized llms in some multi-stage crowdsourcing frameworks, e.g., the find-fix-verify framework. furthermore, @xcite compared the similarities and differences between llm chains and multi-stage crowdsourcing frameworks. in this paper, we propose a specialized creator-aggregator multi-stage crowdsourcing framework for crowd text answer aggregation tasks. ## human-llm hybrid aggregation most of the existing work on crowdsourced annotation is based on the single-stage framework, i.e., all crowd answers are collected from crowd creators at once and the model aggregator estimates the true answers on these raw crowd answers directly. figure 1 shows an example of a single-stage framework for crowdsourced text answer annotation. single-stage frameworks are limited for the requesters to flexibly set diverse additional mechanisms for improving the data quality. multi-stage frameworks are candidate solutions to improve flexibility and quality. because the crowd answers in this study are text, it is intuitive to ask llms to read the multiple crowd text answers to a given instance, and generate the estimated text answer. llms can be used as llm aggregators. similarly, crowd workers can be asked to provide the estimated answer by reading the raw crowd answers of an instance, and be used as crowd aggregators. we regard the crowd creator (c.c.), crowd aggregator (c.a.), and llm aggregator (l.a.) as three different types of workers. to distinguish these three resources of workers and answers, we define crowd creator workers and their answers as w c.c. and a c.c. , crowd aggregator workers and their answers as w c.a. and a c.a. , and llm aggregator workers and their answers as w l.a. and a l.a. . note that crowd creators w c.c. and crowd aggregators w c.a. are allowed to have overlaps; they are not necessary to be the same or completely different. this is a tractable setting for requesters in practice, because a c.c. and a c.a. are collected in separate stages. to effectively leverage these three resources of workers and answers in one crowdsourcing framework, we propose a creator-aggregator multi-stage (cams) framework. figure 2 shows the proposed cams framework. in the first stage, crowd creators provide the raw crowd answers. in the next stage, we ask crowd aggregators and/or llm aggregators to aggregate the raw crowd answers into estimated answers in their own words. in the final stage, we utilize the model aggregators on the combinations of three resources of workers and answers to generate the estimated true answers. there are different versions of our approach. we can combine two or three resources of workers and answers to create the merged sets, which are then used as input for the model aggregator to estimate the true answers. we provide an example based on a version that utilizes all three resources of workers and answers. given the instance set q, we can construct the worker set w by merging three worker sets with @xmath0.c. +w c.a. +w l.a. , and the answer set a by merging three answer sets with @xmath1.c. +a c.a. +a l.a. . we use the symbol '+' rather than the symbol '∪' for w and a. because the answer creation and answer aggregation are different tasks, if the same crowd worker provides answers as both crowd creator and crowd aggregator, he/she may have different behaviors and reliabilities in these two roles. we thus treat such a worker as different workers when merging the worker and answer resources. we can then utilize the model aggregators on these q, w, and a to estimate the true answers, i.e., ẑ = agg(q, w, a) where agg is one of smv, sms or rasa. ## conclusion in this paper, we investigate the capability of llms as aggregators in the scenario of crowd text answer aggregation for the quality control of crowdsourced close-ended text answer annotation tasks. we propose a human-llm hybrid text answer aggregation method with a creator-aggregator multi-stage (cams) crowdsourcing framework. the experimental results show both crowd aggregators and llm aggregators can generate estimated answers with higher quality than raw crowd answers provided by crowd creators. our cams approach with model 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.22 0.24 0.26 0.28 0.30 gleu l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (a) j1 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.16 0.18 0.20 0.22 0.24 gleu l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (b) t1 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.14 0.16 0.18 0.20 0.22 0.24 gleu l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (c) t2 figure 3: gleu results by different number of l.a. (gpt-4 (o)) based on sms and rasa. 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.46 0.47 0.48 0.49 0.50 0.51 0.52 meteor l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (a) j1 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.36 0.38 0.40 0.42 0.44 0.46 meteor l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (b) t1 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.36 0.38 0.40 0.42 0.44 0.46 meteor l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (c) t2 figure 4: meteor results by different number of l.a. (gpt-4 (o)) based on sms and rasa. 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.67 0.68 0.69 0.70 0.71 0.72 0.73 0.74 0.75 embedding similarity l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (a) j1 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.80 embedding similarity l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (b) t1 1 2 3 4 5 6 7 8 9 # of llm aggregator 0.72 0.74 0.76 0.78 0.80 embedding similarity l.a. (sms) l.a. (rasa) c.c. + l.a. (sms) c.c. + l.a. (rasa) c.a. + l.a. (sms) c.a. + l.a. (rasa) c.c. + c.a. + l.a. (sms) c.c. + c.a. + l.a. (rasa) (c) t2 aggregators can further improve the answer quality based on the combinations of three resources of workers and answers. this work presents a solution based on human-ai collaboration for a specific task. humans can leverage ai to enhance the quality and efficiency of tasks like data annotation. we share the datasets and source codes at https://github.com/garfieldpigljy/ humanllmhybridaggregation .