Assessing Crowdsourced Annotations with LLM s: Linguistic Certainty as a Proxy for Trustworthiness (2025)

Abstract: Human-annotated data is fundamental for training machine learning models, yet crowdsourced annotations often contain noise and bias. In this paper, we investigate the feasibility of employing large language models (LLMs), specifically GPT-4, as evaluators of crowdsourced annotations using a zero-shot prompting strategy. We introduce a certainty-based approach that leverages linguistic cues categorized into five levels (Absolute, High, Moderate, Low, Uncertain) based on Rubin’s framework—to assess the trustworthiness of LLM-generated evaluations. Using the MAVEN dataset as a case study, we compare GPT-4 evaluations against human evaluations and observe that the alignment between LLM and human judgments is strongly correlated with response certainty. Our results indicate that LLMs can effectively serve as a preliminary filter to flag potentially erroneous annotations for further expert review.

## introduction human-annotated data remains a cornerstone for training datasets in machine learning applications. however, crowdsourced annotations are often noisy and contain biases @xcite @xcite @xcite . additionally, the context and perspective of annotators can limit the accuracy and comprehensiveness of these annotations @xcite . in digital humanities, where subtle nuances and context is critical to interpretive accuracy, such biases and errors can undermine research findings. therefore, it is essential to continuously evaluate and improve the quality of human annotations in existing datasets. when working with existing annotations, researchers often lack access to the original annotators or their decision rationale. even though dataset documentation frameworks (e.g. data statements or data cards) aim to improve transparency @xcite , in practice most public datasets only provide the final labels. validating crowdsourced annotations after the fact typically requires domain experts or trained annotators to re-annotate a sample and measure agreement @xcite . this approach is reliable but labor-intensive and costly, especially as datasets grow in size. recently, large language models (llms) have shown exceptional performance across various data annotation tasks @xcite @xcite . yet, human input remains crucial in most annotation efforts. this paper explores the potential of llms in delivering reliable evaluations and supporting continuous improvements in the quality of crowdsourced data annotations. we propose leveraging linguistic cues from llm-generated evaluations to gauge the certainty of responses, using this certainty as an indicator of the trustworthiness of the evaluations. this paper presents preliminary results from applying this method, utilizing a zero-shot prompting strategy with gpt-4 to evaluate a general domain event dataset containing event-labeled sentences. ## method we propose leveraging linguistic cues from llmgenerated evaluations to measure response certainty, using this metric as a proxy for the evaluations' trustworthiness. prior research has introduced several methods for assessing certainty in llm outputs. for example, logit-based approaches @xcite are frequently used to quantify uncertainty at the token level. meanwhile, methods based on verbalized confidence @xcite and consistency @xcite have been developed to evaluate overall response accuracy. however, llms can exhibit notable overconfidence when expressing uncertainty @xcite , and consistency-based methods tend to be computationally expensive @xcite . to evaluate the trustworthiness of llmgenerated assessments of data annotations, we propose an approach that uses linguistic cues to determine the certainty of these evaluations. this method is inspired by epistemic uncertainty theory, which notes that humans often signal their level of confidence with phrases like "i guess" or "it's likely." this method is based on the assumption that, although llms do not possess true epistemic certainty -they generate responses based on statistical likelihood -they nonetheless reflect uncertainty through similar linguistic markers. in this study, we adopt the standard rubin's framework @xcite to identify such cues in llm responses. ## results from the 113 llm-generated evaluations, each for one event label, we identified a list of 67 certainty markers to assess the certainty levels of llm responses. we found 60 (52%) of the evaluations expressed absolute and high certainty, while 45 (39%) of the evaluations expressed moderate certainty and only 8 (7%) of the evaluations expresses low certainty. overall, gpt-4 agreed with 87 labels and disagreed with 26 labels. ## discussion in this study, we investigated the feasibility of employing a large language model (llm) as an evaluator of crowd-sourced annotations using a zero-shot prompting strategy. we introduced a certainty-based approach to assess the trustworthiness of llm-generated evaluations. guided by rubin's framework @xcite , we developed a list of certainty markers, categorized their certainty levels, and analyzed the relationship between evaluation quality and these certainty measurements. to validate our approach, we compared the llm-generated evaluations with human evaluations from @xcite as the baseline. ## conclusion this study has explored the feasibility and potential of using large language models (llms), specifically gpt-4, to evaluate crowdsourced annotations. our findings indicate that llms can significantly align with human evaluators, achieving a substantial portion (74.3%) of agreement. this opens exciting avenues for utilizing llms as a complementary tool to assess the quality of crowdsourced data especially in domains where manual validation is expensive or time-consuming. our approach of using linguistic cues for certainty assessment proved effective, providing a reliable metric for assessing the llm-generated evaluation quality. in conclusion, this research paves the way towards the integration of llms like gpt-4 into crowdsourced annotation validation workflows. further research on improving llm certainty calibration and targeted training for specific annotation tasks can further enhance their reliability and effectiveness in data validation tasks.