ARAIDA : Analogical Reasoning-Augmented Interactive Data Annotation (2024)

Abstract: Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge, we propose Araida, an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections. Araida involves an error-aware integration strategy that dynamically coordinates an annotation model and a k-nearest neighbors (KNN) model, giving more importance to KNN’s predictions when predictions from the annotation model are deemed inaccurate. Empirical studies demonstrate that Araida is adaptable to different annotation tasks and models. On average, it reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.

## introduction data annotation is a challenging task that involves a tradeoff between annotation quality and budget. while some platforms offer a cost-effective solution by relying on ml models to annotate data automatically 1 , the quality of such annotations is often compromised @xcite . it is particularly true in the limited data annotation scenario where the annotation budget is limited or when unlabeled data are scarce @xcite @xcite . human-machine interactive annotation methods were introduced to reduce annotation effort while maintaining annotation quality @xcite 2018, 2020; @xcite . as illustrated in fig. 1 , these methods introduce an annotation model to suggest labels (model annotations) to human annotators. the annotators accept a suggested label if it is correct. otherwise, they have to correct the label manually. compared to manual annotation, interactive annotation requires less human effort because human annotators only have to verify the model annotations instead of coming up with an answer from scratch, leading to potential speedup of the annotation process @xcite . evidently, the annotation model's accuracy is crucial because incorrect suggestions require additional human effort to rectify. existing methods update the annotation model based on previously accepted or corrected data (ground-truth annotation), aiming to reduce human corrections by improving prediction accuracy at each iteration @xcite . however, in the context of limited data annotation, the annotation model lacks sufficient labeled training data to reach a reasonable accuracy and is prone to providing incorrect suggestions @xcite . for example, in the span relation annotation example shown in fig. 2 (blue), the annotation model continues to make mistakes on similar examples ([car, tyre]) even after the human annotator corrects the label '[tree, leaf ]=>component'. as a result, this leads to more human corrections. such a problem is crucial for interactive annotation and has been identified by recent work @xcite , @xcite : synonym [tree, leaf]: component [tree, leaf]: ? correction round 𝑡𝑡 update [car, tyre]: synonym [car, tyre]: component [car, tyre]: ? round 𝑡𝑡 2 correction labors [car, tyre], component [car, tyre]: synonym [car, tyre]: ? round t 2 with araida wrong correct 0.3 0.7 error-aware integration strategy knn annotation knn inference no correction synonym 0.3* component 0.7* + error prediction on similar annotated data but it has yet to be addressed. inspired by cognitive studies on efficient learning @xcite @xcite , finding that the human brain can learn from a few examples because our brain is continuously building analogies during the learning process of concepts to facilitate comprehension, we propose analogical reasoning-augmented interactive data annotation (araida), which is designed to improve interactive annotation under the limited data annotation setting. araida provides an annotation reference to the annotation model by retrieving previously human-labeled examples in the proximity of the example in consideration using the k-nearest neighbors (knn) method. as illustrated in fig. 2 (red), the final suggestion combines the model annotation and the annotation reference provided by knn via an error-aware integration strategy. this strategy dynamically coordinates the annotation model and knn, giving more importance to knn's prediction if the predicted label from the annotation model is estimated to be inaccurate. we conduct simulated experiments for the limited data annotation task and estimate the human annotation effort based on the number of human corrections (or the number of suggestion errors) following @xcite and @xcite . we test araida on different wordlevel and sentence-level annotation tasks, combining with different annotation models (i.e., classic and llm-based models). the result shows that araida consistently improves different annotation models' accuracy across various tasks. on average, it reduces human corrections by 11.02%. further analysis attributes this improvement to the few-shot capability of the knn module and the error-aware integration strategy that effectively synergizes complementary annotations. in summary, our contributions are as follows: ## related work our research is tied to interactive data annotation, human analogical reasoning (knn), and retrievalbased language models. we provide a literature review and highlight our differences. ## araida: the proposed method we present araida, an analogical reasoningbased method for interactive data annotation that provides an annotation reference to the annotation model by retrieving previously human-labeled examples in the proximity of the example in consideration. we detail the knn inference module in section 3.1 and the error-aware integration strategy in section 3.2. finally, the optimization details are provided in section 3.3. task formalization and overview. let x denote the dataset that needs to be annotated, with c being the number of classes. given a data batch x t at time t, the annotation model f t predicts label vectors f t (x t ) ∈ r |xt|×c , and the knn module g t infers label vectors g t (x t ) ∈ r |xt|×c using previously annotated data stored in a datastore a t . then, we estimate the probability λ @xmath0, i.e., argmax(f t (x t )) = y t , where argmax(•) returns indices of the classes with the highest predicted probability and y @xmath1. finally, we use λ t to weigh the two predictions f t (x t ) and g t (x t ) through a linear weighted combination: notably, closed-source language models such as chatgpt produce discrete labels rather than predicted distributions. therefore, we cannot combine its predictions with knn's using linear combination. in such case, we use binary values (0 or 1) for λ t , which acts as a function allocation to determine whether f t or g t should apply to each example. once the human approves or corrects the final suggestions, the datastore a t is updated with the newly arrived data batch and its corresponding labels. in addition, the annotation model f t (if applicable), knn module g t , and weighting strategy λ t are updated via back-propagation. ## conclusion in interactive data annotation, an annotation model suggests labels to human annotators to verify. however, the annotation model is prone to errors when trained on limited labeled data. to tackle this challenge, we proposed araida, an approach inspired by analogical reasoning, to compensate for the performance deficiencies of the annotation model and correct its mistakes using an error-aware integration strategy. extensive experiments demonstrated that araida is flexible to combine with different annotation models across various tasks and yields consistent improvement in label suggestion accuracy, which leads to a reduction of human correction effort. in this study, our method explores a new solution to bring more flexibility by allowing the human to design any preferred annotation model according to different annotation tasks. we are devoted to optimizing human-machine utilities by emphasizing the learning of task-specified concepts efficiently from a few human demonstrations. in future work, we plan to extend araida to other annotation tasks and develop it as a general toolkit that can benefit the nlp community.