Multi-News+: Cost-efficient Dataset Cleansing via LLM -based Data Annotation (2024)

Abstract: The quality of the dataset is crucial for ensuring optimal performance and reliability of downstream task models. However, datasets often contain noisy data inadvertently included during the construction process. Numerous attempts have been made to correct this issue through human annotators. However, hiring and managing human annotators is expensive and time-consuming. As an alternative, recent studies are exploring the use of large language models (LLMs) for data annotation.In this study, we present a case study that extends the application of LLM-based data annotation to enhance the quality of existing datasets through a cleansing strategy. Specifically, we leverage approaches such as chain-of-thought and majority voting to imitate human annotation and classify unrelated documents from the Multi-News dataset, which is widely used for the multi-document summarization task. Through our proposed cleansing method, we introduce an enhanced Multi-News+. By employing LLMs for data cleansing, we demonstrate an efficient and effective approach to improving dataset quality without relying on expensive human annotation efforts.

## introduction the significance of dataset quality in deep learning applications cannot be overstated as mislabeled or noisy data can severely degrade performance @xcite . datasets with incorrect labels, noise, or inconsistencies undermine the consistency and stability of model training. cleansing these datasets contributes to enhancing model performance and generalization capabilities. hence, ensuring the quality of the dataset by identifying and eliminating noisy data is imperative. in the realm of natural language processing, several researchers have attempted to improve the quality of noisy datasets @xcite . for example, redo-cred @xcite addressed issues such as source 1 starting in 1996, alexa internet has been donating their crawl data to the internet archive. flowing in every day, these data are added to the wayback machine after an embargo period. source 2 ... for the first time in decades, researchers trying to develop a vaccine for malaria have discovered a new target they can use to attack this deadly and common parasite... source 3 focused crawls are collections of frequently-updated webcrawl data from narrow ( as opposed to broad or wide ) web crawls, often focused on a single domain or subdomain. summary researchers think they've found a promising new potential weapon in the fight against malaria in a fairly unlikely place: the blood of toddlers. in a paper published in science today, ... despite these efforts, relying on human annotators to enhance datasets poses challenges such as high costs and time constraints. the quality of the annotation might also be affected by potential variations, such as subjective bias and the proficiency of the annotator @xcite . furthermore, cleansing a noisy dataset typically requires a larger budget, often involving majority voting by multiple annotators or validation by experts @xcite . given the significance and necessity of enhancing the quality of existing datasets, these obstacles hinder practical efforts to cleanse datasets efficiently. therefore, it is crucial to explore cost-effective methods that can cleanse the existing dataset, minimizing human involvement. in this study, we propose leveraging large language model (llm)-based annotation for dataset cleansing. researchers have explored cost-efficient alternatives to human annotators by employing llms across various tasks @xcite @xcite @xcite . however, the real-world applicability of llm-based annotation on existing datasets is still less explored. building on these insights, we extend the application of llm-based annotations to denoise the existing dataset and improve its quality. specifically, we conduct a case study to cleanse the multi-news @xcite , a dataset for multidocument summarization tasks. this dataset consists of news articles crawled from the internet and is widely used in multi-document summarization research. however, as shown in to accomplish our purpose, we utilize llms to analyze the summary and associated documents, identifying and excluding any documents that are not relevant to the summary. specifically, we employ approaches such as chain-of-thought (cot), providing the rationale for decision-making with enhanced transparency and facilitating human investigation. we further enhance our cleansing process by incorporating self-consistency considerations, which mimic the majority voting process used by human annotators @xcite . based on our carefully designed framework, we introduce multi-news + , an enhanced version of the existing multi-news dataset, achieved through our llm-based cleansing strategy. to the best of our knowledge, this is the first attempt to exploit llms to enhance the quality of real-world datasets. our experiments demonstrate the effectiveness of multi-news + , providing a valuable resource for future research. we make multi-news + and our source code publicly available for further study. ## related work dataset quality has been an interest to researchers because of its importance in ensuring the quality of the model trained with the dataset @xcite . previous studies found that large amounts of data automatically crawled from the web may contain noisy documents, and proper filtering procedures can be an efficient solution against them @xcite @xcite @xcite . accordingly, several studies in text summarization investigated various strategies to filter out noisy data @xcite @xcite and released new datasets with better quality @xcite . however, their strategies are primarily composed of coarse rule-based methods and less interpretable model output, or costly human investigation has been applied for constructing new datasets. furthermore, such strategies have not been applied to multi-document summarization datasets. in the meantime, with the advancement of llms @xcite , researchers have explored the usage of llms for data annotation, a task that traditionally relied on human annotators. initial attempts have revealed the potential capabilities of models like gpt-3 for data annotation (wang et al. @xcite . these studies indicate that gpt-3 can annotate datasets more efficiently and costeffectively than human annotators. this results in enhanced downstream task performance, with the model trained on the gpt-3 annotated dataset outperforming the one trained on the human-annotated dataset. subsequent studies have further demonstrated the capabilities of gpt-3, showing its ability to generate labeled data using external knowledge or instructions about desired labels and domains @xcite . additionally, researchers have examined the usefulness of newer models like gpt-3.5 and evaluated the effectiveness of cot in improving annotation quality @xcite . llmbased annotation has also been extended to lowresource languages where hiring human annotators is challenging @xcite . in this work, we introduce a novel approach to filtering noisy documents from multi-document summarization dataset by extending cost-efficient llm-based annotation beyond traditional data annotation tasks. by leveraging the capabilities of llms, our study facilitates real-world dataset cleansing, enhancing the quality of existing datasets. this attempt is noteworthy as it broadens the scope of llm applications, offering effective solutions for improving dataset quality and streamlining its cleansing process, minimizing reliance on human annotations. ## multi-news + the previous multi-news dataset plays an important role in multi-document summarization research. it consists of sets of documents and their corresponding summaries. however, as shown in to solve this issue and cleanse the dataset, we defined our problem as a classification task determining whether each document is relevant to the summary. to this end, we designed the prompt for the model as shown in appendix j. we integrated cot to enhance the model's performance by evaluating the relevance of each document to the summary. thus, a rationale for the decision can be made available, which marks the difference between llm-based and human annotations. while traditional human annotation through crowdsourcing platforms like amazon mechanical turk usually produces annotation results without underlying reasons due to additional costs, llm-based annotators can easily offer explanations through cot. these rationales can assist human managers in reviewing results and rectifying erroneous decisions. furthermore, we imitated the conventional dataset cleansing procedure which typically involves multiple human annotators and their collective judgments, primarily through majority voting. similarly to the majority voting process used by human annotators, we applied this approach to the llm-based annotators. in particular, we generated five individual llm agents to read the summary and documents and determine if the document is relevant to the summary. this strategy based on self-consistency can boost the quality of annotations, by rectifying potential errors made by individual agents @xcite . figure 1 presents the summary of the overall process. based on the proposed method, we utilized five llm agents to individually annotate 56,216 sets of summaries and documents from the multi-news dataset. specifically, we employed the gpt-3.5-turbo-0125 model foot_0 , the most recent model at the time of this study. with a prompt designed for a 3-shot cot, approximately 3,500 tokens were required to annotate the input summaries and articles, along with around 100 tokens for generating reasoning processes and annotation results. the cost per annotation sample amounted to approximately 0.01@xmath0 per agent), resulting in a total cost of approximately 550$ to annotate the after annotation, we found that 27,052 of the 153,091 articles can be considered noisy documents and do not contribute to the summarization. subsequently, we constructed multi-news + by removing these noisy documents from multi-news while preserving the train/valid/test split. figure 2 presents the comparison of the multi-news and multi-news + datasets in terms of the number of documents per set. more than 15% of the documents in multi-news are irrelevant, diminishing the dataset's quality and degrading the model's performance. furthermore, 379 sets have no relevant source articles, as shown in appendix h. in contrast, by deleting noisy documents, multi-news + demonstrates enhanced quality. ## experiment 4.1 experimental design to validate the efficacy of data cleansing and the development of multi-news + in filtering out noisy documents and improving the performance of downstream task models, we measured the multidocument summarization performance of models trained on each dataset, similar to previous study @xcite . enhanced model performance indicates superior dataset quality @xcite . we fine-tuned two different models, bart @xcite and t5 @xcite on multi-news and multi-news + . performance evaluation metrics included the following metrics: rouge @xcite , bertscore @xcite , and bartscore @xcite . for a fair comparison, we used the test set of multi-news + for each model and reported the average performance across three random seeds. ## discussion and future works in this section, we discuss recent advancements in the field since the submission of the manuscript and propose strategies for incorporating them in future research. cutting-edge models. although we employed five gpt-3.5-turbo-0125 models for our experiments, the field has seen the release of more advanced models, such as gpt-4o (openai, 2024b), gpt-4o-mini (openai, 2024a), and openai o1 (openai, 2024c), along with the continued development of open-source models like llama-3 @xcite , gemma-2 @xcite , and mistral nemo @xcite . models such as gpt-4o-mini and other opensource alternatives offer reduced costs compared to gpt-3.5-turbo-0125, making their adoption promising for both lowering the expense of dataset cleansing and improving the accuracy of detecting noisy documents. ## conclusion in this study, we suggest deploying cost-efficient llm-based data annotation to cleanse real-world datasets by identifying and excluding irrelevant and noisy data. we conducted a case study using this strategy to cleanse the multi-news dataset and proposed the improved multi-news + dataset. our case study revealed that multi-news + provides superior data quality compared to the original multi-news dataset. additionally, we have made multi-news + publicly available, thereby supporting further research in the field of multidocument summarization. our work paves the road to extending our data cleansing strategy to other datasets, broadening the scope of utilizing llms. this extension would enhance the quality of existing datasets across various domains without the need to construct new datasets from scratch. as such, our approach not only contributes to the advancement of multidocument summarization research but also offers a cost-efficient solution for enhancing dataset quality. we are committed to extending our llm-based method to other datasets, further solidifying its applicability to other tasks.