Can Large Language Models Fix Data Annotation Errors? An Empirical Study Using Debatepedia for Query-Focused Text Summarization (2023)

Abstract: Debatepedia is a publicly available dataset consisting of arguments and counter-arguments on controversial topics that has been widely used for the single-document query-focused abstractive summarization task in recent years. However, it has been recently found that this dataset is limited by noise and even most queries in this dataset do not have any relevance to the respective document. In this paper, we study whether large language models (LLMs) can be utilized to clean the Debatepedia dataset to make it suitable for query-focused abstractive summarization. More specifically, we harness the language generation capabilities of two LLMs, namely, ChatGPT and PaLM to regenerate its queries. Based on our experiments, we find that solely depending on large language models for query correction may not be very useful for data cleaning. However, we observe that leveraging a rule-based approach for data sampling followed by query regeneration using LLMs (especially ChatGPT) for the sampled instances may ensure a higher quality version of this dataset suitable for the development of more generalized query-focused text summarization models.

## introduction text summarization is a natural language processing technique that involves generating a concise and coherent summary of a longer piece of text while preserving its most important information @xcite . query-focused text summarization is a specific type of summarization that generates a summary of the given text that is focused on answering a specific question @xcite or addressing a particular topic, rather than providing a general overview of the text. @xcite @xcite @xcite . one widely used dataset for this task is the debatepedia dataset that consists of arguments and counter-arguments on conversational topics @xcite . the query-focused summarization of argumentative text is a challenging task that has gained increasing attention in recent years due to its potential applications in various domains, such as policy-making, journalism, and legal reasoning. however, it has been recently found that the quality of the debatepedia dataset that is widely used for this task is limited by noise, with many of the queries in this dataset having no relevance with the source document @xcite . since debatepedia is a rich source of argumentative text on controversial topics that can serve as a valuable resource for the development and evaluation of summarization models, in this paper, we present a novel methodology to clean the debatepedia dataset via re-annotation of its queries to make it a useful resource for query-focused abstractive summarization. our data annotation approach leverages large pre-trained language models @xcite @xcite , such as chatgpt @xcite and palm-2 @xcite , that have demonstrated impressive capability of generating fluent and coherent text @xcite . using these llms, we regenerate the queries in the debatepedia dataset that are more likely to have no relevance to the document and the summary. more specifically, this paper aims to investigate whether llms can be utilized to fix the existing issues in the debatepedia dataset. our extensive experiments show that utilizing rule-based filtering to eradicate noisy instances alongside leveraging the generative power of llms to regenerate the irrelevant queries leads to performance improvement in terms of both query relevance and summary generation quality. we will make this llm-annotated cleaned version of debatepedia publicly available foot_0 . ## related work query-focused text summarization using neural models has gained increasing attention in recent years @xcite . the recent success of transformer-based models @xcite @xcite on generic abstractive summarization has also inspired researchers to utilize such models for query-based abstractive summarization @xcite @xcite , leading to stateof-the-art performance in benchmark query-based summarization datasets, such as duc @xcite , aquamuse @xcite , qmsum @xcite , debatepedia @xcite , etc. among the datasets mentioned above, one notable exception is the debatepedia dataset since it requires generating summaries from a document containing argumentative text (i.e., arguments and counter-arguments). however, it has been found recently that many samples in the debatepedia dataset are not actually query oriented while models that are trained without considering the query relevance could achieve almost similar performance as the query-focused summarization models @xcite . thus, there remains a scarcity of datasets specifically tailored to generate query-focused summaries of argumentative texts. though some studies (abdullah and chali, 2020) have attempted to generate the queries in generic summarization datasets (e.g., cnndm @xcite ), we find that these queries are generated by directly extracting words from the reference summaries, leading to unexpected access to the keywords in the reference summaries for the summarization models. llms have received a lot of attention recently due to their impressive language generation capability -ensuring high fluency, coherence, and grammatical correctness on the generated texts @xcite @xcite @xcite . more importantly, chatgpt like llms also demonstrated impressive capability for data annotation @xcite @xcite . to this end, in this paper, we study how to fix the queries in debatepedia using llms to construct a cleaned version of the dataset to make it suitable for queryfocused summarization of argumentative texts. ## our annotation methodology debatepedia is a publicly available dataset of arguments and counter-arguments on debate topics, proposed by nema et al. @xcite . it contains about 13k query-document-summary pairs. the average number of words per document, summary, and query in the debatepedia dataset is 66.4, 11.16, and 9.97, respectively. the dataset covers a wide range of topics, such as politics, sports, and technology, and has been extensively used in recent years to build query-based summarization models for argumentative text @xcite . however, the quality of debatepedia as a dataset for query-based summarization has lots of limitations (see ## experimental results in this section, we present our experimental findings. we denote the version of our dataset where we did not apply any filtering as the unfiltered version, whereas we denote the version of our dataset where we also utilize filtering while sampling data instances based on applying some rules as the filtered version. for chatgpt, we use the gpt-3.5-turbo-0301 foot_2 model; while for palm-2, we use the text-bison@001 foot_3 model. we finetune the following models to benchmark the performance in our re-annotated versions of debatepedia since these models achieved impressive performance in query-focused abstractive summarization in recent years @xcite ): (i) bart @xcite , (ii) t5 @xcite , and (iii) pegasus @xcite . similar to prior work on query-focused text summarization (laskar et al., 2022), we concatenate the query with the document and give as input to these models to generate the query-focused summaries. for all models, we use their respective base versions from huggingface @xcite (see appendix a.3 for more details). for all models, the results are evaluated using rouge-1, rouge-2, rouge-l, and bertscore. for bertscore, we use the deberta-xlarge-mnli (he et al.) model. ## conclusions and future work in this paper, we study how to effectively leverage llms to construct a cleaned version of the debatepedia dataset to address the existing limitations in this dataset in order to make it suitable for queryfocused text summarization. based on extensive experiments and evaluation, we demonstrate that our proposed data re-annotation approach using llms (especially chatgpt) results in a cleaner version of debatepedia that is found to be more effective for the query-focused summarization task in comparison to the original dataset. in the future, we will explore whether few-shot examples with llms lead to better performance. our re-annotated versions of debatepedia will also be made publicly available here: https://github.com/tahmedge/cqsumdp .