W eb C ite S : Attributed Query-Focused Summarization on C hinese Web Search Results with Citations (2024)

Abstract: Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still exhibit notable limitations. In this work, we formulate the task of attributed query-focused summarization (AQFS) and present WebCiteS, a Chinese dataset featuring 7k human-annotated summaries with citations. WebCiteS derives from real-world user queries and web search results, offering a valuable resource for model training and evaluation. Prior works in attribution evaluation do not differentiate between groundedness errors and citation errors. They also fall short in automatically verifying sentences that draw partial support from multiple sources. We tackle these issues by developing detailed metrics and enabling the automatic evaluator to decompose the sentences into sub-claims for fine-grained verification. Our comprehensive evaluation of both open-source and proprietary models on WebCiteS highlights the challenge LLMs face in correctly citing sources, underscoring the necessity for further improvement. The dataset and code will be open-sourced to facilitate further research in this crucial field.

## introduction in today's information-driven society, swift access to knowledge is essential. a major limitation of web search engines is the need for users to manually compile information from various sources, which can be time-consuming. large language models (llms) @xcite exhibit potential in this domain by generating straightforward and well-organized responses. however, the potential risks of hallucination @xcite and factual errors @xcite undermine their trustworthiness as knowledge sources. there are several possible reasons for papaya tasting a bit bitter. on the one hand, papaya naturally has a slight bitterness, especially when not fully ripe @xcite . on the other hand, papaya may taste bitter due to being cut, stifled ripe, or affected by frost @xcite [4] … retrieved documents [2]: … as the flavor of papaya is naturally a bit bitter. however, if it's … an emerging solution is generative search engines @xcite which use llms to synthesize web search results into responses with in-line citations. this allows users and developers to verify the generations against the cited sources. however, recent investigations on commercial products and retrieval-augmented llms reveal frequent occurrences of unsupported claims and incorrect citations @xcite , highlighting the challenges of attribution in llms @xcite . nonetheless, the limitations of pertinent datasets and evaluation methods pose obstacles to in-depth explorations within the community. firstly, most existing datasets are deficient in high-quality citation annotations. for instance, the alce benchmark @xcite compiles three question-answering datasets @xcite @xcite without providing citations in the reference answers, limiting its utility for model training. in contrast, webglm @xcite prompts instruct-gpt @xcite to generate training data with citations. it controls the citation quality via a sample filtering method which calculates the rouge @xcite score between the answers and their citations. however, this method focuses on lexical similarity rather than logical entailment, webcites zh real user queries web search results 7k 1025 / 3970 167 1.55 human auto webcpm (qin et al., 2023) zh translated reddit questions web searh results 5k 513 † 244 † 0.34 human n/a webglm (liu et al., 2023b) en eli5 web searh results 45k 304 104 1.20 auto human alce (gao et al., 2023b) en asqa qampari, eli5 wikipedia common crawl 3k 100*k 78 n/a n/a auto secondly, current evaluation methods are insufficient to thoroughly assess attribution. prior works only inspect if the generations are supported by their citations @xcite @xcite without checking all the documents provided in the input context. however, instances of unsupported generations may result from both failing to correctly cite supporting documents and failing to be grounded in all input documents. differentiating these two types of errors is crucial for system optimization. moreover, existing automatic evaluation @xcite solely relies on off-the-shelf natural language inference (nli) models which only recognize entailment (full support) and overlook sentences with multiple sub-claims drawing partial support from different sources. such complexities are common in real-world scenarios @xcite and are indicative of a strong capability of synthesizing information across various sources. to address the above limitations, we present we-bcites, a chinese dataset for attributed query-focused summarization (aqfs). as shown in figure 1 , given a query and the retrieved documents, aqfs aims to summarize all pertinent information from the documents with in-line citations to make the summary attributable. our dataset is built upon real-world user queries and search results from sogou, a widely used chinese web search engine. 2 we employ human efforts to en-2 www.sogou.com sure the quality of summaries and citations. we propose a comprehensive evaluation framework with a cost-effective automatic evaluator. our evaluation metrics distinguish two key aspects: groundedness (if the model outputs are contextually supported) and citation quality (citation accuracy and comprehensiveness), enabling a more nuanced understanding of attribution errors. we also train a tailored claim-split model to extract the sub-claims of a sentence for fine-grained verification. this allows the detection of partial support and improves the alignment between our automatic evaluator and human citations. our evaluation of both open-source and proprietary models on webcites reveals the following key findings: (1) contextual grounding of generations does not guarantee the avoidance of citation errors, indicating the challenge of explicit attribution in all the tested llms; (2) although supervised fine-tuning improves both groundedness and citation quality, the top-performing model only reaches a citation f 1 score of 76.1% with about 20% of sentences not being fully supported by their citations, underscoring the need for further optimization; (3) models perform worse when summarizing full content of web pages rather than shorter snippets, showing that llms are less effective at synthesizing and attributing information in the longer context; (4) making documents more fine-grained leads to poorer attribution results, highlighting the difficulty llms face in pinpointing the exact supporting evidence within the context. query search results [1] … [5] stage 1: manual screening and information extraction query extraction candidate 1 candidate 2 stage 2: llm-based candidate summary generation human-annotated summary stage 3: manual refinement and citation annotation [2][3] [4][5] [1] ## the webcites dataset in this section, we first formulate the aqfs task and then delineate the construction of webcites. ## evaluation framework the aqfs task targets two dimensions: summarization utility and attribution. in this section, we introduce the evaluation metrics based on an evaluator with two key components: a claim-split model ψ and an nli model ϕ. the claim-split model decomposes each sentence s i into a set of sub-claims ψ(s i ) = {c i,1 , c i,2 , . . .}. the nli model ϕ predicts if the given premise entails, contradicts, or is neutral to the given hypothesis. we report the performance of ϕ and ψ in section 4. ## evaluating the automatic evaluator in this section, we assess the reliability of the automatic evaluator with ϕ and ψ. ## experiments on the aqfs task we evaluate various models on the aqfs task via two methods: few-shot prompting (fsp) and supervised fine-tuning (sft). our prompt for fsp consists of the task instruction and one-shot demonstration, while the sft prompt removes the demonstration and condenses the instruction for efficiency. for open-source models, we evaluate mt5, chatglm2-6b, chatglm3-6b @xcite , baichuan2-7b, and baichuan2-13b (yang et al., 2023) via both fsp and sft. 8 for proprietary models, we evaluate chatgpt and gpt-4 via fsp. 9 see appendix e.1 for implementation details. ## conclusion in this work, we formulate the task of attributed query-focused summarization (aqfs) and present webcites, a high-quality dataset derived from realworld user queries and search results. we propose a comprehensive evaluation framework on summarization utility and attribution. notably, we highlight two fine-grained dimensions of attribution: groundedness and citation quality. we further enhance the framework with a carefully-designed automatic evaluator, and validate its substantial agreement with human annotators. finally, we evaluate both open-source and proprietary llms extensively on webcites to underscore the unsolved challenge of attribution, especially in the long-context setting with fine-grained documents. we believe webcites could facilitate more future explorations in attributable language models.