AL ii CE : Evaluating Positional Fine-grained Citation Generation (2025)

Abstract: Large Language Model (LLM) can enhance its credibility and verifiability by generating text with citations. However, existing research on citation generation is predominantly limited to sentence-level statements, neglecting the significance of positional fine-grained citations that can appear anywhere within sentences. To facilitate further exploration of the positional fine-grained citation generation, we propose ALiiCE, the first automatic evaluation framework for this task. Our method employs a dependency tree based approach to parse the sentence-level claim into atomic claims. Then ALiiCE evaluates citation quality using three metrics, including positional fine-grained citation recall, precision, and coefficient of variation of citation positions. We evaluate the positional fine-grained citation generation performance of several LLMs on long-form QA datasets. Our experiments and analyses demonstrate the effectiveness and reasonableness of ALiiCE. We offer our insights into the current advancements and future directions for the positional fine-grained citation generation task.

## introduction large language models (llms; @xcite can improve performance in several nlp tasks by incorporating external knowledge @xcite . in order to improve llms' credibility, @xcite ; @xcite propose a new paradigm for long-form qa, in which llms are required to provide citations to the retrieved passages for the statements they generate. since then, many studies @xcite @xcite have focused on how to enhance llms' citation generation capabilities. however, existing research on citation generation is predominantly limited to sentence-level what can cups be made of ? @xcite : one of the raw materials of the cup is glass… @xcite : plastic can be used to make cups of various… @xcite : tea or coffee rituals involve special cups… query documents a1: cups can be made of glass or plastic @xcite . sentence-level citation a2: cups can be made of glass @xcite or plastic @xcite . ## background & task definition in this section, we briefly introduce the background of our research and provide a definition of positional fine-grained citation generation. ## aliice: automatic llms' positional fine-grained citation evaluation in this section, we give a detailed description of aliice. first, we introduce how we construct the atomic claim parsing pipeline based on dependency trees. then, we present three metrics for the evaluation of positional fine-grained citation quality. ## experimental setup in this section, we describe the datasets and implementation details of our experiments. additional details are provided in appendix a. ## main results in this section, we present our key observations on the experiment results, and then provide our case study to prove the necessity of developing method for positional fine-grained citation evaluation. ## human evaluation we conduct human evaluation to examine the correlation between aliice and human judgment. since gao et al. (2023b); liu et al. (2023) have thoroughly studied sentence-level citation evaluation, we only focus on llms' responses that include positional fine-grained citations. in addition to the citation recall and precision, we also consider: 1) the proportion of positional fine-grained responses to total responses, 2) the answer utility, which assesses whether the llm's response is helpful in answering the question, and 3) the citation utility, evaluates whether the positional fine-grained citation is useful for the response. we recruit three annotators to evaluate the outputs of the models used in the previous experiments. we observe that aliice and human judgment show a strong correlation. the model rankings evaluated by aliice align closely with those evaluated by human judgement. the average cohen's kappa coefficients between aliice and annotators for asqa are 0.71 for citation recall and 0.62 for citation precision, demonstrating high consistency. in addition, responses containing fine-grained citations constitute a small proportion of the total output. for instance, the fine-grained output of gpt-3.5 (5-psg-summ) on eli5 accounts for only 8% of the total samples. this pattern is consistent with the results shown by cvcp. details on the human evaluation are provided in the appendix b. ## discussion based on the experimental results and observations, we discuss our insights on the task of positional fine-grained citation generation, as follows: aliice has a higher decision threshold. compared to alce, aliice calculates lower citation recall, but higher citation precision. and this difference becomes more dramatic when only positional fine-grained citation outputs are evaluated, as illustrated in ## related work attribution attribution refers to the ability of lms to generate and provide evidence @xcite . the source of attribution can be pre-training data @xcite , or out-of-model knowledge @xcite . when the source is documents, citation is a common form of attribution @xcite . retrieval-augmented generation retrievalaugmented generation (rag; @xcite combines the strengths of information retrieval and generation models, demonstrating improvement in several nlp tasks. the primary methods for incorporating external knowledge into generation include modifying model parameters @xcite and chain-of-thought (cot; @xcite . since rag exhibits a blackbox nature @xcite , adding citations in response can effectively mitigate the hallucination problem and enhance verifiability. ## conclusion in this study, we propose aliice, the first evaluation method for positional fine-grained citation generation. our approach incorporates an algorithm for parsing atomic claims based on dependency analysis, along with three metrics designed to assess the quality of positional fine-grained citations. we evaluate several llms and observe that currently, llms lack strong capabilities for generating fine-grained citations. we demonstrate the need of designing dedicated method for positional finegrained citation evaluation and the effectiveness of aliice in addressing this need. we also discuss some useful conclusions: 1) the latest open-source llms narrow the gap between them and closedsource llms in citation generation; 2) current metrics for citation quality lack consideration of citation utility; 3) the logical relationships between atomic claims can be considered when designing methods for positional fine-grained citation generation. we hope that our work can inspire more research into this underexplored task.