Intent-aware Schema Generation and Refinement for Literature Review Tables (2025)

Abstract: The increasing volume of academic literature makes it essential for researchers to organize, compare, and contrast collections of documents. Large language models (LLMs) can support this process by generating schemas defining shared aspects along which to compare papers. However, progress on schema generation has been slow due to: (i) ambiguity in reference-based evaluations, and (ii) lack of editing/refinement methods. Our work is the first to address both issues. First, we present an approach for augmenting unannotated table corpora with synthesized intents , and apply it to create a dataset for studying schema generation conditioned on a given information need, thus reducing ambiguity. With this dataset, we show how incorporating table intents significantly improves baseline performance in reconstructing reference schemas. We start by comprehensively benchmarking several single-shot schema generation methods, including prompted LLM workflows and fine-tuned models, showing that smaller, open-weight models can be fine-tuned to be competitive with state-of-the-art prompted LLMs. Next, we propose several LLM-based schema refinement techniques and show that these can further improve schemas generated by these methods.

## introduction a common use case of large language models (llms) is synthesizing information from collections of documents @xcite @xcite , a task that has become increasingly important in scientific domains amid the rapid growth of published literature @xcite . one important example of this task is the generation of literature review tables from sets of papers that compare them along shared aspects-a common target of literature review systems @xcite @xcite * work done during summer internship at ai2 table intent: describe the datasets used in the experiments to evaluate the performance? schema generation prompting fine-tuning papers schema aspect 1: <@xmath0... aspect 3 ... schema refinement improved schema (modified) aspect 1: <@xmath1... (new) aspect 3 ... candidate schema aspect 1: <@xmath2... aspect 3 ... table intent given a table intent or information need ( §2) via prompting and fine-tuning ( §3) as well as editing existing schema candidates ( §4). @xcite @xcite . typically, generating a literature review table involves two subtasks: (1) schema generation-identifying a relevant set of shared aspects to compare and contrast papers, which correspond to table's columns, and (2) value generationdetermining the content for each aspect-paper pair, i.e., filling in the table entries. while value generation can be framed as question-answering, drawing on recent advances in retrieval-augmented generation (rag) @xcite @xcite , schema generation remains a more open problem without an obvious reduction to standard tasks. identifying appropriate dimensions along which to compare documents is a longstanding problem in information organization and sensemaking @xcite . two issues have hampered progress on this problem: (1) ambiguity inherent in any reference-based evaluation process, and (2) lack of schema editing/refinement approaches given the open-ended nature of the task. our work is the first to address both these issues. any reference-based evaluation for schema generation is inherently ambiguous because there can exist multiple valid schemas for comparing the same set of papers. in fact, we argue the ideal choice of schema depends on the table's intentthe specific communicative goal or information need the author aims to address within the broader narrative of the paper, a notion missing from existing benchmarks @xcite . to illustrate this, consider figure 2 where the papers being compared concern graph theory. candidate 1 is a valid schema that selects aspects connected with the algorithms in both papers. however, this fails to match the actual table intent, which is meant to discuss the differences in the data used. explicitly incorporating the table intent yields a schema much better aligned with this need, such as candidate 2. we address this gap by introducing an approach to augment unannotated table corpora with inferred intents in the form of open-ended research questions reflecting different aspects along which users might compare papers. we demonstrate our approach on a recent table schema dataset, @xcite , showing that adding our intents as additional input to guide schema generation helps align model outputs with specific information needs, allowing for more principled evaluation and increased model performance ( §2). to address the lack of schema refinement methods and building on the observation that human authors often iteratively revise literature review tables, we evaluate a range of llm-based techniques to improve tables by editing them ( §4). we first establish a strong set of single-shot gen-eration methods by comprehensively benchmarking a broad range of prompting schemes for schema generation, varying both in how they use different levels of information from the papers being compared and in how they incorporate table intents into the pipeline @xcite @xcite . unlike prior work, we also fine-tuned open-weight models for schema generation ( §3.3) that are competitive to frontier llms already used in products, offering high value to the research community at a significantly lower inference cost. then, we examine three categories of llm-based techniques to refine existing schema candidates that support varying levels of user control: unguided editing, where a model is trained to revise schemas without explicit instruction, ( §4.1), heuristic-guided editing, where models are finetuned to perform atomic operations like adding or removing aspects, ( §4.2), critique-guided editing, where models edit schemas by generating and applying natural language critiques ( §4.3). however, generating reliable critiques is challenging, and that the small heuristic edits most consistently improves schema candidates ( §4.4). in all, our contributions are: (1) an approach for augmenting unannotated table corpora with intents, (2) benchmarking diverse schema generation methods (various prompting workflows vs finetuning, closed vs open models) demonstrating the value of table intents as well as identifying trade-offs in performance based on how various methods process papers, and (3) proposing three approaches to editing table schemas that demonstrate ways to consistently improve existing candidates. we contribute a range of open-source artifacts, including arx-ivdigestables augmented with table intents, and a suite of open-weight llms trained for schema generation and editing to further research. 1 ## inferring table intents for better task specification to assess llm capabilities on schema generation, we evaluate their performance on producing literature review tables, a task recently formalized by @xcite in the arxivdigestables dataset. this dataset is a collection of 2, 228 literature review tables scraped from arxiv papers. each instance consists of a table, where the rows represent each paper being compared and the columns represent the aspects of comparison. we treat the set of columns as the target schema. ## improving schema generation: prompting and finetuning with this improved task specification and evaluation strategy, we explore methods for better schema generation. we assess two categories of methods: (i) prompting-based methods that decompose schema generation into atomic sub-tasks ( §3.2), and (ii) fine-tuning language models ( §3.3). we evaluate all methods on arxivdigestables as well as arxivdigestables-clean, a subset we curate as described in §3.1, and report results in §3.4. ## refinement of schemas motivated by the iterative process humans follow while creating schemas @xcite , which 7 we confirm that this result holds with significance at the 5% level of a two-tailed t-test in §l. ## conclusion large language models have the potential to serve as valuable tools for literature review by automatically synthesizing schemas for comparing research papers. in this work, we show that existing formulations of the task miss an important component, an explicit specification of table intent. we show that augmenting arxivdigestables with synthetically generated table intents significantly improves schema generation performance when prompting llms. we also demonstrate that fine-tuning openweight models that match the performance of stateof-the-art black-box llms at a fraction of the model size. we also explore methods for refining schema candidates-unguided, heuristic-guided, and critique-guided edits-and show that openweight models can be fine-tuned to further improve schemas. we note limitations in the abilities of llms to critique existing candidates, highlighting an important direction of future work. to support future work, we release all fine-tuned models, our augmented arxivdigestables, and the manually curated arxivdigestables-clean.