A rxiv DIGEST ables: Synthesizing Scientific Literature into Tables using Language Models (2024)

Abstract: When conducting literature reviews, scientists often create literature review tables—tables whose rows are publications and whose columns constitute a schema, a set of aspects used to compare and contrast the papers. Can we automatically generate these tables using language models (LMs)? In this work, we introduce a framework that leverages LMs to perform this task by decomposing it into separate schema and value generation steps. To enable experimentation, we address two main challenges: First, we overcome a lack of high-quality datasets to benchmark table generation by curating and releasing arxivDIGESTables, a new dataset of 2,228 literature review tables extracted from ArXiv papers that synthesize a total of 7,542 research papers. Second, to support scalable evaluation of model generations against human-authored reference tables, we develop DecontextEval, an automatic evaluation method that aligns elements of tables with the same underlying aspects despite differing surface forms. Given these tools, we evaluate LMs’ abilities to reconstruct reference tables, finding this task benefits from additional context to ground the generation (e.g. table captions, in-text references). Finally, through a human evaluation study we find that even when LMs fail to fully reconstruct a reference table, their generated novel aspects can still be useful.

## introduction conducting literature reviews by reading and synthesizing information across a large set of documents is vital for scientists to stay abreast of their fields yet is increasingly laborious as the number of scientific publications grows exponentially @xcite . at the core of this sensemaking process is identifying a schema, a set of important aspects that are useful for comparing and contrasting prior literature @xcite . the results of this process are often presented in the form of literature review tables, whose rows are a set of papers and whose columns are a set of aspects that the papers share (figure 1 ). in this work, we conceptualize the task of literature review table generation by decomposing it into two sub-tasks: (1) schema-generation: determining a set of relevant shared aspects given a set of input papers, and (2) value-generation: determining the value given an aspect and a paper. for example, a table for a set of computer vision papers on video datasets (rows) might have a schema with aspects like "task" or "size" (columns); cell values under the "task" column may say "vqa" or "classification" @xcite . prior work has largely investigated each of the two sub-tasks independently. in particular, the large body of literature on document-grounded question-answering @xcite @xcite , information extraction @xcite , and query @xcite or aspect-based summarization @xcite ., paper 4 sports-1m 1,133,158 classification -(auto) original reference table model generated table figure 2: side-by-side comparison of a reference literature review table from an arxiv paper and a model-generated table given the same input papers. the generated table has reconstructed two gold aspects: the pink and blue aspects are the same, despite surface form differences (e.g., "task" vs "intended application"). the generated table has also proposed two novel aspects that are still relevant and useful, like "evaluation metric" (green) or "annotation method" (yellow) not to be confused with reference table's "annotations". in contrast, schema generation from a set of documents remains relatively under-explored, even though it is a crucial and effortful part of the manual literature review process. prior work like @xcite infers new schemas from pre-existing ones, while recent work like @xcite assumes users can clearly articulate a schema in a short natural language query to infer aspects directly. this paper studies the use of language models for literature review table generation with a focus on unifying these two sub-tasks. this presents us with two research challenges: first, we note a lack of large-scale, high-quality datasets of literature review tables to serve as a benchmark for this task. second, similar to challenges faced in summarization and other grounded generation tasks, semantically similar content can be expressed with different surface forms, which makes automatic evaluation difficult even with a high-quality dataset. an example of these surface form differences is in figure 2 . to address these challenges: we formalize the literature review table generation task ( §3) and introduce our framework for literature review table generation and detail our implementations using open and closed models ( §4). finally in §6, we evaluate lms on this generation task, addressing two key questions: (1) what contextual information is needed to steer language models to reconstruct human-authored schemas? and (2) are generated aspects that don't match gold still useful? for (1), we find that language models have higher recall by conditioning on more context that specifies the purpose of the table (e.g., captions, in-line references, other example tables). for (2), we find that novel aspects not in the reference tables can still be of comparable usefulness, specificity, and insightfulness. ## creating arxivdigestables desiderata to enable research in synthesizing literature review tables, we first collect and curate a set of reference tables to ground our task and enable evaluation. to ensure this data is realistic, high-quality, and focused on supporting literature review, we decide on the following desiderata for including tables in our arxivdigestables dataset: 1. tables should be ecologically valid-reflecting real syntheses authored by researchers rather than artificial annotation; 2. tables should be focused on summarizing multiple aspects of a set of papers as opposed to tables for reporting empirical results; extracting tables the first step in our data collection pipeline is to extract the tables from papers published on the arxiv preprint server. to start, we consider approximately 800,000 papers that have latex source available. we then use unarxive @xcite to convert the arxiv source into xml. from these xml documents, we extract ∼2.5 million tables. filtering tables as a first filtering pass, we remove tables that are likely to be misparsed or unusable, filtering those with fewer than 400 or more than 15,000 characters. we also remove tables that have no table cell tags within them. toward desiderata 3, we filter out tables that have fewer than two citations, two rows, or two columns. we also remove any tables that have citations in more than one column, as these are often tables where papers are values rather than rows. this leaves approximately 211,000 tables. matching rows to papers we use heuristics to convert xml-formatted tables into json objects that allow us to directly index the tables by paper and aspect (see §a.1 for details). at this stage, the citation information is usually contained within a cell in a table. for instance, an example cell with the header "model" might have the value "bert (devlin et al., 2019)". we extract the citations from these cells and place them in their own column called "references". rows without citations are assumed to refer to the source paper containing the table. after this step in the process we remove any tables where the algorithm failed and any tables that now have fewer than two rows, leaving 47,876 tables. obtaining table citation metadata unarxive @xcite helpfully links each citation in the table to a bibliography item. we use endpoints from the semantic scholar api @xcite to obtain titles and abstracts. this occasionally fails for various reasons (e.g., the bibliography text is missing information, the paper is missing from or could not be found in the semantic scholar database). we filter out any tables that have fewer than two matched citations, leaving us with 44,617 tables. fi gure 3: pipeline for curating arxivdigestables involves extensive data cleaning and filtering. the full pipeline filters from 2.5 million starting tables published in 800,000 papers to 2,228 tables published in 1,723 papers. ## literature review table generation equipped with our dataset, we formalize the task of generating literature review tables. ## experiments we prompt language models to perform either joint or decomposed generation. ## developing an automatic metric below we describe the design of our automatic evaluation procedure with two components: evaluating the schema and values for a generated table. ## conclusion language models have the potential to help scientists organize papers during literature review by synthesizing tables with schemas that aid comparison. in this work, we curate arxivdigestables, a dataset of such tables and additional contexts that can be used to evaluate systems' abilities to produce such tables. we present decontexteval, an automatic evaluation framework for comparing model-generated and human-authored reference tables. we then use this evaluation framework to investigate two research questions: what context is needed to reconstruct human-authored tables, and whether generated aspects that don't align with references are also useful, specific and insightful. we release our artifacts to help spur development of literature review table generation systems, and seed potential for their role in evaluating systems' scientific synthesis abilities.