Scalable and Domain-General Abstractive Proposition Segmentation (2024)

Abstract: Segmenting text into fine-grained units of meaning is important to a wide range of NLP applications. The default approach of segmenting text into sentences is often insufficient, especially since sentences are usually complex enough to include multiple units of meaning that merit separate treatment in the downstream task. We focus on the task of abstractive proposition segmentation (APS): transforming text into simple, self-contained, well-formed sentences. Several recent works have demonstrated the utility of proposition segmentation with few-shot prompted LLMs for downstream tasks such as retrieval-augmented grounding and fact verification. However, this approach does not scale to large amounts of text and may not always extract all the facts from the input text.In this paper, we first introduce evaluation metrics for the task to measure several dimensions of quality.We then propose a scalable, yet accurate, proposition segmentation model. We model proposition segmentation as a supervised task by training LLMs on existing annotated datasets and show that training yields significantly improved results. We further show that by using the fine-tuned LLMs (Gemini Pro and Gemini Ultra) as teachers for annotating large amounts of multi-domain synthetic distillation data, we can train smaller student models (Gemma 1 2B and 7B) with results similar to the teacher LLMs. We then demonstrate that our technique leads to effective domain generalization, by annotating data in two domains outside the original training data and evaluating on them. Finally, as a key contribution of the paper, we share an easy-to-use API for NLP practitioners to use.

## introduction from retrieval systems that build indices over passages rather than documents @xcite , to automatic evaluation metrics for generative tasks that evaluate sentence-level similarity to references (e.g. amplayo et al. ( 2023 )), to structured event representations used for crossdocument summarization @xcite , segmenting a document into significantly finer units that retain relevant meaning is a major component of many nlp systems. in "well-formed" prose, an easy and frequently used choice for segmenting documents is sentence segmentation. but for most applications, sentences are an imperfect fit: they are often still too complex, containing multiple units of underlying information @xcite ; they typically require context from elsewhere in the document to understand the meaning @xcite . furthermore, well-formed sentences are not always available in situations ranging from casual speech @xcite to non-prose formats @xcite , where "sentences" are not even a natural unit of discourse. to provide useful fine-grained segmentation, several recent works have taken the approach of proposition segmentation foot_0 @xcite @xcite , seeking to break text into fine-grained, minimal units of meaning that together convey all the information in the source text. similarly to the extractive-vs-abstractive contrast in the summarization literature, the two strands of proposition segmentation work so far have considered either (a) an extractive approach, representing propositions as one or more spans in the source text @xcite ; @xcite ; etc.); or (b) few-shot llm prompts for abstractive proposition segmentation, generatively writing out each unit as a well-formed sentence @xcite @xcite . more formally, abstractive proposition segmentation (aps), the focus of this paper, is to transform a given document into a collection of propositions represented as natural-language sentences which: 1. are atomic and minimal semantic unit that cannot be further decomposed into meaningful units @xcite ; 2. are fully decontextualized @xcite -i.e. they can be understood just as well with no access to the rest of the document; 3. present information explicitly given in the document; and 4. when taken together, cover all of the information in the document. aps has already found applications in grounding @xcite , summarization evaluation @xcite , and fact checking @xcite . in this paper, we focus on making abstractive proposition segmentation practical. the few-shot prompting approaches are typically too costly to run at large scales, and, furthermore, we show that they tend to under-extract compared to our proposed solutions. our core contributions are: ## related work linguistic compositionality, the idea that sentences are comprised by smaller units of meaning, has been debated since the early 1800s @xcite , and understood surely long beforehand. in the context of modern nlp, the value of proposition segmentation for standard tasks can be seen from the empirical measurements in, e.g., @xcite , which shows for document-level nli that 72% of sentences partially aligned between two highly related documents don't fully entail each other, and in @xcite , which shows that 40% of chatgpt sentences at that time contained a mix of supported and unsupported propositions. indeed, several previous results have shown aps by few-shot prompted llms benefits retrievalaugmented fact verification and grounding @xcite . a concurrent result in @xcite looks more specifically at aps itself with few-shot prompting. @xcite also perform few-shot prompting followed by distillation. other formats of proposition segmentation have also been explored. extractive proposition segmentation is shown in @xcite to benefit document-level nli and retrieval. several openbook qa and grounding works have generated finegrained questions corresponding implicitly to the fine-grained claims in the text @xcite @xcite . in the summarization evaluation literature, "summary content units", initially human-annotated @xcite , later generated from syntactic signals @xcite have long been used for summary evaluation. before the llm era, decomposing text into semantic triples, known as open information extraction @xcite , drove a variety of downstream applications. our desiderata for proposition segmentation include context-independence, earlier studied at the sentence level by @xcite . @xcite perform document-level claim extraction for fact checking. they specifically extract claims that are check-worthy, where these claims are decontextualized, but not necessarily atomic. in our work, we propose a suite of automatic evaluation metrics. previous efforts have not focused much on metric definition. the exceptions are two concurrent works: a) @xcite propose a specific single metric for aps, decomp-score, that combines our reference-free precision metric ( §3.2) with the count of claims generated. b) @xcite propose metrics based on rouge @xcite and similar to our referencebased precision and recall. our metrics are based on nli that is more suitable for checking semantic equivalence of predicted and gold propositions. ## abstractive proposition segmentation in this section, we formally define the task ( §3.1) and propose metrics ( §3.2). ## domain-general aps given an input text (passage) t, our goal is to generate a list of propositions {p 1 , . . . , p k }, where propositions should be well-formed, atomic, selfcontained, supported, and comprehensive. in this section, we discuss our proposed method to distill a relatively small, yet domain general proposition segmentation model: a) we train a teacher llm on an existing proposition segmentation dataset ( §4.1). b) we generate a large set of multi-domain synthetic data with different lengths ( §4.2). c) we generate a large synthetic dataset with pairs of (text, propositions list) and train a student model on it ( §4.3). ## experiments we explore the effectiveness of our distillation approach for training a scalable and domain-general proposition segmentation approach. we describe the datasets we have used for training and evaluation ( §5.1). we then introduce our baselines ( §5.2). we first compare our proposed method with multiple baselines on the rose dataset ( §5.3). we then show that our method is effective on two datasets from new and unseen domains ( §5.4). ## human correlation studies for aps metrics we measure how well our defined metrics ( §3.2) align with human judgements in order to vali-input text ## the propositions api we showed that our student models resolve two issues with the commonly used few-shot prompting approach: under-extraction (low rl r ) and cost. 11 the reference-less precision metric is almost always equal to 1 except for a few examples. the nli accuracy compared to human annotation is 0.985. in addition, both the automatic metric and the human evaluated metric @xmath0.98. ## conclusion we define the abstractive proposition segmentation task more formally by specifying the desired properties of propositions and present a suite of automatic evaluation metrics that allow us to measure different dimensions of quality. while previous work often uses few-shot prompting, we show that supervision from existing datasets yields significant quality improvement. we then propose a distillation approach for training scalable and domain-general models that get on-par results with the teachers (and sometimes even better). we release an api based on gemma 7b student models and invite researchers to use that instead of fewshot prompting llms.