A Closer Look at Claim Decomposition (2024)

Abstract: As generated text becomes more commonplace, it is increasingly important to evaluate how well-supported such text is by external knowledge sources. Many approaches for evaluating textual support rely on some method for decomposing text into its individual subclaims which are scored against a trusted reference. We investigate how various methods of claim decomposition—especially LLM-based methods—affect the result of an evaluation approach such as the recently proposed FActScore, finding that it is sensitive to the decomposition method used. This sensitivity arises because such metrics attribute overall textual support to the model that generated the text even though error can also come from the metric’s decomposition step. To measure decomposition quality, we introduce an adaptation of FActScore, which we call DecompScore. We then propose an LLM-based approach to generating decompositions inspired by Bertrand Russell’s theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposition quality over previous methods.

## introduction recent work uses claim decomposition to determine how well supported a claim is for applications in factual precision of generated text @xcite , entailment of human generated text @xcite , and claim verification @xcite @xcite , with similar ideas going back over a decade @xcite . in each of these cases, a claim is decomposed into natural language subclaims, 1 typically using a large language model (llm), and each sub-figure 1 : modes of claim decomposition. the extent to which textual support can be determined depends on how the generated text (yellow box) is decomposed into its subclaims (white boxes). higher quality decompositions enable more complete identification of discrepancies between generated text and a reference (not shown), which consequently increases the reliability of the downstream textual support metric. checks and xs denote that the statement is claimed or is not claimed, respectively, by the generated text. claim decompositions with various characteristics are shown in figure 1 . coverage denotes how much of the information in the claim is present in the subclaims, coherence denotes whether the information in the subclaims accurately reflects what is stated in the claim, and atomicity denotes how separated the information in each subclaim is. evaluating subclaims individually, as opposed to the entire claim at once, we can assign partial credit to a claim (e.g., for partial support), identify which parts of the claim differ from reference texts (such as a retrieved or pre-specified document or passage), and more easily identify relevant source material for each part of the claim. 2 claims can come from human-authored text based on cited documents @xcite or from machine-generated text based on dynamically provided grounding text or text observed during pre-training @xcite . since claim decomposition determines the number and scope of each evaluated subclaim, any analysis or resulting metric will be inherently tied to the decomposition method. nevertheless, prior work has left decomposition itself largely untested. how do different decomposition strategies affect downstream analysis? what are their qualitative and quantitative similarities and differences? we show that a downstream metric of textual support such as factscore @xcite is sensitive to the decomposition method it uses (figure 2 ). while factscore aims to measure the factual precision of generated text, the number and nature of the subclaims it evaluates from that text depend on the metric's claim decomposition method. the higher the quality of the decomposition method, and the better we understand its characteristics, the more we can attribute the factual precision that factscore aims to measure to the text generation model rather than to artifacts of the decomposition. finding that the method of claim decomposition matters, we introduce decompscore, an adaptation of factscore that measures decomposition quality, an important step in determining the reliability of the downstream metric. decompscore measures the number of subclaims supported by the original claim that was decomposed. because a decomposition with high atomicity and coverage will have more subclaims than a decomposition that doesn't, we then favor the decomposition method with the greatest decompscore, especially when 2 for example, separating the claim "charles babbage was a french mathematician" into the atomic subclaims "charles babbage was french" and "charles babbage was a mathematician" enables a claim verification system to determine that the subclaim about his occupation is supported by trusted reference documents and that the subclaim about his nationality is not supported. the non-atomic original claim as written, however, is not supported. with a way to compare decomposition methods in hand, we propose an llm-based decomposition approach inspired by bertrand russell's theory of logical atomism and neo-davidsonian semantics. our approach gives far more subclaims than other methods while maintaining high coherence with the claim being decomposed, and thus results in greater confidence in the entire pipeline for evaluating the level of textual support. our contributions are: name instruction in-context examples static dynamic sentences decompositions dfactscore "please breakdown the following sentence into independent facts:" (min et al., 2023) 7 1 min et al. (2023) min et al. (2023) dwice "segment the following sentence into individual facts:" (kamoi et al., 2023) 6 0 kamoi et al. (2023) kamoi et al. (2023) dchen et al. "given the following sentence, tell me what claims they are making. please split the sentence as much as possible, but do not include information not in the sentence:" (chen et al., 2023c) 7 1 min et al. (2023) min et al. (2023) dconll-u "the sentence below is given in conll-u format. word lines contain the annotation of a word/token/node in 10 fields separated by single tab characters. sentences consist of one or more word lines. please break down the following sentence given in conll-u format into independent facts:" 1 1 min et al. (2023) + conll-u parse min et al. (2023) dr-nd "please decompose the following sentence into individual facts:" 7 1 min et al. (2023) manual (ours) and then aggregate results to give a single score or label for the entire claim. since each subclaim is evaluated, we get a localized view of which parts of the claim are supported. the more atomic the subclaims are, the more precisely we can localize the information in the claim that differs from a trusted reference. since these approaches rely on decomposition, the better the decomposition method the more reliable the results. factscore @xcite measures factual precision of model-generated text with respect to a knowledge source. a generated passage is split into sentences, which are decomposed into subclaims by an llm. the percentage of subclaims supported by a retrieved knowledge source (e.g., wikipedia excerpts) is the factscore for the passage. faithscore @xcite takes a similar approach for evaluating the outputs of vision-language models, in which the knowledge source against which the subclaims are evaluated is an image. they additionally require that the subclaims fit into certain domain-specific categories such as color and count. the wice dataset @xcite contains annotations for whether subclaims in humanwritten text are supported, partially supported, or not supported by external reference documents, from which claim-level support labels are derived. @xcite also apply their llm-based claim-split approach to entailment classification, in which entailment scores for each subclaim are aggregated to give an entailment score for the whole claim. ## evaluating decomposition quality previous work on evaluating the veracity of generated text attributes the resulting score to the quality of the generation, overlooking the role of metric's decomposition step. however, higher quality decompositions mean that we can more reliably measure the quality of the generation. depending on the characteristics of the decomposition method (e.g., how atomic its decompositions are), a metric like factscore can change for the same underlying generated text (figure 2 ). furthermore, factscore implicitly assumes complete and coherent decompositions. however, the decomposition step can introduce unclaimed information or omit existing (possibly incorrect) claims, which introduces measurement error into factscore. ## methods of claim decomposition we study three types of claim decomposition methods, which are discussed below. ## russellian and neo-davidsonian decomposition the notion of claim decomposition has roots in the philosophical literature. we draw inspiration from bertrand russell's theory of logical atomism for how claims should be decomposed into their atomic components. russell defines atomic facts as properties of individuals or relations between individuals from which all other facts are composed @xcite . 7, 8 we take individuals to be entities and eventualities mentioned in the sentence. this kind of russellian analysis accords with neo-davidsonian analysis @xcite (building on @xcite ), in which the logical form of a sentence is decomposed fully to a conjunction of unary predicates (akin to properties of individuals) and binary predicates (akin to relations between individuals). we manually decompose the 21 in-context examples from @xcite into lists of such russellian atomic propositions that we further decompose following neo-davidsonian intuitions into unary and binary relations to obtain the smallest units that are claimed in each sentence: each subclaim designates a property of an individual or a relation between two individuals. 9 our decompositions are listed in ## data we use the released data from @xcite , which consists of biographies of 500 individuals generated from each of 12 lms (following their notation, we call the text generation models lm subj ). 10 we do not modify the biographies generated by @xcite , nor do we generate 7 ludwig wittgenstein theorizes a similar idea of elementary propositions that assert atomic "states of affairs". on the whole, we find wittgenstein's theory to be less actionable than russell's. incidentally, wittgenstein later abandoned this theory in part due to the color exclusion problem, which we avoid by not requiring independence of subclaims, instead requiring only that each subclaim is claimed by the sentence. ## experiments we use the data described in §6 for sentence-level decomposition with the methods outlined in §4 and §5. model specifications are listed in appendix b. we evaluate using decompscore with inst-llama from @xcite (llama trained on super natural instructions @xcite ) and factscore with the inst-llama + retrieval + npm setting. in total, generating decompositions took 120 gpuhours, computing decompscore took 250 gpuhours, and computing factscore took 450 gpuhours, all using a quadro rtx 6000. ## results ## conclusion we observe that a downstream metric of textual support, namely factual precision as measured by factscore, is sensitive to the method it uses to decompose a claim into its subclaims. this finding leads us to measure decomposition quality using our proposed metric decompscore so that we can use the most appropriate decomposition method among those we consider. we show that an llm prompted with in-context learning examples that we manually decompose by following intuitions from logical atomism and neo-davidsonian semantics outperforms other methods. decompositions generated by our method contain the greatest number of subclaims supported by the original claim among the methods we consider. qualitative analysis and comparison to manual decompositions demonstrate that all the decomposition methods we consider still miss subclaims and many generate non-atomic subclaims, indicating there still remains room for improvement.