Identifying Quantifiably Verifiable Statements from Text (2023)

Abstract: Humans often describe complex quantitative data using trend-based patterns. Trend-based patterns can be interpreted as higher order functions and relations over numerical data such as extreme values, rates of change, or cyclical repetition. One application where trends abound are descriptions of numerical tabular data. Therefore, the alignment of numerical tables and textual description of trends enables easier interpretations of tables. Most existing approaches can align quantities in text with tabular data but are unable to detect and align trend-based patterns about data. In this paper, we introduce the initial steps for aligning trend-based patterns about the data, i.e. the detection of textual description of trends and the alignment of trends with a relevant table. We introduce the problem of identifying quantifiably verifiable statements (QVS) in the text and aligning them with tables and datasets. We define the structure of these statements and implement a structured based detection. In our experiments, we demonstrate our method can detect and align these statements from several domains and compare favorably with traditional sequence labeling methods.

## introduction there is a wealth of information locked in the numerical tables, spanning different domains and real world applications (e.g., financial reports). since numerical tables can contain dense, highdimensional quantitative data, they are often accompanied by textual descriptions that support easy interpretation. in many cases, these textual interpretations are used without inspecting the raw data in the numerical table. 1 when humans generate textual descriptions of numerical data, they rarely refer to the individual quantitative points, but frequently use trendbased patterns in their statements. trend-based 1 non-archival submission patterns in numerical data are description of functions and patterns over one or more dataset points in the numerical dataset. in other words, trendbased patterns are created by quantitative analysis over numerical data. the ability to identify these statements and their underlying data source is a prerequisite for many tasks such as fact-checking, natural language understanding in specific domains (e.g stock market), question answering and etc. to date, many existing works @xcite @xcite have focused on the extraction of subject, predicate, object triples from text. triple representations readily align with factual data stored as triples in knowledge graphs. however, in a trend-based statement, the supporting data is generally a derived measure on dimensionally-aligned data (such as time series) which cannot readily be validated with triple-based data. another major limitation of current extraction and alignment methods @xcite @xcite is that they are limited to the statements with first-order trends and are unable to detect and match the second-order descriptions over quantities. in this work, we focus on statements containing higher order trends about numerical data. these higher order trends are created by quantitative analysis over data. hence, their detection and alignment requires linguistic, symbolic and quantitative reasoning. in this paper, we introduce a pipeline, for identification and alignment of quantifiably verifiable statement i.e., statements that contain trend-based patterns about data. in the first step of our pipeline, the quantifiably verifiable statements are identified. then they are aligned with the relevant evidence from a pre-collected dataset. we define a quantifiably verifiable statement (qvs) as a textual span that expresses a numerical relationship in a dataset and can be objectively validated using an authoritative data set. for example, the statement "us gas prices rose in 2018." describes a change in value (rise) and can be objectively validated using a dataset of commodity pricing information collected by the world bank. in the effort to align these statements, the detection component converts these statements into an indicator and a trend structure, formally defined in the next section. intuitively, indicators allow a system to identify a specific dataset as reference dataset that is described the claim, while the trend expresses a particular data relationship that can be computationally checked on the data. the next step after identifying qvs is the alignment. the alignment step aims to find the relevant information that can be used in verification of the statement. in this paper, we presume the relevant information appears as datasets from which the qvs can be generated without any external source of information or reasoning step. for example, the statement " house prices continued their record-setting growth into may," can be generated using the us house price index dataset. as the initial step for finding the relevant information, the alignment component finds candidate datasets from a pool of pre-created datasets. the candidate selection is based on finding the datasets which are semantically similar to the indicator of the statement. e.g the indicator "mortgage rate" is more likely to be related to the table "us house price" rather than "cigarette sales". our contributions are: ## problem definition in this section, we formally define the problem of identification and alignment of quantifiably verifiable statements(qvs). let t be a textual corpus consisting of assertions @xmath0. a quantifiably verifiable assertion makes a claim about a value or set of values in a single or multiple datasets. in this paper, we focus on a subset of verifiable assertions that make a claim about a single dataset. we assume all such claims can be represented by a function f (a, d a ), that, given a claim (a) and a dataset (d a ) as input, designates the claim as true (⊤) or false (⊥). let v be the set of all claims in verifiable assertions. formally, for each ## method in this section, we describe the process of identification and alignment of qvs. given a set of documents as input, the identification method determines if they are qvs based on their structure. it also provides the trend and indicator representation ⟨t, i⟩ of the detected statements which will be used in the alignment method. the alignment method finds d a for a given quantifiably verifiable statement @xmath1, i⟩ by using the indicator structure. ideally the alignment method is provided with a list of all of the datasets (i.e. time series indicators in tables) and finds the dataset (time series) which the statement is making claim about. the dataset alignment is based on the detected i in the statement. for example in the statement "european gas demand has been rising", where "european gas demand" is identified as the indicator by detection component, the alignment selects the "europe gas consumption" as the output. we describe the method for each task in the following subsections. ## experiments we designed and conducted experiments to show the effectiveness of qvs detection and alignment.