Automatic Verification of Data Summaries (2021)

Abstract: We present a generic method to compute thefactual accuracy of a generated data summarywith minimal user effort. We look at the prob-lem as a fact-checking task to verify the nu-merical claims in the text. The verification al-gorithm assumes that the data used to generatethe text is available. In this paper, we describehow the proposed solution has been used toidentify incorrect claims about basketball tex-tual summaries in the context of the AccuracyShared Task at INLG 2021.

## introduction natural language generation (nlg) can be used to generate texts out of relational data, where the goal is to have correct and clear statements that describe what can be found in tables @xcite . however, this is not always the case, since nlg tools, although producing correct sentences grammar-wise, sometimes fail to generate accurate texts, eventually containing some factual errors @xcite . the goal of our work is to evaluate generated texts by identifying numerical claims and factchecking them with the relational data available at hand. we apply different techniques on the provided summaries and use the available relational data about the matches to state whether the claims are true or false. the idea behind fact checking using relational tables is to create an automated verification pipeline using data-driven algorithms, such as deep learning models @xcite . building upon previous work in fact-checking statistical claims (karagiannis et al., 2020), we focus on such kind of claims due to the availability of trustworthy relational tables to verify them. the goal is not only to be accurate, but also to limit the user effort, such as labelling data or writing scripts, in the setup of the system. team pts ast .. tov kings 99 22 .. 21 nets 107 20 .. 9 (a) sample of team statistics in a match. team pts ast .. fga ben mclemore 11 2 .. 9 sergey karasev 5 3 .. 5 .. .. .. .. .. kevin garnett 10 2 .. 10 (b) sample of player statistics in a match. statistical claims form a significant part of the set of claims in the shared task (around 40%) as the tables mostly contain numerical facts, rather than textual. consider samples of the relational tables of players and teams in tables 2a and 2b , respectively. sentence "sacramento kings scored 99 points." is verified by identifying the team name (key value "kings") and column (label "pts"). a comparison between the value in the identified cell and the value in the text (99) validates the claim. more complex claims, such as "kings defeated nets", require comparisons between two cells. to verify the claims above, our solution consists of three main steps: (i) claim identification, (ii) identification of properties that construct a val-idation query over the data for every claim, and (iii) claim verification. in the following, we first describe our solution and then report some experimental results and possible directions for future work. ## method we target claims that can be verified computationally by using operations over the cell values in the tables (relations). we follow an approach inspired from previous work on computational factchecking for statistical claims @xcite . we begin the process by extracting claims from the input sentences (section 2.1) and then identify query properties (section 2.2), which are used for building the query that verifies the claim (section 2.3). figure 3 represents the architecture of the solution, where we input sentences and get a collection of properties, including the claims to verify and the elements of the data that are needed for this task. every sentence can contain more than one named entity (were we focus only on players/teams) as well as several claims. we have to associate each claim and the resulting properties to the entity in question. by looking at the summaries in the task, it is possible to observe that many sentences fall into two categories: for our example, 107 is assigned to kings and 99 to nets. as for the "defeated" claim, we associate it to both kings and nets. after claim and properties are identified, we use the latter to create a query that fact checks the claim. the query returns a boolean value in the final output and it is self-explanatory, i.e., it is a declarative specification easy to interpret as an explanation of the checking process. an example of a query that returns the number of points (pts) of the team sacramento kings from an identified table t is select t.pts from t where t.@xmath0; whose output is compared against the value of the extracted claim. ## results the evaluation of our solution over the test data shows that we obtain a precision of 0.329 and a recall of 0.205. while we expected a limited recall, as we focus on a specific subset of claims among all the possible ones in the summaries, we investigated the possible causes for the low performance and found three main explanations. missing support for coreference resolution. in some sentences, names are not explicitly mentioned and the concerned entity is referred to as "he" or "the visiting team". as we did not implemented a specific solution for coreference resolution, this is a weak point in our system. for example, consider the sentence "it was his second double-double in a row, as he's combined for 54 points and 13 rebounds over his last two games.". the sentence is not checked by the system since no names were singled out, leading to the system missing 3 claims. claims requiring complex retrieval. some statistical values are hard to fact check, such as the number of wins in the season. consider the sentence "over his last three games, he's combined for 34 points, 13 rebounds and five assists, while playing just 21 minutes per game.'. the verification of this claims requires to identify the last 3 games played by the player, and to sum up their scores. as another example, "greg monroe was the only other laker [...]". the only way to know if he was the only other laker to achieve something is to look at all the lakers players scoring. these kinds of processes go beyond the ability of our data retrieval modules. limits in the identification of the claims. some of the wording used in the text turned out to be hard to process both with the regex function and the bert model. for example, all claims containing an expression like "double-double" or "6-for-13". ## future work we presented a solution for verifying statistical data claims in data summaries with limited human supervision. we believe our work shows some of the opportunities and challenges for the problem of verifying data summaries with computational methods @xcite . while there are some promising results for specific cases, the road to accurate and general solutions is still long. the main challenge lies in the limited amount of training data and the large variety of claim kinds, as we discuss next. the claims in figure 1 vary from those expressed using adjectives (e.g., strong) to others pivoting on verbs (e.g., leading). fact-checking these claims requires to go beyond a lookup in a table, but rather, we seem to suggest the need for domain-specific rules or models, depending on the nature of the claim. for instance, for the sentence "the grizzlies had a strong first half", if we interpret that the claim "strong" relates to the points scored in the first half, it means that we are interested in the sum of the first two quarters, team-pts qtr1 + team-pts qtr2, and we have to set a threshold to the number of points starting from which the word "strong" applies. but this rule would apply only to "strong" (and similar adjectives) used for points and the threshold would change for "fouls". in other words, we have to handle all the possible qualitative adjectives that can appear in the text for all attributes, which is challenging to do exhaustively and accurately. it is easy to see that this problem is challenging as it would either require a lot of manually defined rules or the annotations of a large number of claims to train models that handle all cases.