Logical Inference for Counting on Semi-structured Tables (2022)

Abstract: Recently, the Natural Language Inference (NLI) task has been studied for semi-structured tables that do not have a strict format. Although neural approaches have achieved high performance in various types of NLI, including NLI between semi-structured tables and texts, they still have difficulty in performing a numerical type of inference, such as counting. To handle a numerical type of inference, we propose a logical inference system for reasoning between semi-structured tables and texts. We use logical representations as meaning representations for tables and texts and use model checking to handle a numerical type of inference between texts and tables. To evaluate the extent to which our system can perform inference with numerical comparatives, we make an evaluation protocol that focuses on numerical understanding between semi-structured tables and texts in English. We show that our system can more robustly perform inference between tables and texts that requires numerical understanding compared with current neural approaches.

## introduction natural language inference (nli) @xcite is one of the most fundamental tasks to determine whether a premise entails a hypothesis. recently, researchers have developed benchmarks not only for texts but for other kinds of resources as well, a table being one example. previous studies have targeted database-style structured tables @xcite @xcite and semi-structured tables, such as the infoboxes in wikipedia @xcite . our focus here is on the nli task on semi-structured tables, where we handle a semi-structured table as a premise and a sentence as a hypothesis. in figure 1 , for example, we consider the semistructured table as a given premise and take joe joe biden born joseph robinette biden jr. november 20, 1942 (age 79) scranton, pennsylvania, u.s. political party democratic (1969-present) spouse(s) neilia hunter (m. 1966; died 1972) jill jacobs (m. 1977) hypothesis 1: joe biden was born in november. hypothesis 2: joe biden has had more than two wives. figure 1: a semi-structured table describing joe biden foot_0 and two hypothesis sentences. this table entails hypothesis 1 and contradicts hypothesis 2. biden was born in november as hypothesis 1. we can conclude that hypothesis 1 is entailed by the table. a semi-structured table has only two columns and describes a single object, which is indicated in the title. we call elements of the first column, such as political party, keys, each of which has an associated value in the second column such as @xcite . pairs of keys and values are called rows. it is relatively difficult to understand the information contained in infobox tables because (i) values are not limited to words or phrases, and sometimes whole sentences, and (ii) a row can contain more than one type of information, such as the birthday and birthplace in the born row. in recent years, modern neural network (nn) approaches have achieved high performance in many natural language understanding benchmarks, such as bert @xcite . nnbased approaches @xcite have also achieved high accuracy on the nli task between semi-structured tables and texts, but previous studies have questioned whether nn-based models truly understand the various linguistic phenomena @xcite @xcite @xcite . these studies have shown that nnbased approaches have failed to achieve a high performance in numerical reasoning. in this paper, we focus on a numerical type of inference on semi-structured tables, which requires understanding the number of items in a table as well as numerical comparisons. numerical comparatives are among the more challenging linguistic phenomena that involve generalized quantifiers. for example, the phrase more than in hypothesis 2 in figure 1 is a numerical comparative and compares two and the number of wives. for dealing with numerical comparatives, @xcite achieved high performance by developing a logical inference system based on formal semantics. however, @xcite concentrated on the inference between texts only, and inference systems that reliably perform inference between tables and texts involving numerical comparatives have not yet been developed. thus, we aim to develop a logical inference system between semi-structured tables and texts, especially for numerical reasoning. while previous work @xcite @xcite has provided semantic parsers of constructing query languages such as sqls for question answering on databasestyle tables, we present logical representations for semi-structured tables to enable a numerical type of inference on semi-structured tables. furthermore, the existing nli dataset for semi-structured tables @xcite does not contain sufficient test cases for understanding numerical comparatives. thus, there is a need for an evaluation protocol that investigates the numerical reasoning skills of nli systems for semi-structured tables. given this background, our main contributions in this paper are the following: our system and dataset will be publicly available at https://github.com/ynklab/sst_ count . ## related work and background this section explains the related work of logicbased nli approaches and the background of model checking, which is used for inference between semi-structured tables and sentences in our proposed system. ## experiments we evaluate the extent to which our system can perform inference with numerical comparatives. we make an evaluation protocol that focuses on the numerical understanding between semistructured tables and texts in english. ## conclusion in this study, we proposed a logic-based system for an nli task that requires numerical understanding in semi-structured tables. we built an nli dataset that focuses on numerical comparatives between semi-structured tables and texts. using this dataset, we showed that our system performed more robustly than the previous nn-based model. in future work, we will improve knowledge injection process to cover various problems. we also seek to handle other generalized quantifiers such as many. we believe that our system and dataset for performing numerical inference between semistructured tables and texts could pave the way for applications of inference between resources other than texts.