FORTAP : Using Formulas for Numerical-Reasoning-Aware Table Pretraining (2022)

Abstract: Tables store rich numerical data, but numerical reasoning over tables is still a challenge. In this paper, we find that the spreadsheet formula, a commonly used language to perform computations on numerical values in spreadsheets, is a valuable supervision for numerical reasoning in tables. Considering large amounts of spreadsheets available on the web, we propose FORTAP, the first exploration to leverage spreadsheet formulas for table pretraining. Two novel self-supervised pretraining objectives are derived from formulas, numerical reference prediction (NRP) and numerical calculation prediction (NCP). While our proposed objectives are generic for encoders, to better capture spreadsheet table layouts and structures, FORTAP is built upon TUTA, the first transformer-based method for spreadsheet table pretraining with tree attention. FORTAP outperforms state-of-the-art methods by large margins on three representative datasets of formula prediction, question answering, and cell type classification, showing the great potential of leveraging formulas for table pretraining.

## introduction tables store rich numerical data, so a wide range of tasks require numerical reasoning over (semi-)structured tabular context, such as question answering over tables @xcite @xcite @xcite , spreadsheet formula prediction @xcite , and table structure understanding @xcite . take table#2 in figure 1 as an example, both suggesting the formula (c4-b4)/b4 for cell d4 and answering "0.61%" to the question @xmath0. % increase involves compositional calculations of subtraction and division. ## preliminaries architecture. tuta bases on bert @xcite with several enhancements: (1) a positional encoding layer based on a unified bi-dimensional coordinate tree to describe both the spatial and hierarchical information of cells; (2) a number encoding layer to encode magnitude, precision, the first digit, and the last digit; (3) a tree-based attention mechanism that enables local cells to aggregate their structurally neighbouring contexts within a tree-based distance threshold. model input/output. the input consists of a table t and optional nl texts c. by traversing the cell matrix of a table from left to right and from top to bottom, the input is linearized to "[cls], c 0 , ..., c k-1 , [sep], t (0,0) , [sep], t (0,1) , ..., @xcite , t (m -1,n -1) ", where k is the token length of nl texts, and m and n are the numbers of rows and columns of the table, respectively. note that t (i,j) refers to the token sequence of the cell string in the (i + 1) th row and (j + 1) th column, and each token has token, number, position, and format input embeddings. the output of the encoder contains token-level, cell-level, and table-level embeddings. fortap follows these input/output settings except when inputting formula token sequence. ## pretraining tasks as mentioned in section 1, empowering table modeling with stronger numerical reasoning capabilities is a fundamental need. spreadsheet formulas naturally contain information of numerical references (cell) and calculations (op/func), motivating us to devise effective tasks to leverage them for numerical-reasoning-aware pretraining. based on information parsed from the formula expression, we carefully devise two complementary objectives, numerical reference prediction (nrp) and numerical calculation prediction (ncp), to exploit the reasoning process behind referencing local cells (as operands) and applying calculations (on operands), respectively. meanwhile, to get better representations of the spreadsheet formula, which could be further used in downstream applications like formula error detection @xcite , we extend mlm @xcite from nl contexts to formulas. figure 2 gives an illustration of these tasks. ## conclusion in this paper, we present fortap , a numericalreasoning-aware table pretraining model that learns numerical reasoning capabilities from spreadsheet formulas. specifically, we design two pretraining tasks to capture numerical reasoning capabilities by explicitly predicting cell reference and calculation relations. experiments show that fortap achieves new sota on formula prediction, question answering, and cell type classification. further analyses indicate that formula pretraining indeed improves numerical reasoning skills of the model. one limitation of fortap is that we haven't fully exploit spreadsheet formulas beyond numerical reasoning. for example, logic functions like vlookup and text functions like len can be leveraged to guide complex logic and text reasoning, which will be a promising direction in the future.