ST able: Table Generation Framework for Encoder-Decoder Models (2024)

Abstract: The output structure of database-like tables, consisting of values structured in horizontal rows and vertical columns identifiable by name, can cover a wide range of NLP tasks. Following this constatation, we propose a framework for text-to-table neural models applicable to problems such as extraction of line items, joint entity and relation extraction, or knowledge base population. The permutation-based decoder of our proposal is a generalized sequential method that comprehends information from all cells in the table. The training maximizes the expected log-likelihood for a table’s content across all random permutations of the factorization order. During the content inference, we exploit the model’s ability to generate cells in any order by searching over possible orderings to maximize the model’s confidence and avoid substantial error accumulation, which other sequential models are prone to. Experiments demonstrate a high practical value of the framework, which establishes state-of-the-art results on several challenging datasets, outperforming previous solutions by up to 15\\% .

## introduction it has been previously shown that encoder-decoder models are capable of unifying a variety of problems involving natural language. in this setting, unification is achieved by casting different tasks as question answering with a plain-text answer, i.e., assuming the text-to-text @xcite @xcite or document-to-text scenario @xcite . we argue that the restriction of output type to raw text is suboptimal for the plethora of nlp problems and propose a decoder architecture able to infer aggregate data types such as a list of ordered tuples or a databaselike table (see figure 1 ). though the encoder-decoder architecture was formerly used to infer lists @xcite , named tuples @xcite , or even more complex structures @xcite , it was often achieved in an autoregressive manner, without any architectural changes. a model intended for the generation of unstructured text in natural language was used to infer an output with formal structure. in contrast, we exploit regularities and relationships within the output data and employ a grammar-constrained decoding process (section 2.5). specifically, we focus on the text-to-table inference with applications to problems such as extraction of line items, key information extraction of multiple properties, joint entity and relation extraction, or knowledge base population. tables as we understand them are equivalent to database tables and defined as a set of values structured in horizontal rows and vertical columns identifiable by name. from receipts and invoices, through paycheck stubs and insurance loss run reports, to scientific articles, real-world documents contain explicitly or implicitly tabular data to be extracted. these are not necessarily represented as a table per se within the input document, e.g., the currency name on the invoice or policy number on the loss run can be mentioned once and be related to all the line items within. in other cases, the evidence one intends to comprehend and represent as a table may be available in free-text only, as can be found in problems of joint entity and relation extraction (see figure 1 2 ). finally, the data may require some postprocessing, such as the normalization of dates, before returning them to the end-user. ## stable -text-to-table framework serialized representation of the table permits to treat it as a text sequence, and hence, use textcentric methods to perform an autoregressive generation of the output sequence by employing a vanilla transformer decoder. however, this approach does not exploit the two-dimensional structure of the table as it expands the answer sequentially and utilizes only uni-directional context. consequentially, two challenging problems arise. firstly, how to approach the fact that some information in the table may depend on other cells (e.g., name and surname or the same tax rate for similar items on a receipt) while some may not be dependent (prices of different articles on the shopping list). in general, a model possesses flexibility with respect to this dependence-independence assumption when it can leverage dependencies during decoding but is not forced to do so in any specific order. our idea (presented in figure 3 ) is to solve this problem by delaying the generation of the most challenging and complex answers to later stages and conditioning them on the already generated answer. moreover, the decoding must remain free of train-inference discrepancies. generally, the train-inference alignment means that the state of the table at every step while decoding a particular example must also be possible to achieve in the training phase. formulating the training that allows for flexible cell generation without providing any additional information remains a non-trivial problem. we rise up to the challenge and demonstrate the solution below. ## ablation studies models were trained three times with different random seeds on the rotowire, dwie, and pwc ⋆ datasets. to reduce the computational cost, we relied on base variants of the models reported in section 3 -please refer to appendix d for detailed specifications and results. while results on a single dataset can be considered noisy, aggregation over a diverse set of them helps diminish the randomness's impact and stabilize results on the new ones. ## limitations the state-of-the-art performance of stable is its foremost advantage, while the constraining factors come from different aspects. of them, the generated sequence's length seems to incure the most long-term cost during inference, while the increase in training time per example is a short-term obstacle. the underlying issue is that the full table context negatively influences the computational cost of the attention on the decoder side. this however is also the case for the family of encoder-decoder models generating the whole table such as these proposed by @xcite or @xcite . a possible solution here is a model with table context limited to the row and column a given table cell belongs to. such a change would have a positive impact on the memory consumption in the decoder, as self-attention complexity decreases from o(n m ) to o(n + m ), where n, m denotes the number of rows and columns respectively. the exploitation of this optimization is an interesting future direction. to navigate the intricacy of the order employed by the stable framework, we performed a system-atical analysis that did not conclude in finding a visible decoding pattern that could be described formally beyond the observation already provided in figure 5 and in constrained-decoding ablations. studying the generation order in the context of data calls for designing a new explainability-related method, which is not in the scope of this work. ## summary we equipped the encoder-decoder models consuming text (t5, t5 2d) and documents (tilt) with the capabilities to generate tables in a data-dependent order. firstly, an aligned training procedure based on permuting factorization order of cells was presented, and secondly, the parallelizable decoding process that fills the table with values in a flexible and unconstrained order was proposed. the important design choices for both contributions were motivated by an extensive ablation study. the proposed stable framework demonstrates its high practical value by yielding state-of-the-art results on pwc ⋆ and outperforming linearized models on cord and rotowire team datasets, as well as outperforming reference models on several confidential datasets. the highest gains due to the permutative training were accomplished on the pwc ⋆ dataset, where 4.0 points (26.8 → 30.8) amounts to 14.9% relative improvement, while the 8.8 point gain on bank statements (61.1 → 69.9) exceeds 14.4% relative improvement.