D i SC o M a T : Distantly Supervised Composition Extraction from Tables in Materials Science Articles (2023)

Abstract: A crucial component in the curation of KB for a scientific domain (e.g., materials science, food & nutrition, fuels) is information extraction from tables in the domain’s published research articles. To facilitate research in this direction, we define a novel NLP task of extracting compositions of materials (e.g., glasses) from tables in materials science papers. The task involves solving several challenges in concert, such as tables that mention compositions have highly varying structures; text in captions and full paper needs to be incorporated along with data in tables; and regular languages for numbers, chemical compounds, and composition expressions must be integrated into the model. We release a training dataset comprising 4,408 distantly supervised tables, along with 1,475 manually annotated dev and test tables. We also present DiSCoMaT, a strong baseline that combines multiple graph neural networks with several task-specific regular expressions, features, and constraints. We show that DiSCoMaT outperforms recent table processing architectures by significant margins. We release our code and data for further research on this challenging IE task from scientific tables.

## introduction advanced knowledge of a science or engineering domain is typically found in domain-specific research papers. information extraction (ie) from scientific articles develops ml methods to automatically extract this knowledge for curating largescale domain-specific kbs (e.g., @xcite ). these kbs have a variety of uses: they lead to ease of information access by domain researchers @xcite , provide data for developing domainspecific ml models @xcite , and potentially help in accelerating scientific discoveries @xcite . significant research exists on ie from text of research papers (see @xcite for a survey), but less attention is given to ie (often, numeric) from tables. tables may report the performance of algorithms on a dataset, quantitative results of clinical trials, or other important information. of special interest to us are tables that mention the composition and properties of an entity. such tables are ubiquitous in various fields such as food and nutrition (tables of food items with nutritional values, see tables 1 2 3 4 in @xcite and in materials science (matsci) articles, the details on synthesis and characterization are reported in the text @xcite , while material compositions are mostly reported in tables @xcite . a preliminary analysis of matsci papers reveals that ∼85% foot_0 of material compositions and their associated properties (e.g., density, stiffness) are reported in tables and not text. thus, ie from tables is essential for a comprehensive understanding of a given paper, and for increasing the coverage of resulting kbs. to this extent, we define a novel nlp task of extraction of materials (via ids mentioned in the paper), constituents, and their relative percentages. for instance, fig. 1a should output four materials a1-a4, where id a1 is associated with three constituents (moo 3 , fe 2 o 3 , and p 2 o 5 ) and their respective percentages, 5, 38, and 57. a model for this task necessitates solving several challenges, which are discussed in detail in sec. 3. while many of these issues have been investigated separately, e.g., numerical ie @xcite , unit extraction @xcite , chemical compound identification @xcite , nlp for tables @xcite , solving all these in concert creates a challenging testbed for the nlp community. here, we harvest a distantly supervised training dataset of 4,408 tables and 38,799 compositionconstituent tuples by aligning a matsci database with tables in papers. we also label 1,475 tables manually for dev and test sets. we build a baseline system discomat, which uses a pipeline of a domain-specific language model @xcite , and two graph neural networks (gnns), along with several hand-coded features and constraints. we evaluate our system on accuracy metrics for various subtasks, including material id prediction, tuple-level predictions, and material-level complete predictions. we find that discomat's gnn architecture obtains a 7-15 points increase in accuracy numbers, compared to table processors @xcite , which linearize the table for ie. subsequent analysis reveals common sources of discomat errors, which will inform future research. we release all our data and code 2 for further research on this challenging task. ## related work recent works have developed neural models for various nlp tasks based on tabular data, viz, tabular natural language inference @xcite , qa over one or a corpus of tables @xcite @xcite @xcite , table orientation classification @xcite , and relation extraction from tables @xcite . several recent papers study qa models-they all linearize a table and pass it to a pre-trained language model. for example, tapas @xcite does this for wikipedia tables to answer natural language questions by selecting table cells and aggregation operators. tabert @xcite and @xcite also use similar ideas alongside some architectural modifications to handle rows and columns better. tabbie @xcite consists of two transformers that encode rows and columns independently, whereas tapex uses encoder-decoder architecture using 2 https://github.com/m3rg-iitd/discomat bart. tabbie and tapex also introduce pretraining over tables to learn table representations better. similar to our work, tables have also been modeled as graphs for sequential question answering over tables @xcite . however, all these works generally assume a fixed and known structure of tables with the same orientation, with the top row being the header row in all cases -an assumption violated in our setting. orientation and semantic structure classification: deeptable @xcite ) is a permutation-invariant neural model, which classifies tables into three orientations, while tabnet @xcite uses rnns and cnns in a hybrid fashion to classify web tables into five different types of orientations. infotabs @xcite ) studies natural language inference on tabular data via linearization and language models, which has been extended to the multilingual setting @xcite , and has been combined with knowledge graphs @xcite . some earlier works also focused on annotating column types, entity id cells, and pair of columns with binary relations, based on rule-based and other ml approaches, given a catalog @xcite . ## challenges in composition extraction from tables we analyze numerous composition tables in matsci research papers (see figures 1 , 6 and 4 for examples), and find that the task has several facets, with many table styles for similar compositions. we now describe the key challenges involved in the task of composition extraction from tables. table shown in figure 4a is a row oriented table-different compositions are written in different rows. the table in figure 1a is a column-oriented table. a common case is when a dopant is used; its amount is reported in excess. ## problem formulation our goal is automated extraction of material compositions from tables. formally, given a table t , its caption, and the complete text of publication in which t occurs, we aim to extract compositions expressed in t , in the form {(id, c id k , p id k , u id k )} k id @xmath0. here, id represents the material id, as used in the paper. material ids are defined by matsci researchers to succinctly refer to that composition in text and other tables. c id k is a constituent element or compound present in the material, k id is the total number of constituents in the material, p id @xmath1, and u id k is the unit of p id k (either mole% or weight%). for instance, the desired output tuples corresponding to id a1 from figure 1a are (a1, moo 3 , 5, mol%), (a1, fe 2 o 3 , 38, mol%), (a1, p 2 o 5 , 57, mol%). ## dataset construction we match a matsci db of materials and compositions with tables from published papers, to automatically provide distantly-supervised labels for extraction. we first use a commercial db (ngf, 2019) of glass compositions with the respective references. then, we extract all tables from the 2,536 references in the db using text-mining api (els). we use a table parser @xcite for raw xml tables and captions. this results in 5,883 tables of which 2,355 express compositions with 16,729 materials, and 58,481 (material id, constituent, composition percentage, unit) tuples. we keep tables from 1,880 papers for training, and the rest are split into dev and test (see the db does not contain information about the location of a given composition in the paper -in text, images, graphs, or tables. if present in a table, it can appear in any column or row. since we do not know the exact location of a composition, we use distantly supervised train set construction @xcite . first, we simply match the chemical compounds and percentages (or equivalent fractions) mentioned in the db with the text in a table from the associated paper. if all composition percentages are found in multiple cells of the table, it is marked as mcc-ci (multi-cell composition with complete information). however, due to several problems (see appendix 3), it misses many composition tables. to increase the coverage, we additionally use a rule-based composition parser (described below), but restricted to only those compounds (cpd non-terminal in figure 2 ) that appear in the db for this paper. our distant supervision approach obtains tablelevel annotation (nc, scc, mcc-pi, mcc-ci), where a table is labeled as non-composition, single/multi cell composition with partial/complete information. it also obtains annotation for each row or column into four labels: id, composition, constituent, and other. while training data is created using distant supervision, dev and test sets are hand annotated. we now explain the dataset construction process in further detail. rule-based composition parser: the parser helps find names of constituents from mcc tables, and also match full compositions mentioned in scc tables. recall that in scc tables, the full composition expression is written in a single cell in the row/column corresponding to each material id. such compositions are close to regular languages and can be parsed via regular expressions. figure 2 shows the regular expression (simplified, for understandability) used by the parser. here cmp denotes the matched composition, pats are the three main patterns for it, csts are sub-patterns, cpd is a compound, num is a number, and ob and cb are, respectively, open and closed parentheses (or square brackets). w is zero or more whitespace characters, and sep contains explicit separators like '-' or '+'. start and end are indicators to separate a regular expression from the rest of the text. the first pattern parses simple numbercompound expressions like 40bi 2 o 3 * 60b 2 o 3 . here each of the two constituents will match with cst 1 . the other two patterns handle nested compositions, where simple expressions are mixed in a given ratio. the main difference between the second and third patterns is in the placement of outer ratios -after or before the simple composition, respectively. example match for pat 2 is (40bi 2 o 3 +60b 2 o 3 )30 -(agi+agcl)70, and for pat 3 is 40bi 2 o 3 ,40b 2 o 3 ,20(agi:2agcl). to materialize the rules of the rule-based composition parser, we pre-label compounds. for our dataset, we use a list-based extractor, though other chemical data extractors @xcite may also be used. after parsing, all coefficients are normalized so that they sum to hundred. for nested expressions, the outer ratio and the inner ones are normalized separately and then multiplied. the compositions parsed by rule-based composition parser are then matched with entries in the db. a successful matching leads to a high-quality anno-tation of composition expressions in these papers. if this matching happens: (i) in a single cell, the table is deemed as scc, (ii) on caption/paper text that has an algebraic variable (or compound) found in the table, it is marked as mcc-pi (see figure 1(b) ). in case of no matching, the table is marked as nc. this automatic annotation is post-processed into row, column and edge labels. one further challenge is that material ids mentioned in papers are not provided in the db. so, we manually annotate material ids for all the identified composition tables in the training set. this leads to a train set of 11,207 materials with 38,799 tuples from 4,408 tables. since the train set is distantly supervised and can be noisy, two authors (one of them is a matsci expert) of this paper manually annotated the dev and test tables with row/column/edge labels, units, tuples, compositions, and table type, resulting in over 2,500 materials and over 9,500 tuples per set. we used cohen's kappa measure for identifying inter-annotator agreement, which was 86.76% for glass id, 98.47% for row and column labels, and 94.34% for table types. conflicts were resolved through mutual discussions. further statistics and the description of the developed in-house annotation tools used for manual annotations are discussed in a.2. ## discomat architecture figure 3 : the design of discomat figure 3 illustrates the basic pipeline for extraction in discomat. we find that the simplest task is to identify whether the table t is an scc table, owing to the distinctive presence of multiple numbers, and compounds in single cells. disco-mat first runs a gnn-based scc predictor, which classifies t as an scc table or not. for the scc table, it uses the rule-based composition parser (described in sec. 5). for the other category, dis-comat runs a second gnn (gnn 2 ), and labels rows and columns of t as compositions, material ids, constituents, and others. if no constituents or composition predictions are found, then t is deemed to be a non-composition (nc) table. else, it is an mcc table, for which discomat predicts whether it has all information in t or some information is missing (partial-information predictor). if it is a complete information table, then gnn 2 's predictions are post-processed into compositions. if not, the caption and text of the paper are also processed, along with gnn 2 's predictions leading to final composition extraction. we note that our system ignores statistically infrequent corner cases, such as single-cell partial information tables -we discuss this further in our error analysis. we now describe each of these components, one by one. ## experiments baseline models: we implement discomat with lm as matscibert @xcite , and the gnns as graph attention networks @xcite . we compare disco-mat with six non-gnn baseline models. our first baseline is tapas @xcite , a stateof-the-art table qa system, which flattens the table, adds row and column index embeddings, and passes as input to a language model. to use tapas for our task, we use table caption as a proxy for the input question. all the model parameters in this setting are initialized randomly. next, we use tabert @xcite , which is a pretrained lm that jointly learns representations for natural (nl) sentences and tables by using pretraining objectives of masked column prediction (mcp) and cell value recovery (cvr). it finds table cell embeddings by passing row linearizations concatenated with the nl sentence into a language model and then applying vertical attention across columns for information propagation. finally, we use tabbie, which is pretrained by corrupt cell detection and learns exclusively from tabular data without any associated text, unlike the previous baselines. additionally, we replace the lm of all models with matscibert to provide domain-specific embeddings to obtain the respective adapted versions. we also implement a simple rule-based baseline for mcc-ci and nc tables. the baseline identifies constituent names using regex matching and a pre-defined list of compounds, extracts numbers from cells and finds the units using simple heuristics to generate the required tuples. further details on baselines is provided in app. a.3. evaluation metrics: we compute several metrics in our evaluation. (1) table-type (tt) prediction accuracy -it computes table-level accuracy on the 4-way table classification as nc, scc, mcc-ci and mcc-pi. (2) id f 1 score computes f 1 score for material id extraction. (3) tuple-level (tl) f 1 score evaluates performance on the extraction of composition tuples. a gold is considered matching with a predicted 4-tuple if all arguments match exactly. (4) material-level (matl) f 1 score is the strongest metric. it evaluates whether all predicted information related to a material (including its id, all constituents and their percentages) match exactly with the gold. finally, (5) constraint violations (cv) counts the number of violations of hard constraints in the prediction. we consider all four types of constraints, as discussed in section 6.4. implementation details are mentioned in app. a.4.