Statements: Universal Information Extraction from Tables with Large Language Models for ESG KPI s (2024)

Abstract: Environment, Social, and Governance (ESG) KPIs assess an organization’s performance on issues such as climate change, greenhouse gas emissions, water consumption, waste management, human rights, diversity, and policies. ESG reports convey this valuable quantitative information through tables. Unfortunately, extracting this information is difficult due to high variability in the table structure as well as content. We propose Statements, a novel domain agnostic data structure for extracting quantitative facts and related information. We propose translating tables to statements as a new supervised deep-learning universal information extraction task. We introduce SemTabNet - a dataset of over 100K annotated tables. Investigating a family of T5-based Statement Extraction Models, our best model generates statements which are 82% similar to the ground-truth (compared to baseline of 21%). We demonstrate the advantages of statements by applying our model to over 2700 tables from ESG reports. The homogeneous nature of statements permits exploratory data analysis on expansive information found in large collections of ESG reports.

## introduction it is invaluable to assess mankind's impact on climate. climate change related information is often published in so-called "environment, social, and governance (esg)" reports. corporations report valuable quantitative data regarding their efforts to improve their impact on environment, working conditions, and company culture in these esg reports @xcite . like most technical documents, esg reports present their key information in tables, making table understanding and information extraction (ie) an important problem @xcite . this problem becomes further complicated due to the large variety and diversity of tabular representations used in these reports. despite efforts to standardize these reports, this diversity makes the task of extracting information from these documents extremely challenging (see appendix fig. 5 for an example table ). large language models (llms) have turned out to be excellent tools for ie, due to their ability to parse, understand, and reason over textual data @xcite . this, in combination with their in-context learning ability, makes them excellent for ie from text @xcite . this approach breaks down when applying the same techniques on tables @xcite . in this paper, we present a general approach for universal ie from tables. universal ie involves named entity recognition and relationship extraction among other tasks. to this end, we propose a new tree-like data structure, called 'statement', which can combine multiple (named) entities and (n-ary) relations (fig. 1 ). it allows us to represent information in a homogeneous domain agnostic fashion. a statement tree can contain content from different subjects, allowing for universal ie approach to tables across multiple domains. with the introduction of statements, the ie problem from tables becomes a translation problem which we call 'statement extraction' -translating the original table into a set of statements. esg reports, to this day, are manually analyzed by consultancy firms and professional organisations @xcite . with our proposed statement extraction, this process can now be fully automated. to evaluate our model generated statements, we propose a novel application of the well-established tree edit distance @xcite . we propose tree similarity score (t s ) for measuring the similarity between two trees. as baseline, we experiment with in-context learning using stateof-the-art llms like mistral @xcite , mixtral @xcite , llama2 @xcite , and falcon @xcite . these models show an average t s varying from 0% to 21%. on the other hand, our best-performing finetuned t5 based model shows a t s of 82%.our main contributions are: statement for mapping complex, irregular, and heterogeneous information to a uniform domain agnositc structure. the fine-tuned models show significant im-provement over baseline experiments providing competitive benchmarks for the community. we begin, in sect. 2 discussing related works. in sect. 3 we explain the concept of 'statements' and present the semtabnet dataset in sect. 4. in sect. 5, we discuss the various experiments we performed and their results. we end the paper with an application of our model on esg reports. another class of problem which is similar to the data transformation approach is (generative) information extraction (ie) which involves adopting llms to generate structural information from an information source. recent studies have found that llms can also perform universal ie @xcite @xcite . in a universal ie task, a model is trained to generate desirable structured information y, given a pre-defined schema s, and information source x @xcite . using pre-trained language models, @xcite perform ie in two steps: argument extraction and predicate extraction. based on this, they introduced a text-based open ie benchmark. wang et al. (2021) presented deepex for extracting structured triplets from text based data. wang et al. (2022a) demonstrate that pre-training models on task-agnostic corpus lead to performance improvement on tasks like ie, entity recognition, etc. however, these approaches are limited to textual data. bai et al. (2024) have shown that llms can perform ie on tabular data when prompted with a table and a relevant extraction schema. their approach is based on a human-in-the-loop in-context learning. a domain-expert is necessary for producing robust extraction schema, which instructs the model to generate structured records from a table . this strongly limits the adaptability of their approach to different domains. although limited to text, @xcite also propose a schema-driven universal ie system. they use a structure extraction language which generates structural schema prompt which guides the model in its ie tasks. as we show, the statements data structure removes several limitations of previous universal ie approaches and is applicable to 'wild' heterogenous information sources. ## definition of statements the statements data structure aims to homogenize data coming from complex, irregular, heterogeneous information source (text or tables). at its core, the statements data structure is a tree structure (fig. 1 ). from the root of the tree, we have 'subject'-nodes, which contain information regarding the 'subject' and the 'subject-value' keys. from each subject-node, there are one or more predicate nodes, which define the 'property', 'propertyvalue', and 'unit' keys. each predicate node carries an atomic piece of quantitative information. the statement knowledge model can be applied to both text and tables. in fig. 2 , we show the same statements structure which could be obtained from a text or a corresponding table. as such, the statements structure is not bound only to tables, however, it shows its usefulness particularly when normalising information from heterogeneous tables. the details of how we create trees are presented with examples in appendix c. the tree structure of statements allows us to quantify, with a single number, the transformation of information from a table. this is accomplished by computing the tree similarity score (based on the tree editing distance (ted) pawlik and augsten (2016); @xcite ) between predicted and ground-truth statements. ted is defined as the minimum-cost sequence of node operations that transform one tree into another. like the levenshtein distances on strings @xcite , ted involves three kinds of operations: node insertions, deletions, and renaming. the cost of each operation can be freely defined, which makes this metric both flexible and powerful. two trees are exactly same when their tree similarity score is 100%. to ensure high quality statement extraction, we setup robust ted costs such that minor differences can lead to poor tree similarity scores. in appendix c.2, we demonstrate tree similarity score with some examples. it is also instructive to look at the edit types which converted the predicted statements into ground-truth statements. for this, we measure the ratio of edit type to the total number of edits. we find that the ratio of insertions and ratio of deletions carries the information about the structural similarity of two trees. if the model predicted too few nodes, the ratio of insertions will be high. correspondingly, if the statements from the model's prediction has too many nodes, the deletion ratio dominates. if two trees are structurally similar, then the ratio of both insertion and deletions is low. in this case, the edits are dominated by renaming. while tree-based metrics are sensitive to both entity and relationship extraction, we also would like to understand the ability of a statement extrac-tion model (sem) to extract entities alone foot_0 . for this, we concatenate all the predicate nodes in a statement. we create sets of values corresponding to: subject, subject value, property, property value, unit. we count true positives when an entity is found in both the sets from model prediction and ground truth. true negatives are counted when an entity is present only in the ground truth set and false positives when the entity is present only in the predicted set. based on these, we measure the standard accuracy, recall and f1 measures. ## semtabnet: statements data there are many large data sets of annotated tables which suffer from two major limitations: (1) they focus on understanding table structure only i.e. demarcating table headers from table content, and (2) contain little diversity in shape, size, and complexity of the table. tables found in esg reports are of high complexity with little common underlying pattern. in this work, we advance deep learning on table understanding by annotating the content of the table and annotating complex tables. we used the deep search toolkit foot_1 to collect over 10k esg reports from over 2000 corporations. deep search crawled these pdf reports, converted them into machine readable format, and provided this data along with the metadata of each report in json format. we compiled a list of important keywords which capture many important concepts in esg reports (see appendix a). next, we select only those tables which have some relevance with the keywords. for this we used the following conditions: the rouge-l precision (longest common sub-sequence) score between raw data and keywords must be greater than 0.75 and there must be quantitative information in the table. we need a strategy for understanding the content of a table and extracting statements from it. after manually observing hundreds of table, we decided a two step approach to prepare our groundtruth data. first, we classify all the cells in a table based on the semantic meaning of their content into 16 categories which helps us in constructing statements. for each table, this step creates a 'labels-table' with the same shape and structure as the original, but the cells of this labels-table only contain category labels (see fig. 3 ). secondly, we create a program which reads both the labels- additionally, we observed that most tables can be reasonably classified into three baskets: simple, complex, and qualitative. there are simple tables whose structure cannot be further subdivided into any smaller table. there are complex tables whose structure can be further divided into multiple smaller tables. finally, there are qualitative tables (like table of contents) which contain little valuable information for our endeavour. we collected about 2,800 tables and found ∼ 20% had simple layout, ∼ 20% had complex layout (composed of multiple simpler tables arranged hierarchically), and ∼ 60% were qualitative. we discarded all qualitative tables from any further analysis. to ensure that our data is not biased towards either simple or complex tables, we manually annotated all the cells of 569 simple tables and 538 complex tables. in total, we annotated 1,107 tables (84,890 individual cells) giving rise to 42,982 statements. due to the nature of our strategy, one can extract statements from tables either directly in a zero shot manner (direct se) or by predicting cell labels and then using the rule-based approach to construct statements (indirect se) (see fig. 3 . we have experimented with both approaches. we further augmented the annotated tables to create a large training data. we shuffle the rows and columns of tables corresponding to property-values to create new augmented tables, while keeping their contents the same. while this is straightforward for simple tables, special care was taken for complex tables such that only rows/columns which belonged together within a category were shuffled. the maximum number of augmented tables emerging from the shuffling operations was limited to 130, leading to over 120k tables. to promote further research and development, we open source this large dataset of semantic cell annotations as semtabnet foot_2 . since the nature of these tasks naturally fits the paradigm of sequence-to-sequence models, we finetune t5 models @xcite . t5 models are encoder-decoder transformer architecture models which are suitable for many sequence-to-sequence tasks. in our experiments, we train t5 variants (small, base, large, and 3b) to create a family of statement extraction models (sem). in our training data for tables, the input token count is less than 512 for 50% of the data, and it is less than 1024 for 90% of the data. thus, except where mentioned, we train t5 models (small, base, large) with context windows of 512 and 1024, and t5-3b with context window of 512. all models are fine-tuned in a distributed data parallel (ddp) manner simultaneously across 4 gpu devices (nvidia a100-40gb for t5-small, t5-base, t5-large and nvidia a100-80gb for t5-3b). additionally, the largest possible batch size was used for all models. the batch size is impacted by factors like model size, gpu memory, and context window. in turn it affects the number of epochs we can fine-tune in a reasonable time. for all tasks, we stop the fine-tuning process either after 500,000 steps or after 7 days. we use the adamw optimizer with β 1 = 0.9 and β 2 = 0.999. all models are trained with a maximum learning rate of 5 × 10 -4 . there is a warm-up phase of 1000 steps in which the learning rate increases linearly from 10 -10 to 5 × 10 -4 . after another 1000 steps, the learning rate is exponentially decayed until it reaches its lowest value of 10 -6 , where it remains until the end of the training. baseline experiments: for baseline experiments, several state of the art llms were tested for their in-context learning ability. in the prompt, we show the model an example of direct statement extraction (1-shot), followed by a test table. the models produce statements in markdown format, which are evaluated against ground truth statements. the average tree similarity score across 1100 annotated tables varies from 0% for fal-con40b to 20% for mixtral (8×7b models). for entity extraction, llama2-13b performed the best with an average f1 score of 38. not all outputs generated by the model were in correct markdown format. minor changes in the prompt were found to create vast differences in the quality of extracted statements. in appendix d, we show examples of the prompt and the model output for some cases. statement extraction indirect 1d: all models trained on this task have context window of 512. their performance tends to scale with model size. these models can learn to extract entities, but relationship extraction is difficult. for sem-t5small, the ratio of insertion is ≈ 98% which means that the predicted statements does not have enough nodes. statement extraction indirect 2d: all models trained on this task perform well on entity extraction with average f1 scores of over 95%. the highest performing model is the sem-t5-3b (512) with an average tree similarity score of 81.76%. statement extraction direct:based on tree similarity score, most models show poor performance in direct se. the best performing model is sem- the comparison between the ground truth and the model-predicted statements is encapsulated by the tree similarity score (t s ). t s measures if two trees are similar (100% being an exact match). for each statement, the precision, recall and f1 score (reported) of entity extraction extraction was also measured. for all reported values, the 99% confidence interval, assuming a gaussian distribution, is ∼ 0.1%. the standard error of the mean in all cases is below 0.005%. ## application to esg results due to their homogeneous structure, statements enable large-scale exploratory data analysis and data science. to demonstrate the advantage of statements over traditional tabular data science, we applied sem-t5-large (512 se indirect 2d) over 2700 tables published in over 1000 esg reports in 2022. this lead to 14,766 statements containing over 100k predicates. this dataset containing esg related kpis is invaluable to researchers, policymakers, and analysts. we filter this large dataset to contain only those predicates with quantitative property values. this subset contains 47 901 predicates from 601 corporate esg reports. we search the properties in this dataset for some keywords representative of esg kpis. fig. 4 (top) shows the distribution of the number of predicates and the number of distinct organizations which matched our simple keyword search. for example, using 'emission' as a keyword, we obtain over 4000 hits with results coming from over 300 distinct corporations. fig. 4 (bottoms) shows the total scope 1 emissions (left) and total scope 2 emission (right). each box shows the distribution of emission from multiple corporations across sectors (∼ 20 in healthcare to ∼ 100 in technology and industrial goods) containing data from several years. the data reported in the original report contained emissions in different units, which were harmonized for creating this plot. since we only took a small subset of 1000 reports for this analysis, our data is incomplete and is only representative. the statements dataset al-lows one to study how emissions from individual companies or across sectors have evolved over time. this dataset can also serve as a starting point for many other downstream applications like questionanswering, fact checking, table retrieval, etc. ## conclusion & future works we have presented a novel approach to map complex, irregular, and heterogeneous information to a uniform structure, statements. we presented statement extraction which is a new supervised deep-learning information extraction task. we contribute the field of table understanding by opensourcing semtabnet consisting of 100k esg tables wherein all cells. investigating three variations of the statement extraction task, we found that using a model to generate table annotations and then construct statements produces best results. this approach has the advantage, that it produces homogeneous structured data with reduced hallucinations. statements are an advantageous vehicle for quantitative factual information. they enable down-stream tasks like data science over a large collection of documents. we extracted over 100k facts (predicates) from only 1000 esg reports. this work can be easily extended to include domains other than esg. it can also be extended towards multi-modality by including text data. we leave for future exploration, the use of statements in downstream tasks like qa or document summarization.