Named Entity Recognition in Industrial Tables using Tabular Language Models (2022)

Abstract: Specialized transformer-based models for encoding tabular data have gained interest in academia. Although tabular data is omnipresent in industry, applications of table transformers are still missing. In this paper, we study how these models can be applied to an industrial Named Entity Recognition (NER) problem where the entities are mentioned in tabular-structured spreadsheets. The highly technical nature of spreadsheets as well as the lack of labeled data present major challenges for fine-tuning transformer-based models. Therefore, we develop a dedicated table data augmentation strategy based on available domain-specific knowledge graphs. We show that this boosts performance in our low-resource scenario considerably. Further, we investigate the benefits of tabular structure as inductive bias compared to tables as linearized sequences. Our experiments confirm that a table transformer outperforms other baselines and that its tabular inductive bias is vital for convergence of transformer-based models.

## introduction there has been growing interest in developing special model designs intended to capture tabular structure @xcite @xcite . a recent survey named these models tabular language models (talms) and provided an overview of the different architectures and pretraining objectives @xcite . one of the downstream tasks where talms are applicable is table interpretation (ti) with its sub-tasks: entity linking, column type annotation and relation extraction @xcite . most talms for ti use bert as the backbone language model (lm) for encoding the content of table cells and aggregate their representations on although tabular data is omnipresent in industry, talms such as table transformers, have not found their way into industrial applications yet. one reason being the nature of data stored in industrial tables which is different and more dynamic than data in academic datasets where the schema of the table is consistent and each cell contains a single entity of one type @xcite . as shown in figure 1 , industrial tables contain multiple subcell entities from different types, hence the talms which provide cell-level aggregation are not sufficient. in this direction, we formulate the problem of sub-cell named entity recognition (ner) in tables using talms. another challenge is that tabular data in industry is often lacking labels, especially labels reflecting the high variance across examples. due to the very technical and domain-specific nature only experts can effectively provide such labels, which is -for most tasks -too expensive. these lowresource scenarios are challenging for statistical nlp models and usually prohibit fine-tuning of large-scale transformer-based models. a popular strategy to remedy low-resource scenarios is data augmentation (da) @xcite , which allows to increase data diversity without having to collect new examples. common da techniques in nlp range from using external knowledge such as wordnet @xcite , machine-translation models for back-translation @xcite or mixing of examples inspired from computer vision @xcite ). an empirical study by @xcite showed that applying off-the-shelf da techniques @xcite for fine-tuning of lm like bert or roberta bring little to no improvement and become even less beneficial in crossdomain settings @xcite . these studies emphasize the challenge of developing domain-specific da techniques which would help improve the existing pretrained transformer models. although, there are no domain-specific da techniques applicable to a tabular dataset, in many industrial domains there exist external resources which can be exploited for creating augmented tables. in this paper we study a da technique for industrial spreadsheet tables leveraging publicly available resource based on an industrial standard. specifically, the contributions of this paper are the following: ## industry ner use case as motivation for tabular ner in an industrial context, we describe a real-world dataset from which information about industrial plant equipment, such as actuators, sensors, vessels, etc. and their physical quantities should be extracted. this information is typically collected and maintained by engineers in spreadsheets. the spreadsheets are roughly organized in a tabular format, as can be seen from the example table in figure 1 . the engineers do neither comply to a fixed schema nor to unified spelling of equipment or properties. the goal is to automatically extract relevant entities for creating a structured specifications of the plant equipment. we phrase this problem as ner task with the following types of entities. the type tag refers to a systematic identifier of an equipment. there are some conventions for generating equipment tags (e.g. norsok, kks standards), but most plant operators customize them and some sheets do not contain identifiers at all. type eq is for surface names of equipment types. the type quant refers to the physical properties/quantities describing the functional specifications of equipment and the type uom stands for unit of measurement. table statistics it is not obvious why performing ner in tables would benefit from sophisticated language models. in fact, looking at common tabular benchmark datasets, such as the ones used in the semtab challenge @xcite , detecting entities is usually very straightforward. since all tokens in a cell are assumed to represent a single entity, sub-cell ner is an unnecessary step and we only need to perform entity/cell linking. looking at the example table in figure 1 , however, gives the impression that these industrial spreadsheets are very differently structured from common benchmarks. there can be quite some text and even multiple sub-cell entities in a single cell. even more obvious is the difference at the column level where the tables in the semtab challenge contain on average 4 times less columns (µcol) than the tables describing plant equipment specification with much lower variance as well. this suggests that every token in our ner task has a much broader intra-and inter-cell context. ## related work there has been some research focused on extracting entities and their quantities from web tables. ibrahim et al. @xcite phrased this problem as entity linking using a table-biased markov random field and distant supervision. wu et al. @xcite employed bilstm models to encode rich-format documents (unstructured text, headings, tables) that mention electronic components, quantities and units of measure. they used hand-crafted labeling functions for collecting (weakly) labeled entities and relations which can be used as weak supervision. a recent work on table classification @xcite compared talms like tabert @xcite versus non-contextual word embedding methods for generating table vector representations. turl @xcite uses a transformer @xcite with table-specific attention mechanism which has been pre-trainined and fine-tuned towards solving the tasks of table interpretation: column type annotation, entity linking and relation extraction. however, this methods gen-erates representations on a cell level and therefore can not be applied for solving our ner problem. we are not aware of any work that uses talms for sub-cell table ner in an industrial setting. ## method we now define the table ner problem and outline how we encode tokens in tables using talms. we define a table as a tuple @xmath0, h), where @xmath1,1 , c 1,2 , . . . c i,j , . . . , c n,m } is the set of table body cells for n rows and m columns. every cell c i,@xmath2,j ,1 , w c i,j ,2 , . . . , w c i,j ,t is a sequence of tokens of length t. the table header @xmath3, h 2 , . . . , h m } is the set of corresponding m column header cells, where h @xmath4,1 , w h j ,2 , . . . , w h j ,q is a sequence of header tokens with length q. we use t @xcite to refer to the i-th row (@xmath5,:] ) and t [:,j] = {h j , c 1,j , . . . , c n,j } to refer to the j-th column of t . each labeled cell has an ner-tag sequence: (y 1 , y 2 , . . . , y |cell| ), where each y @xmath6. we use io tags, thus y is {o} ∪ {i-ent}, where @xmath7, eq, quant, uom}. ## experiments in this section we empirically study the performance of tabner compared against several baselines as well as the benefits of our domain-specific ## results and analysis convergence first, we analyze the progress of the training loss to study the convergence of the different ner models, see figure 4 . the loss of vanilla bert is quite flat from the beginning and after a few epochs gets stuck at a bad local optimum -hits early stopping based on validation. we argue that the global attention and position encoding across the full table are blurring the ner training signal for bert and since we could not find a setting to make it converge properly, we excluded it from further experiments. a more detailed analysis can be found in the appendix. in contrast to bert, the training loss for tab-ner is converging quickly. using only the training data, without augmentation, has the least steepest decline, which is due to observing less labels per epoch. lwtr shows a very steep decline in the beginning which, however, flattens out sooner than rdltab. our hypothesis here is that lwtr adds helpful variance in the labels at the beginning, but has less variance to add in the long run, since it can only sample from known training tables. rdltab on the other hand produces a more novel table context over time as the rdl has richer external vocabulary. ## conclusion in this paper, we demonstrate the applicability of talms to a novel ner problem in industrial spreadsheets. our experiments show that the tabular inductive bias of talms is not only beneficial for this problem, but may even a necessary condition when relying on pre-trained transformer-based models. in addition to that we present a da technique leveraging publicly-available industrial standard information models to produce augmented tables with physically sound and consistent context. compared to an off-the-shelve da, this technique shows improved ner performance. future work includes understanding how much tabular context is needed to make training largescale model more efficient. another fruitful area is active learning for tasks using talms to reduce the time for collecting expert labels. sangdoo yun, dongyoon han, sanghyuk chun, seong joon oh, youngjoon yoo, and junsuk choe. 2019. cutmix: regularization strategy to train strong classifiers with localizable features. in 2019 ieee/cvf international conference on computer vision, iccv 2019, seoul, korea (south), october 27 -november 2, 2019, pages 6022-6031. ieee. xiang zhang, junbo jake zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in advances in neural information processing systems 28: annual conference on neural information processing systems 2015, december 7-12, 2015, montreal, quebec, canada, pages 649-657. victor zhong, mike lewis, sida i. wang, and luke zettlemoyer. 2020. grounded adaptation for zeroshot executable semantic parsing. in proceedings of the 2020 conference on empirical methods in natural language processing, emnlp 2020, online, november 16-20, 2020, pages 6869-6882. association for computational linguistics.