Joint Entity and Relation Extraction Based on Table Labeling Using Convolutional Neural Networks (2022)

Abstract: This study introduces a novel approach to the joint extraction of entities and relations by stacking convolutional neural networks (CNNs) on pretrained language models. We adopt table representations to model the entities and relations, casting the entity and relation extraction as a table-labeling problem. Regarding each table as an image and each cell in a table as an image pixel, we apply two-dimensional CNNs to the tables to capture local dependencies and predict the cell labels. The experimental results showed that the performance of the proposed method is comparable to those of current state-of-art systems on the CoNLL04, ACE05, and ADE datasets. Even when freezing pretrained language model parameters, the proposed method showed a stable performance, whereas the compared methods suffered from significant decreases in performance. This observation indicates that the parameters of the pretrained encoder may incorporate dependencies among the entity and relation labels during fine-tuning.

## introduction the purpose of a joint entity and relation extraction is to recognize entities and relations in a text. a task can be decomposed into two subtasks: named entity recognition (ner) and relation extraction @xcite . in recent years, several researchers have built high-performance ner and re systems based on contextualized representations @xcite @xcite @xcite . these contextualized representations obtained from pretrained language models, such as bidirectional encoder representations from transformers (bert) @xcite , have significantly improved the performance for various nlp tasks. as a result, studies on ner and re have focused on the design of task-specific layers stacked on top of pretrained language models. a common idea is to formulate ner and re as table-filling problems @xcite . the core concept is to extract entities and relations by filling a table with entity labels in the diagonal cells and relation labels in the off-diagonal cells. based on this concept, @xcite proposed tablert, which is a combined system of ner and re based on a pretrained bert. tablert predicts the diagonal cells sequentially and offdiagonal cells simultaneously. although the system is simple and effective, it ignores the dependencies among predicted relation labels. as noted in @xcite , this does not improve the performance with label dependencies incorporated through refined decoding orders. we propose tablert-cnn, a novel ner and re system that encodes the dependencies among the cells within the table . our method employs two-dimensional convolutional neural networks (2d-cnns), which are widely used neural architectures for object detection @xcite . we considered each table as a 2d image and each cell as a pixel, transforming the task into a tablelabeling problem at the cell level. by applying 2d-cnns to the output of bert, the system is expected to implicitly perceive local information and label dependencies from neighboring cells. notably, the range of cells to be processed is expandable by stacking multiple cnn layers, we model the dependencies among distant cells. we evaluated tablert-cnn based on multiple benchmarks: conll04 @xcite , ace05 @xcite , and ade @xcite . the experimental results showed that the performance of the proposed method is on par with those of current state-ofart systems. we hypothesized that parameter updates during fine-tuning helped the bert encoder capture the necessary dependencies for label predictions; thus, incorporating dependencies using the cnn became less helpful. to verify this hy-pothesis, we compared the performance of several ner and re systems while keeping the bert parameters frozen and updating them during fine tuning. in addition, we used different layers from which the prediction model extracts token embeddings to analyze how parameter updates within each layer contribute to the performance. as a result, tablert-cnn still performed well while keeping the bert parameters unchanged, whereas the performance of the other systems significantly decreased. this observation indicates the ability of the bert architecture to consider token-and label-wise dependencies during task-specific fine tuning. the source code for the proposed system is publicly available at https://github.com/ youmima/tablert-cnn . ## analysis although our system exhibited a good performance based on multiple datasets, no significant improvement was observed against tablert @xcite . we hypothesize that the reason for this is the parameter updates within the bert encoder during fine-tuning, which overshadowed the ability of the cnns in the prediction model. selfattention modules within bert potentially learn to encode the dependencies between word pairs during fine-tuning, overlapping with those captured by the cnns. experiments were conducted to verify this hypothesis. specifically, we trained multiple bertbased ner and re systems (i.e., systems using a pretrained bert as the encoder) while freezing the bert parameters ( § 5.1). in this manner, we prevented the encoder from obtaining task-specific features during fine-tuning. the performance of these encoder-frozen systems was compared with that of their counterparts, whose encoder parameters were updated during fine-tuning. based on this comparison, we investigated the extent to which the parameter updates within bert contribute to method parameter encoder layer updates 0 1 2 4 6 8 10 12 tablert @xcite in addition, we are interested in how each bert layer encodes dependencies that are helpful for ner and re. previous studies have utilized the outputs of the top bert layers to produce word representations @xcite @xcite . however, we are curious whether the bottom or middle bert layers also store useful information for solving the ner and re. therefore, we fed hidden states at the {0, 1, 2, 4, 6, 8, 10, 12}th bert layer into the prediction model and examined the difference in performance ( § 5.2). here, the 0th layer denotes the embedding layer of the bert encoder. our analysis includes spert (eberts and ulges, 2020), tablert @xcite and the proposed method. we included tablert for comparison because it is a counterpart of our system, incorporating no dependencies while performing re. we included spert for comparison because it is a strong baseline utilizing a pretrained bert encoder. systems were trained on the conll04 @xcite training set and evaluated on the development set, using bert base @xcite as the encoder. the experimental results are listed in tables 3 and 4 . the plots corresponding to these results are presented in appendix b. finally, we analyze the effect of 2d-cnns ( § 5.3). ## conclusion we presented tablert-cnn, a novel method for jointly extracting entities and relations with 2d-cnns. the method casts ner and re as tablelabeling problems, representing each table cell as a pixel and each table as a 2d image. by applying 2d-cnns, the method predicts the label of each table cell to extract entities and relations. experiments conducted on conll04, ace05, and ade demonstrated that tablert-cnn performed on par with current state-of-art systems when the pretrained encoders were aligned. to explore why tablert-cnn did not outperform existing systems by a significant margin, we conducted experiments to compare their performance with and without parameter updates of the bert encoder during the fine-tuning. we observed that tablert-cnn performed reasonably well even without updating the encoder parameters, whereas its competitors suffered a decrease in performance. these results indicate that the bert encoder can capture task-specific dependencies among tokens and labels within its architecture, based on parameter updates during fine-tuning. in the future, we plan to model the dependencies among table cells using other neural architectures. prospective directions include 2d-transformers that compute the attention across element pairs in a 2d array, or routing transformers that utilize local attentions.