ITNLP at S em E val-2021 Task 11: Boosting BERT with Sampling and Adversarial Training for Knowledge Extraction (2021)

Abstract: This paper describes the winning system in the End-to-end Pipeline phase for the NLPContributionGraph task. The system is composed of three BERT-based models and the three models are used to extract sentences, entities and triples respectively. Experiments show that sampling and adversarial training can greatly boost the system. In End-to-end Pipeline phase, our system got an average F1 of 0.4703, significantly higher than the second-placed system which got an average F1 of 0.3828.

## introduction the knowledge graph (kg) describes the concepts, entities and their relationships in the objective world in a structured form, expresses internet information in a form closer to the human cognitive world. information extraction is the first step of the kg construction. information extraction is a technology that extracts structured information such as entities and relationships from semistructured or unstructured data automatically. similarly, as the rate of research publications increases, it is critical to construct knowledge graphs to represent scholarly knowledge efficiently. the target of the nlpcontributiongraph task @xcite is to find a systematic set of patterns of subject-predicate-object statements for the semantic structuring of scholarly contributions that are generically applicable for nlp research articles, then apply the discovered patterns in the creation of a larger annotated dataset for ingesting the dataset into the open research knowledge graph infrastructure to assist users manually manage their article contributions. our task consists of three sub-tasks: sentences extraction (se), phrases extraction (pe) and triples extraction (te). the dataset used in the nlpcontributiongraph task contains hundreds of natural language pro-cessing (nlp) scholarly articles annotated for their contributions. each article is written in english and contains three types of annotation information: 1) contribution sentences; 2) scientific term and predicate phrases from the sentences; and 3) subject-predicate-object triple statements from the phrases toward kg building. our code is available at https://github.com/ itnlp606/nlpcb-graph . ## related work in recent years, pretrained language models @xcite @xcite have achieved impressive performance in various nlp tasks including information extraction. bert @xcite uses bidirectional transformers @xcite to pretrain the model on the masked language model (mlm) task and the next sentence prediction (nsp) task and advances the state-of-the-art for eleven nlp tasks. the system presented in this paper is based on fine-tuning bert. this section will introduce two strategies to boost the bert model. ## system description for three sub-tasks in nlpcontributiongraph, we designed four modules to implement these tasks. these modules use fine-tuning bert with fat as ## experimental setup in this section, we did some ablation analysis on the validation set for the boosting methods proposed in this paper, and gave some analysis through the experimental results. ## conclusion bert is a powerful model that has considerable applications in numerous fields of nlp. using bert in the nlpcontributiongraph task allows researchers to read papers more efficiently. the methods of adversarial training and sampling proposed in this paper can greatly boost the performance of bert on this task and can also offer some thoughts for future work on knowledge extraction of papers.