YNU - HPCC at S em E val-2021 Task 11: Using a BERT Model to Extract Contributions from NLP Scholarly Articles (2021)

Abstract: This paper describes the system we built as the YNU-HPCC team in the SemEval-2021 Task 11: NLPContributionGraph. This task involves first identifying sentences in the given natural language processing (NLP) scholarly articles that reflect research contributions through binary classification; then identifying the core scientific terms and their relation phrases from these contribution sentences by sequence labeling; and finally, these scientific terms and relation phrases are categorized, identified, and organized into subject-predicate-object triples to form a knowledge graph with the help of multiclass classification and multi-label classification. We developed a system for this task using a pre-trained language representation model called BERT that stands for Bidirectional Encoder Representations from Transformers, and achieved good results. The average F1-score for Evaluation Phase 2, Part 1 was 0.4562 and ranked 7th, and the average F1-score for Evaluation Phase 2, Part 2 was 0.6541, and also ranked 7th.

## introduction as the number of research publications increases, there is a growing need for digital libraries to equip researchers with alternative knowledge representations. in addition, because scientific literature is growing at a rapid rate and researchers today are faced with a publication deluge, it is difficult to keep up with the research progress even within ones own narrow discipline. the open research knowledge graph (orkg) @xcite is posited as a solution to the problem of keeping track of research progress without the cognitive overload imposed by reading dozens of full papers. to this end, the aim of this task is to build a comprehensive knowledge graph that represents the research contributions of scholarly publications per paper and also shows where the contributions are interconnected across papers (d'souza and auer, 2020). the task was defined on a dataset containing natural language processing (nlp) scholarly articles with their contributions structured to be integrable within a knowledge graph infrastructure, such as the orkg. the structured contribution annotations were provided as follows: (1) contribution sentences: a set of sentences about the contribution in the article; (2) scientific terms and relations: a set of scientific terms and relational cue phrases extracted from the contribution sentences; and the difficulty of this task lies in text classification @xcite and sequence labeling @xcite . text classification refers to determining which of the two or more labels a one-dimensional linear sequence belongs to. similarly, sequence labeling is used to tag each element in a one-dimensional linear sequence with a label from a set of labels. before the popularity of deep learning, the common solutions to the sequencelabeling problem were all based on either the hidden markov model @xcite or conditional random field (crf) (ye and ling, 2018), with crf being the mainstream method. with the development of deep learning, convolutional neural networks (cnn) @xcite and recurrent neural networks (rnn) @xcite have achieved great success in text classification and sequence labeling. since then, long short-term memory (lstm) @xcite , bi-lstm (bi-directional long short-term memory), and other models @xcite have performed better than cnn and rnn in text classification and sequence labeling . however, since the introduction of bidirectional encoder representations from transformers (bert) @xcite , the accuracy and training efficiency in both text classification and sequence labeling have reached new heights. the semeval-2021 shared task 11 (d'souza et al., 2021) consists of three subtasks: in this study, after analysis, we converted the above three subtasks into four downstream tasks in the field of nlp: binary classification for solving subtask 1, sequence labeling for solving subtask 2, multiclass classification and multi-label classification for solving subtask 3. then, we used a pre-trained language model, bert, to generate word embeddings and integrated them into the corresponding models for the different tasks. after completion of the task, our results were satisfactory. our submission ranked 7th in both part 1 and part 2 of evaluation phase 2. the implementation for our system is made available via github foot_0 . the remainder of this paper is organized as follows. section 2 describes the details of the bert model used in our system. section 3 presents the experimental results. finally, the conclusions are presented in section 4. ## system description we used a pre-trained bert model to accomplish the task, which was defined in terms of three dataset annotation elements, where the extraction of each data element relied on the extraction of the previous data element. ## experimental results datasets. the nlpcontributiongraph shared task comprises a dataset of nlp scholarly articles with annotated contributions. the annotations were provided in terms of three data elements: (1) contribution sentences, (2) scientific term and predicate phrases from the sentences, and (3) (subject, predicate, object) triple statements. all the triples together formed the contribution-centered knowledge graph of the articles. the dataset released by the organizers contained 237 annotated articles as training data and 155 annotated articles as testing data for the final evaluation phases. for the training data, the annotations of each scholarly article were provided in a directory. the directory contained the full article in plain text, which was pre-processed for tokenization and sentence splitting. the annotations were provided in the following three files: ## conclusions in this paper, we presented the system we submitted to the semeval-2021 task 11, which leveraged a pre-trained bert model to extract contributions from an nlp scholarly article using binary classification, sequence labeling, multiclass classification, and multi-label classification. the experimental results showed that the proposed models achieved a good performance in the final evaluation phases. furthermore, in the three subtasks, there appeared to be significant room for improvement compared to the top-ranked participant systems. therefore, in future research, we will attempt to generalize models with better capabilities to obtain better results.