UIUC _ B io NLP at S em E val-2021 Task 11: A Cascade of Neural Models for Structuring Scholarly NLP Contributions (2021)

Abstract: We propose a cascade of neural models that performs sentence classification, phrase recognition, and triple extraction to automatically structure the scholarly contributions of NLP publications. To identify the most important contribution sentences in a paper, we used a BERT-based classifier with positional features (Subtask 1). A BERT-CRF model was used to recognize and characterize relevant phrases in contribution sentences (Subtask 2). We categorized the triples into several types based on whether and how their elements were expressed in text, and addressed each type using separate BERT-based classifiers as well as rules (Subtask 3). Our system was officially ranked second in Phase 1 evaluation and first in both parts of Phase 2 evaluation. After fixing a submission error in Pharse 1, our approach yields the best results overall. In this paper, in addition to a system description, we also provide further analysis of our results, highlighting its strengths and limitations. We make our code publicly available at https://github.com/Liu-Hy/nlp-contrib-graph .

## introduction with the deluge of scientific publications in recent years, keeping pace with the literature and managing information overload have become increasingly challenging for researchers. there is a growing need for tools that can automatically extract and structure semantic information from scientific publications to facilitate advanced approaches to information access and knowledge curation @xcite . the field of natural language processing (nlp) has witnessed an enormous growth in recent years with advances in deep learning, and there are increasing efforts in developing methods to extract scholarly knowledge from nlp pub-lications @xcite . one such effort is nlpcontributions, an annotation scheme for describing the scholarly contributions in nlp publications and a corpus annotated using this annotation scheme (d' @xcite . this corpus has been proposed for training and testing of machine reading models, whose output can be integrated with the open research knowledge graph framework (orkg) @xcite . orkg formalizes the research contributions of a scholarly publication as a knowledge graph, which can further be linked to other publications via the graph. the goal of the nlpcontributiongraph (ncg) shared task (d @xcite is to facilitate the development of machine reading models that can extract orkg-compatible scholarly contribution information from nlp publications. the shared task consists of three subtasks (see d @xcite for a more detailed description): in this paper, we describe our contribution to ncg shared task. we built a cascade of neural classification and sequence labeling models based on bert @xcite . for subtask 3, we characterized triples based on whether and how their elements are expressed in text, and employed different models for each category. we also explored rule-based heuristics to improve model performance. our models had the best overall performance in the shared task (57.27%, 46.41%, and 22.28% f 1 score in subtasks 1, 2, and 3, respectively). the results are encouraging for extracting scholarly contributions from scientific publications, although there is much room for improvement. ## system overview in this section, we first describe our data preprocessing steps. next, we discuss our models for each subtask, and the experimental setup for our end-toend system (phase 1). we provide an overview of the system in figure 1 and provide examples for illustration, when necessary. ## results all the subtasks were evaluated on f 1 scores, and among them, triple extraction is evaluated by the micro-average of f 1 scores on each information unit. in the end-to-end evaluation (phase 1), the participants were provided with the raw input to perform all three subtasks sequentially. we were officially ranked second in phase 1, due to a submission error that resulted in phrase extraction f 1 of zero. our correct submission achieved an average f 1 of 49.7%, the best score among all participating teams. we observe that, although the performance of our system on sentence classification is lower than human performance (57.27% vs. 67.44% f 1 ), using its own sentence predictions, our system outperforms human annotators on phrase recognition (46.41% vs. 41.84% f 1 ), and reaches comparable performance to human annotators on triple extraction. we also note that our system generally performs better in terms of recall than precision. we were officially ranked first in both parts of evaluation phase 2. in part 1, the participants were provided with the sentences labels to conduct phrase recognition and triple extraction sequentially; in part 2, both the sentence labels and the phrase labels were provided to extract triples. we essentially followed our method in phase 1 on phrase recognition and triple extraction, but made several attempts to improve the performance, which we discuss in section 4. our results in both parts of the phase 2 evaluation are shown in ## performance analysis in this section, we analyze the performance of several components of our system and compare different schemes for entity representation and triple extraction. we also discuss some possible methods for improvement based on our shared task results and follow-up experiments. ## conclusion we developed a system to generate structured representations of research contributions described in nlp publications in a manner compatible with the orkg framework, achieving the top performance in the ncg shared task. we combined a cascade of state-of-the-art bert-based classification and sequence labeling models with rule-based methods. in particular, we proposed a novel approach for triple extraction, where we tackled triples with different characteristics using different relation classification methods. we also explored various alternatives to the components in our end-to-end system to analyze the contribution of individual components. in future work, we plan to improve the differentiation of similar units (e.g., model vs. ap-proach), improve the extraction of type b triples, and address coordinated triples more thoroughly. we did not attempt to extract approximately 6% of the triples that did not fit in our classification (