ECNUICA at S em E val-2021 Task 11: Rule based Information Extraction Pipeline (2021)

Abstract: This paper presents our endeavor for solving task11, NLPContributionGraph, of SemEval-2021. The purpose of the task was to extract triples from a paper in the Nature Language Processing field for constructing an Open Research Knowledge Graph. The task includes three sub-tasks: detecting the contribution sentences in papers, identifying scientific terms and predicate phrases from the contribution sentences; and inferring triples in the form of (subject, predicate, object) as statements for Knowledge Graph building. In this paper, we apply an ensemble of various fine-tuned pre-trained language models (PLM) for tasks one and two. In addition, self-training methods are adopted for tackling the shortage of annotated data. For the third task, rather than using classic neural open information extraction (OIE) architectures, we generate potential triples via manually designed rules and develop a binary classifier to differentiate positive ones from others. The quantitative results show that we obtain the 4th, 2nd, and 2nd rank in three evaluation phases.

## introduction the notion of open research knowledge graph (orkg) is first proposed by @xcite who take steps toward a knowledge graph based infrastructure that acquires scholarly knowledge in machine actionable form. in that form, researchers can keep up with cutting edge academic achievements and eliminate cognitive overload. to accelerate the construction of orkg, an automatic system is expected. the semeval-21 task11 is a triple extraction task targeted at building that system. as shown in table1, the task is divided into three sub-parts corresponding to different processing steps: sub-task a detects contribution sen- * equal contribution tences in english articles and classifies them into information units such as approaches, models, and ablation -analysis; sub-task b extracts scientific terms and relational cue phrases from contribution sentences; sub-task c infers subject-predicateobject triples for kg building with the results of two previous sub-tasks. for sub-task a-contribution sentence detection-the evaluation data covers a larger sphere than the training data. additionally, the amount of annotated samples differs among research fields. hence, we use self-training to generate a set of silver samples for fields lacking gold data. an ensemble of fine-tuned plm based classifiers is then deployed to categorize sentences. for subtask b-scientific term extraction-we compared bert based sequence labeling systems in detail and chose the best architecture. for sub-task c-triple generation-we give insight into the construction of triples and designed a rule for potential triples generation. a binary classifier is then applied to distinguish the positive triples. our quantitative results show data augmentation via self-training is of paramount importance for sub-task a. although seldom is crf used with transformer-based language models together, in the system for sub-task b, an additional crf layer after a roberta based encoder can still boost performance. in sub-task c, popular neural information extraction models are inferior to the rule based methods. ## system description systems applied for contribution sentence detection and scientific term extraction are based on the roberta @xcite ,scibert @xcite and basic bert model with taskspecific modifications. for triple classification tasks, a scibert @xcite based model is used for candidate triple classification. moreover, self-training, ensemble and rule design enhanced system performance in different ways. ## experimental setup in this section, we describe the models we used in the final submission and their parameters in detail. it should be possible to reproduce our work. ## conclusion we engaged in semeval-2021 task11 nlpcontri-butiongraph with models integrating features suitable for disparate tasks. we took insight on the impact of different parts on the final results, finetuned hyperparameters, and attempted various feature engineering methods. we ranked 4 th , 2 nd , and 2 nd in three evaluation phases and our final model demonstrated its superiority over several strong baselines.