D uluth at S em E val-2021 Task 11: Applying D e BERT a to Contributing Sentence Selection and Dependency Parsing for Entity Extraction (2021)

Abstract: This paper describes the Duluth system that participated in SemEval-2021 Task 11, NLP Contribution Graph. It details the extraction of contribution sentences and scientific entities and their relations from scholarly articles in the domain of Natural Language Processing. Our solution uses deBERTa for multi-class sentence classification to extract the contributing sentences and their type, and dependency parsing to outline each sentence and extract subject-predicate-object triples. Our system ranked fifth of seven for Phase 1: end-to-end pipeline, sixth of eight for Phase 2 Part 1: phrases and triples, and fifth of eight for Phase 2 Part 2: triples extraction.

## introduction the rapid rate at which scientific literature grows makes it difficult to keep up with new research even in one's own field. automated solutions are challenging since text formatted and written for human consumption does not lend itself to machine processing. @xcite have proposed a knowledge graph based system for collecting and structuring articles in a machine-readable format. this requires the annotation of many scholarly articles, a task that is time consuming to do by hand. the purpose of this semeval task @xcite is to enable the construction of a scholarly contributions graph over english language nlp articles by automating the task of annotating scientific papers. this annotation processes consists of: 1. selecting sentences that describe the contribution of the article and outputting them to a sentences.txt file, 2. extracting scientific entities and relations from the selected sentences and outputting them to an entities.txt file, 3. and structuring the entities and relations into triples of the form subject-predicate-object. these triples are sorted into one of twelve information units, which are labels that describe the type of contribution being made by the sentence outlined in the triples file. following is an example of the annotation process using a sentence taken from the training data with the human-labeled entities in braces: we @xcite this task requires the 1) identification of this sentence as a contribution sentence belonging to the information unit hyperparameters, 2) the extraction of the entities in braces, and 3) the formatting of those entities into triples. this example and various others will be used throughout this paper to help contextualize our system description. semeval-2021 task 11 was organized into three phases, there are four datasets mentioned in this paper. gold data refers to the sentences.txt, entities.txt, and triples folders released by the organizers after each phase of the task. test data refers to the data used to test the system during evaluation phases. training data refers to the entire training dataset provided by the organizers. the trial dataset is the segment of the training dataset that we used to test the duluth system during validation, before the first evaluation phase began. for more information on how this dataset was created, see appendix b. our approach foot_0 employs a variety of techniques to address each component of the task. the selection of contribution sentences was done by finetuning the base deberta model @xcite on the training data, as fine-tuned bert @xcite models have been found to perform well on multi-class text classification tasks @xcite . the fine-tuned model is used to classify each sentence as either non-contributing, or one of the twelve information units. the selected contribution sentences were then tagged to indicate likely scientific entities using a maximum entropy markov model (memm) @xcite that was trained on the noun phrases selected as entities in the training data. a dependency parse (manning et al., 2014) was then found for each sentence. the dependency parse and entity tags were then leveraged to select contributing phrase spans and triples by using the entity tags to determine whether a subject or object noun phrase ought to be considered, and using the dependencies to extract subject-predicate-object patterns from sentence. ## previous work two previous semeval tasks were also concerned with the extraction of relations and key phrases from scientific publications: semeval 2017 task 10: (scienceie -extracting keyphrases and relations from scientific publications) @xcite , and semeval 2018 task 7: (semantic relation extraction and classification in scientific papers) @xcite . many of the approaches in these tasks use neural models to extract entities and their relations. the ai2 system @xcite at semeval-2017 task 10, which ranked first and second for task scenarios one and three respectively, approached this by building separate entity and relation models, each of which contain layers of lstms. the eth-ds3lab system @xcite at semeval-2018 task 7, which ranked first in three of four subtasks, built an entity and relation classifier using a combination of rnns and cnns. other approaches used supervised machine learning algorithms while leveraging grammatical features. the lipn system @xcite approached semeval-2017 task 10 by first filtering possible keyphrases by labeling phrases with their pos sequence. candidate keyphrases are filtered by comparing the pos tags of the phrase with pos sequences developed from the training data. they then trained a crf model using the candidate phrases labeled with iob tags. they were able to improve recall for keyphrase extraction by filtering candidate sentences before using a crf. ## selection of contribution sentences we approached the selection of a contribution sentence as a multi-class sentence classification problem with 13 classes, where each of the twelve information units is a class, and class 0 represents non-contributing sentences. although the subtask of sentence extraction could be performed using a binary classifier to label sentences as either contributing or non-contributing, we decided to sort the sentences further into their information units. the alternative would require classifying phrases or triples further down the pipeline; the benefit to classifying the contributions during the sentence extraction step is that the whole context of each sentence is taken into account. the main challenge with this approach was in the unbalanced nature of the data; 90.11% of the sentences used to train the classification model for phase 1: end-to-end pipeline were non-contributing sentences, and the standard deviation of the frequencies of contributing classes was 8.59%. these frequencies can be seen in appendix d. logistic regression and decision tree classifiers @xcite were not able to identify the underrepresented classes such as tasks and dataset, and were heavily skewed towards the dominant class of non-contributing sentences. during initial experiments using the trial dataset described in appendix b, decision tree classifier earned a macro-f1 score of 0.1736 and the logistic regression classifier earned a macro-f1 score of 0.1738. the base deberta model performed better than both decision tree and logistic regression classifiers on the trial dataset, resulting in a macro-f1 score of 0.3079. ## entity and relation extraction the duluth system for phase 1: end-to-end pipeline combined statistical and rule-based approaches for extracting scientific entities and their predicates from the contribution sentences. we used a dependency parser to extract noun phrases and their predicates, and trained an maximum-entropy markov model (memm) on the training data entities files to predict whether each noun phrase contains a scientific entity. ## triples extraction during evaluation phase 1: end-to-end pipeline, almost all of the task of extracting subject-predicate-object triples into information units files was already performed by previous steps. the entity extractor described in section 4 extracts phrase spans three at a time, following the subject-predicateobject format needed to organize phrase spans into triples. the sentence extractor described in section 3 classifies sentences into their information units, so the class label for the sentence that the triple is extracted from can be used to determine the information unit that the triple belongs to. for phase 1: end-to-end pipeline and phase 2 part 1: phrases and triples, the system formed triples based on the subject, predicate, and object phrases determined by the entity selection process described in section 4. for example, if the sentence "dropout, with a rate of 0.5, is employed on the word embedding layer" is classified by the sentence extractor as belonging to the information unit experimental setup, and the phrases "dropout", "is employed", and "word embedding layer" were selected as a subject-predicate-object pattern, then triples would be formatted in the experimental-setup.txt triples file like so: ## sentence selection results the official competition f1 score on the gold standard test data for selection of contribution sentences in phase 1: end-to-end pipeline was 0.38095. this score evaluates the results as a binary classification problem, where sentences are either contributing or non-contributing. because our sentence selector performs multi-class classification, labeling each sentence as either non-contributing or belonging to one of twelve information units, we provide a confusion matrix in ## entity and relation extraction results the gold test data had 13,028 total phrase spans, of which our system identified 4,277. 39.72% of these predicted phrase spans were exact matches with gold phrase spans. 29.97% of predicted phrase spans were complete false positives, not overlapping with any gold phrase spans. 30.30% of predicted phrase spans were not perfect matches, but did overlap with gold phrase spans. the majority of partial matches were missing a word, reflecting the fact that the average span length for the gold data was 12.37 characters but the average span length for the predicted data was 10.49 characters. 42.44% of partial matches were missing characters in the beginning of the phrase span, 52.11% were missing characters in the end of the phrase span, 6.56% had extra characters in the beginning of the phrase span, and 14.97% had extra characters in the end of the phrase span. ## triples extraction results the official f1 score on the gold standard test data for triples extraction from phase 2 part 2 was 0.2762, and the f1 score for information units was 0.7556. the duluth system scored the lowest at extracting triples for the information units dataset, tasks, experiments, receiving f1 scores of 0.0, 0.0, and 0.0597 respectively. the extracted triples are input into files named for the information unit they belong to. triples extracted from a sentence classified as research problem are output into a file named research-problem.txt. the gold data does not contain any tasks.txt files, so all tasks.txt files generated by our system were false positives. the gold data contained two dataset.txt files, both of which were missed by our system. these scores are consistent with the initial results from sentence classification, as seen in our triples extraction system relies heavily on part-of-speech tagging to determine whether an entity belongs to the edge of a triple or the middle (for example, noun-phrases are more likely to be subjects or objects, and verb-phrases are more likely to be predicates). in order to determine the efficacy of this approach, we look at the words that were improperly positioned in our system triples. one pattern that emerged is that many words were wrongly positioned in the duluth system data. some phrases and their pos tags that only exist in the middle position in the gold data that are found in the edges of triples in the system data include "achieves" (vbz), "propose" (vb), "performs" (vbz), and "uses" (vbz). other phrases that only exist in the edges of triples in the gold data that are found in the middle position in the system data include "outperformed" (vbg), "worse" (jjr), "outperforming" (vbg), and "randomly initialized" (rb vbn). this observation is consistent with the pos distributions found for the gold triples not identified by the system and the system triples that were wrongly identified; 70 of the 792 phrases belonging to the edge of the gold triples missed by the duluth system have the pos pattern rb vbn, while 46 of the 730 phrases that were wrongly positioned in the middle of triples by the duluth system also have the pattern rb vbn. similarly, 324 of the 1,656 phrases belonging to the middle of gold triples missed by the duluth system have the pos tag vbz, while 712 of the 4,242 phrases that were wrongly positioned at the edge of triples by the duluth system have the same pos tag of vbz. this shows that the duluth system sometimes shifts the phrases to the left or right of where they ought to be in the triple pattern. ## future work one weakness of this system is that the selection of contributing sentences only uses sentence-level information; the classifier misses useful contextual information such as headers and the predicted class of preceding and following sentences. future work may be able to address this problem by fine-tuning bert to classify sequences of sentences rather than isolated sentences @xcite . generally, a document-level approach could be beneficial in terms of capturing important context. another issue is that the end of the system does not have the ability to provide feedback to earlier parts of the pipeline; the only agency it has in terms of contributing sentence selection is the ability to discard a sentence provided by the sentence classifier. future work could incorporate a neural entity classification model into the entity and triple extraction subsystem, which could be used to validate or invalidate the classification made at the sentence level @xcite .