K now G raph@ IITK at S em E val-2021 Task 11: Building Knowledge Graph for NLP Research (2021)

Abstract: Research in Natural Language Processing is making rapid advances, resulting in the publication of a large number of research papers. Finding relevant research papers and their contribution to the domain is a challenging problem. In this paper, we address this challenge via the SemEval 2021 Task 11: NLPContributionGraph, by developing a system for a research paper contributions-focused knowledge graph over Natural Language Processing literature. The task is divided into three sub-tasks: extracting contribution sentences that show important contributions in the research article, extracting phrases from the contribution sentences, and predicting the information units in the research article together with triplet formation from the phrases. The proposed system is agnostic to the subject domain and can be applied for building a knowledge graph for any area. We found that transformer-based language models can significantly improve existing techniques and utilized the SciBERT-based model. Our first sub-task uses Bidirectional LSTM (BiLSTM) stacked on top of SciBERT model layers, while the second sub-task uses Conditional Random Field (CRF) on top of SciBERT with BiLSTM. The third sub-task uses a combined SciBERT based neural approach with heuristics for information unit prediction and triplet formation from the phrases. Our system achieved F1 score of 0.38, 0.63 and 0.76 in end-to-end pipeline testing, phrase extraction testing and triplet extraction testing respectively.

## introduction given the advancements in natural language processing (nlp), a large number of research papers are published every year. however, given the field's dynamic nature, keeping track of all the papers is a non-trivial task. this motivated the formulation of an open research knowledge graph @xcite , a knowledge graph of research contributions and the relation between them. task 11 of semeval 2021 @xcite formalizes the building of a contributions-focused knowledge graph of nlp literature. the task is divided into three sub-tasks: recently, transformer-based approaches have been popular for nlp applications @xcite . for the sub-task a, we propose a sentence level classifier, leveraging scibert @xcite and bidirectional long short-term memory (bilstm) @xcite . scibert model has been trained on 1.14m scientific papers from semantic scholar corpus, which has 18% papers from the computer science domain. our system for subtask b also uses scibert based model using crf 2 -learning semantic sentence embeddings using discriminator 3 -abstract 4 -we propose method for obtaining sentence -level embeddings . 22 -our model consists of a sequential encoderdecoder that is further trained using discriminator . 26 -this , however , does not imply that the whole sentence is correctly generated . 28 -we further ensure that this is close to desired ground -truth embeddings while being far from other embeddings. 148 -we start with baseline model which we take as a simple encoder and decoder network with only the local loss. ## system overview the proposed system is shown in figure 2 depicting the entire pipeline and its respective model. ## results we built several models using scibert, lstm and cnn for each sub-task to understand each method's significance ( in sub-task c, some iu triplet's such as model, hyperparameters, results were present in large number comparatively and while tasks and dataset iu triplets are scarce in number. this skewness resulted in biasing of our multi-label classification model for iu prediction. in triplet formation, our predicate approach fails when the predicate is not present in the sentence and selected from the organizer's closed set of predicates. the proposed system also fails in the case of branching between the triplets, i.e. multiple triplets share the same subject phrase. our triplet formation is unable to predict approach, dataset and tasks triple. on visualizing, we found that model triplets are predicted instead of approach triplets. further, the triplets lack enough features to be classified into an information unit using a neural-based method. our system performance on sub-task c is poor in the end-to-end pipeline and phrase extraction testing because we submitted our initial experimentation model. ## conclusion in this paper, we have presented our system for nlpcontributiongraph task of semeval 2021. we found that neural models combined with heuristics can build a knowledge graph by dividing it into small tasks. the heuristic-based model can outperform the neural approach in the triplet formation of some information units. in future work, neural models can use sparse transformers to encode long documents without increasing much memory. the pre-defined predicate incorporation could also be a future direction of work for our system.