S em E val-2021 Task 11: NLPC ontribution G raph - Structuring Scholarly NLP Contributions for a Research Knowledge Graph (2021)

Abstract: There is currently a gap between the natural language expression of scholarly publications and their structured semantic content modeling to enable intelligent content search. With the volume of research growing exponentially every year, a search feature operating over semantically structured content is compelling. The SemEval-2021 Shared Task NLPContributionGraph (a.k.a. ‘the NCG task’) tasks participants to develop automated systems that structure contributions from NLP scholarly articles in the English language. Being the first-of-its-kind in the SemEval series, the task released structured data from NLP scholarly articles at three levels of information granularity, i.e. at sentence-level, phrase-level, and phrases organized as triples toward Knowledge Graph (KG) building. The sentence-level annotations comprised the few sentences about the article’s contribution. The phrase-level annotations were scientific term and predicate phrases from the contribution sentences. Finally, the triples constituted the research overview KG. For the Shared Task, participating systems were then expected to automatically classify contribution sentences, extract scientific terms and relations from the sentences, and organize them as KG triples. Overall, the task drew a strong participation demographic of seven teams and 27 participants. The best end-to-end task system classified contribution sentences at 57.27% F1, phrases at 46.41% F1, and triples at 22.28% F1. While the absolute performance to generate triples remains low, as conclusion to the article, the difficulty of producing such data and as a consequence of modeling it is highlighted.

## introduction traditional search models over scholarly communication are now changing toward knowledge graph (kg) models operating on structured fine-grained scholarly content offering enhanced contextual search results. several initiatives exist to this end: google scholar, web of science @xcite , microsoft academic graph @xcite , openaire research graph @xcite , open research knowledge graph @xcite , semantic scholar @xcite to name just a few. these kg models differ in their content, their level of detail, etc., as they represent diverse aspects of scholarly communication. text, of course, is of seminal importance to science. it is as important as experimentation itself; unpublished research lacks validity. seen in another angle, it is hard to imagine a medium other than discourse that can convey a comprehensive picture of the scholarly investigation. for the wider research audience, it is interesting to read the full "stories" of science. nonetheless, since scientific literature is growing at a rapid rate @xcite and researchers today are faced with this publications deluge @xcite , it is increasingly tedious, if not practically impossible to keep up with the research progress even within one's own narrow discipline. in this regard, among the existing scholarly knowledge structuring initiatives, the open research knowledge graph (orkg) @xcite is posited as a solution to the problem of keeping track of research progress minus the cognitive overload that reading dozens of full papers impose. it aims to build a comprehensive kg that publishes the research contributions of scholarly publications per paper, where the contributions are interconnected via the graph even across papers. the orkg digital library (dl) framework can be accessed here https://www.orkg.org . motivated by the availability of a nextgeneration dl, we present the semeval-2021 nlp-contributiongraph (ncg) shared task as a step in the easier knowledge acquisition of contri-butions for researchers -the automated structuring of the unstructured article contributions. to this end, via the ncg task, we have formalized the building of such a scholarly contributions-focused graph over nlp scholarly articles as an automated task. in the subsequent paper, we detail our task in terms of its resources, organization, participants, and evaluations. ## data the ncg shared task comprised a dataset of nlp scholarly articles annotated for their contributions. the contributions were structured to be integrable within kg infrastructures such as the orkg @xcite that capture research overviews. the contributions were annotated in three different information granularities, i.e. (1) contribution sentences: a set of sentences about the article's contribution; (2) scientific terms and relations: a set of terms and relational predicates in the contribution sentences; and (3) triples: semantic statements that pair the terms with a predicate, modeled toward subject-predicate-object rdf statements for kg building. this latter set of annotations formed the actual graph. inspired after article sections, the triples were organized under three (mandatory) or more of 12 total information units (ius), viz. researchproblem, approach, model, code, dataset, experimentalsetup, hy-perparameters, baselines, results, tasks, experiments, and ablationanalysis. ## task description our comprehensive ncg shared task formalism was as follows. given a scholarly article a in plaintext format, the goal was to extract (1) a set of contribution sentences c @xmath0, ..., c sent n }, (2) a set of scientific knowledge terms and predicates from c sent referred to as entities @xmath1, ..., e n }, and (3) to organize the entities e as a set of (subject,predicate,object) triple statements @xmath2, ..., t n } toward kg building organized under three or more of the 12 total ius. ## participating system descriptions the ncg shared task received public entries from 7 participating teams in all. in this section, we briefly describe the teams' systems in terms of the three parts of the ncg task, i.e. contribution sentence classification, scientific terms and relations extraction, and triples extraction. ## shared task results in this section, we present the results of the seven participating teams' systems. the results in ## discussion finally, we conclude our shared task paper with a discussion on the perceived limitations of our dataset that can potentially be addressed in future work. thereby, a new dataset will present new opportunities to evaluate systems on this novel task. ## conclusions we have detailed the nlpcontributiongraph shared task that entailed structuring research contributions in nlp articles as structured kgs. this task is the first-of-its-kind to be organized in the semeval series. it attracted a strong participation demographic of 27 participants and seven teams -bert transformer models were a popular choice among the participant systems in two different capacities, i.e. as classifiers or sequence labelers. our task also saw the use of traditional parsers such a dependency syntax parsing technology. further, some systems leveraged a hybrid approach including a combination of heuristics and machine learning. while the end-to-end task performance was low showing the task considerably challenging, each individual subtask toward obtaining an ncg, i.e. contribution sentence classification, scientific terms and relations extraction, and triples formation, demonstrated high performances in the subtask-only evaluation setting, i.e. when given gold data from the previous stage. the best system adopted a hybrid approach which seemed the most effective strategy for building the ncg. the ncg dataset is publicly available (d @xcite and a kg overview of a structured form of our paper is here https://www.orkg.org/ orkg/comparison/r74774 .