NLPS hared T asks: A Corpus of Shared Task Overview Papers in Natural Language Processing Domains (2022)

Abstract: As the rate of scientific output continues to grow, it is increasingly important to develop systems to improve interfaces between researchers and scholarly papers. Training models to extract scientific information from the full texts of scholarly documents is important for improving how we structure and access scientific information. However, there are few annotated corpora that provide full paper texts. This paper presents the NLPSharedTasks corpus, a new resource of 254 full text Shared Task Overview papers in NLP domains with annotated task descriptions. We calculated strict and relaxed inter-annotator agreement scores, achieving Cohen’s kappa coefficients of 0.44 and 0.95, respectively. Lastly, we performed a sentence classification task over the dataset, in order to generate a neural baseline for future research and to provide an example of how to preprocess unbalanced datasets of full scientific texts. We achieved an F1 score of 0.75 using SciBERT, fine-tuned and tested on a rebalanced version of the dataset.

## introduction scholarly document processing (sdp) research is concerned with developing methods for improving the retrieval and organization of information from academic papers. this interest is partly driven by the rapid growth rate of scientific publications, which @xcite estimate to be between 2.7 and 13.5 percent between 1997 and 2006. some disciplines are expanding even more rapidly. @xcite examined the global output of machine learning research between 2009 and 2018 and estimated a growth rate of roughly 28 percent per year, while @xcite suggest an average annual growth rate of 152.9 percent in the deep learning domain between 2013 and 2019. because of the rapid expansion of scientific literature, it is beneficial to use natural language processing (nlp) and information extraction (ie) techniques to structure scientific and bibliometric data into machine-actionable forms. one method is to automatically identify scientific and bibliometric entites and relations from scholarly literature and organize them into knowledge graphs, which can be used to improve access to scholarly documents by enhancing digital libraries @xcite . one scientific entity type relevant to nlp and machine learning domains is task. machine learning and nlp tasks can be useful to extract, as they are a unit of information relevant to understanding research trends and constructing leaderboards @xcite . we are particularly interested in the utility of augmenting scholarly digital library resources with automatically extracted task descriptions such that a reader could quickly understand the nlp task described in the paper at hand. we find that nlp shared task workshop overview papers are a rich resource for training a model to extract such task descriptions. our contribution towards information extraction (ie) from scientific articles is a new gold-standard corpus of task description annotations from shared task overview papers. this corpus provides an interesting ie situation for two reasons. first, the full texts are provided for each paper in the corpus rather than individual sentences or paragraphs. second, the annotation goal was to extract a single span of text from each paper rather than any number of qualifying phrases. the benefit of this kind of annotation strategy is that it provides test data that is close to the "real world" data that downstream applications might encounter, such as a digital library tool tasked with extracting the task descriptions from nlp papers. this ie scenario is also difficult, since extracting a single span from full paper texts results in an extremely unbalanced dataset. for this reason, we describe in detail the data preparation and preprocessing steps we performed for the sentence classification task we ran over the nlpsharedtasks corpus. the original nlpsharedtasks corpus, preprocessed dataset, and experimental code is available at https://github.com/anmartin94/ martin-masters-thesis-2022 . ## related work numerous corpora for scientific information extraction have been hand-annotated by experts in computational linguistics and nlp domains. many of these corpora provide parts of scientific papers, such as paragraphs @xcite , abstracts @xcite @xcite , and sentences @xcite . the semeval-2021 task 11 (nlp contribution graph) @xcite provided the corpus that serves as the main source of inspiration for our annotation project. the nlpcontributiongraph corpus comprises 442 scholarly papers in nlp domains, with 12 different types of information annotated at three levels of granularity (d'souza and auer, 2020). it is similar to our work in that full paper texts and sentence-level annotations are provided, but the annotation scheme allowed for multiple spans to be extracted for each entity type, rather than a single sequence from each paper. additionally, the nlpcontributiongraph annotation scheme includes a tasks information unit, which was applied to 277 triples found across approximately 69 sentences in eight papers. the differences between the nlpcontribution-graph and nlpsharedtask annotation schemes relate to the different intended downstream tasks. the information extracted by d'souza et al., 2021 is designed to populate a research knowledge graph with a variety of types of scientific information, while the information extracted in nlpshared-tasks is intended to convey to a human reader the task described in the shared task overview paper. ## corpus selection the resource we drew from was the annual research workshop semeval and similar initiatives. these venues host shared tasks that approach a wide variety of semantic problems and provide a rich resource for understanding the state of the art in semantic analysis. we assembled our task description corpus by searching the acl anthology for shared task description papers, including all se-meval task description papers from the year 2001 to 2021, all conll foot_0 shared tasks 2000-2020, and selected shared tasks from a variety of other conferences and workshops (see additionally, this second batch of papers contained the newly published set of papers from semeval 2021. the final dataset contains a total of 254 shared task description papers between the years 2000 and 2021 and encompasses twenty natural language processing research topics that we identified (see figure 1 ). ## annotation methodology the aim of this annotation project was to develop a gold standard corpus of shared task overview papers with annotations of shared task descriptions. we define "shared task description" as a span of text containing information on an nlp or computational linguistics task to be performed by participating systems. this information must describe in brief what is to be done to accomplish the task, and may also contain details on the dataset the task is performed over. strict scores were calculated by comparing the exact spans of text. the relaxed scores were calculated by including the full sentence(s) containing the span. the difference between rows 2 and 3 is due to guideline revisions. the annotators often chose sequences that overlapped but were not exactly the same, resulting in the difference between columns 2 and 3. the first annotator extracted a task description sequence foot_1 from every paper in the corpus, generated a set of guidelines for the second annotator to follow, and created two representative sets of twenty papers each. intra-annotator agreement was determined using cohen's kappa coefficient. a strict score and a relaxed score were calculated for each dataset, where the strict score compared the exact sequence spans and the relaxed score counted overlapping annotations as matches. after the first subset was annotated by the second annotator, the guidelines were refined by the first annotator to address ambiguities before releasing the second set to annotator 2. ## annotation results we calculated the inter-annotator agreement between annotator 1 and annotator 2 using cohen's kappa coefficient @xcite . the strict cohen's kappa coefficient for the first subset was 0.383, and the relaxed cohen's kappa coefficient was 0.6401, indicating fair to substantial agreement @xcite . after we made revisions and clarifications to the annotation guidelines, we annotated the second subset, and achieved a strict score of 0.4373 and a relaxed score of 0.9488, indicating moderate to almost perfect agreement. the difference between the strict and relaxed scores indicates that, though the annotators often spans from the same sentence context, mutually choosing equivalent sequences is somewhat difficult. for example, from the following sentence the 2020 iteration of our task is similar to @xcite @xcite in that participants are required to design a model that learns to generate inflected forms from a lemma and a set of morphosyntactic features that derive the desired target form. ## corpus statistics one benefit of annotating shared task overview papers published over a long period of time is that this resource could potentially be used to study nlp research progress and trends. for this reason, we provide some basic statistics on the content of the papers included (see section 6.1). we also provide data on the extracted task descriptions in section 6.2, as such information may be useful to others for building task description extraction systems. ## sentence classification experiments despite the fact that task descriptions are defined as sequences that can be longer or shorter than a single sentence, we designed a sentence classification task because we achieved much higher inter-annotator agreement scores when we compared the chosen sentences spanned by the sequences rather than the exact sequences (see section 5). we fine-tuned the cased and uncased base versions of bert @xcite and scibert @xcite on every hyperparameter combination in ## conclusion our primary contribution is the creation of a new scholarly document processing corpus that provides full paper texts rather than short, curated contexts, and a method for reducing and rebalancing the dataset for an information extraction task. corpora such as nlpsharedtasks can be used in scholarly information extraction systems to automatically identify and display fine grained scientific information to users of digital libraries. our most significant finding is the importance of the data preparation and preprocessing decisions. these choices about how to build and filter the datasets had a much greater impact on the results than the hyperparameter settings. a future annotation project could be conducted that is generally based on our rules but is more lenient in terms of the sentences to be extracted. instead of focusing on conciseness, this project would prioritize obtaining as much information as is required to produce a more thorough account of the shared task. this resource might subsequently be utilized as the basis for an extractive task summary effort. a span-based information extraction task could be designed over our corpus to extract the original annotated sequences rather than full sentences. sentence classification could be used as a preprocessing step to narrow down the search space.