TDMS ci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics (2021)

Abstract: Tasks, Datasets and Evaluation Metrics are important concepts for understanding experimental scientific papers. However, previous work on information extraction for scientific literature mainly focuses on the abstracts only, and does not treat datasets as a separate type of entity (Zadeh and Schumann, 2016; Luan et al., 2018). In this paper, we present a new corpus that contains domain expert annotations for Task (T), Dataset (D), Metric (M) entities 2,000 sentences extracted from NLP papers. We report experiment results on TDM extraction using a simple data augmentation strategy and apply our tagger to around 30,000 NLP papers from the ACL Anthology. The corpus is made publicly available to the community for fostering research on scientific publication summarization (Erera et al., 2019) and knowledge discovery.

## introduction the recent years have witnessed a significant growth in the number of scientific publications and benchmarks in many disciplines. as an example, in the year 2019 alone, more than 170k papers were submitted to the pre-print repository arxiv foot_0 and among them, close to 10k papers were classified as nlp papers (i.e., cs.cl). each empirical field of science, including nlp, will benefit from the massive increase in studies, benchmarks, and evaluations, as they can provide ingredients for novel scientific advancements. however, researchers may struggle to keep track of all studies published in a particular field, resulting in duplication of research, comparisons with old or outdated benchmarks, and lack of progress. in order to tackle this problem, recently there have been a few manual efforts to summarize the stateof-the-art on selected subfields of nlp in the form of leaderboards that extract tasks, datasets, metrics and results from papers, such as nlp-progress foot_1 or paperswithcode. 3 but these manual efforts are not sustainable over time for all nlp tasks. over the past few years, several studies and shared tasks have begun to tackle the task of entity extraction from scientific papers. @xcite formalized a task to identify three types of entities (i.e., task, process, material) in scientific publications (semeval 2017 task10). @xcite presented a task (semeval 2018 task 7) on semantic relation extraction from nlp papers. they provided a dataset of 350 abstracts and reuse the entity annotations from zadeh and @xcite . @xcite released a corpus containing 500 abstracts with six types of entity annotations. however, these corpora do not treat dataset as a separate type of entity and most of them focus on the abstracts only. in a previous study, we developed an ie system to extract {task, dataset, metric} triples from nlp papers based on a small, manually created task/dataset/metric (tdm) taxonomy @xcite . in practice, we found that a tdm knowledge base is required to extract tdm information and build nlp leaderboards for a wide range of nlp papers. this can help researchers quickly understand related literature for a particular task, or to perform comparable experiments. as a first step to build such a tdm knowledge base for the nlp domain, in this paper we present a specialized english corpus containing 2,000 sentences taken from the full text of nlp papers which have been annotated by domain experts for three main concepts: task (t), dataset (d) and metric (m). based on this corpus, we develop a tdm tagger using a novel data augmentation technique. in addition, we apply this tagger to around 30,000 nlp papers from the acl anthology and demonstrate its value to construct an nlp tdm knowledge graph. we release our corpus at https:// github.com/ibm/science-result-extractor. on augenstein et al. (2017) and gábor et al. (2018), luan et al. (2018) created scierc, a dataset containing 500 scientific abstracts with annotations for six types of entities and relations between them. both semeval 2017-task 10 and scierc do not treat "dataset" as a separate entity type. instead, their "material" category comprises a much larger set of resource types, including tools, knowledge resources, bilingual dictionaries, as well as datasets. in our work, we focus on "datasets" entities that researchers use to evaluate their approaches because dataset is one of the three core elements to construct leaderboards for nlp papers. ## related work concurrent to our work, @xcite develop a new corpus scirex which contains 438 papers on different domains from paperswithcode. it includes annotations for four types of entities (i.e., task, dataset, metric, method) and the relations between them. the initial annotations were carried out automatically using distant signals from paperswithcode. later human annotators performed necessary corrections to generate the final dataset. scirex is the closest to our corpus in terms of entity annotations. in our work, we focus on tdm entities which reflect the collectively shared views in the nlp community and our corpus is annotated by five experts who all have 5-10 years nlp research experiences. ## a tdm entity tagger our final corpus tdmsci contains 2,000 sentences with 2,937 mentions of three entity types. we convert the original brat annotations to the standard conll format using bio scheme. 7 we develop a tagger to extract tdm entities based on this corpus. ## an initial tdm knowledge graph in this section, we apply the flair-tdm tagger to around 30,000 nlp papers from acl anthology to build an initial tdm knowledge graph. we downloaded all nlp papers from the acl anthology 9 covering the period of 1974-2019. for each paper, we collect sentences from the title, the abstract/introduction/dataset/corpus/experiment sections, as well as from the table captions. we then apply the flair-tdm tagger to these sentences. based on the tagger results, we build an initial graph g using the following steps: 9 https://www.aclweb.org/anthology/ by applying the above simple process, we get a noisy tdm knowledge graph containing 180k nodes and 270k links. after checking a few dense areas, we find that our graph encodes valid knowledge about nlp task/dataset/metric. figure 1 shows that in our graph, the task "srl" (semantic role labelling) is connected to a few datasets such as "framenet", "propbank", and "nombank" that are standard benchmark datasets for this task. based on the tagged acl anthology and this initial noisy graph, we are exploring various methods to build a large-scale nlp tdm knowledge graph and to evaluate its accuracy/coverage in an ongoing work. ## conclusion in this paper, we have presented a new corpus (tdmsci) annotated for three important concepts (task/dataset/metric) that are necessary for extracting the essential information from an nlp paper. based on this corpus, we have developed a tdm tagger using a simple but effective data augmentation strategy. experiments on 30,000 nlp papers show that our corpus together with the tdm tagger can help to build tdm knowledge resources for the nlp domain. anonymous entities. do not annotate anonymous entities, which include anaphors. the following examples are anonymous entities: abbreviation. if both the full name and the abbreviation are present in the sentence, annotate the abbreviation with its corresponding full name together. for instance, we annotate "20-newsgroup (20ng)" as a dataset entity in example 2. factual entity. only annotate "factual, contentbearing" entities. task, dataset, and metric entities normally have specific names and their meanings are consistent across different papers. in example 3, "a high-coverage sense-annotated corpus" is not a factual entity.