Benchmark for Research Theme Classification of Scholarly Documents (2022)

Abstract: We present a new gold-standard dataset and a benchmark for the Research Theme Identification task, a sub-task of the Scholarly Knowledge Graph Generation shared task, at the 3rd Workshop on Scholarly Document Processing. The objective of the shared task was to label given research papers with research themes from a total of 36 themes. The benchmark was compiled using data drawn from the largest overall assessment of university research output ever undertaken globally (the Research Excellence Framework - 2014). We provide a performance comparison of a transformer-based ensemble, which obtains multiple predictions for a research paper, given its multiple textual fields (e.g. title, abstract, reference), with traditional machine learning models. The ensemble involves enriching the initial data with additional information from open-access digital libraries and Argumentative Zoning techniques (CITATION). It uses a weighted sum aggregation for the multiple predictions to obtain a final single prediction for the given research paper. Both data and the ensemble are publicly available on https://www.kaggle.com/competitions/sdp2022-scholarly-knowledge-graph-generation/data?select=task1_test_no_label.csv and https://github.com/ProjectDoSSIER/sdp2022 , respectively.

## introduction with the recent demise of the widely used microsoft academic graph (mag) @xcite , the scholarly document processing community is facing a pressing need to replace mag with an open-source community-supported service. in order to create a comprehensive scholarly graph, it is challenging to correctly represent each paper as a node on the graph. this requires condensing meta-information, such as authorship, research organizations, research themes etc., of research papers to one node. so far, the task of identifying research themes for a given scholarly document has been challenging due to the lack of large high-quality labelled data. this made it difficult both to train highperformance classification models as well as to compare models' performance across studies. this paper provides a benchmark for research theme classification based on a large humanannotated corpus of scholarly papers across 36 themes defined by the uk research excellence framework, the largest overall assessment of university research outputs ever undertaken globally (the research excellence framework -2014) foot_0 @xcite . the outcome of this paper is the product of the scholarly knowledge graph generation shared task which was part of the scholarly document processing (sdp) workshop at coling2022. we started with a labelled dataset containing publications and subjects to which they belong (section 3), which contains descriptions or abstracts, the first author, doi, year of publication, and identifier to link the publication to the core @xcite aggregator. we later enriched this dataset with further information including the full text, where available. this represents a new gold-standard dataset for theme classification of scholarly documents. to establish a benchmark for research theme classification, we present experiments and evaluation results with traditional machine learning models and compare them to a more sophisticated transformer-based ensemble model. our transformer-based ensemble model exploits all textual fields for each scholarly document and maps these documents to core and semantic scholar @xcite to gather further external information. thus, the ensemble consists of a transformer-based classifier used to produce multiple predictions for individual publications (split into multiple textual fields) that are aggregated to produce a single final prediction. we aggregate predictions from titles, abstracts, references, citations, and related titles for every publication, when available. furthermore, we use abstracts, pdfs and full texts available to identify argumentative zones @xcite to use them as additional fields. we report on the results of using aggregation for different combinations of these predictions. the rest of the paper is organized as follows: section 2 presents a discussion of the related work, focusing mainly on scientific document classification approaches and their evaluation. section 3 describes details of building the new benchmark for theme classification. section 4 discusses the ensemble we propose as a baseline and the system components in more detail. in section 5, we describe the experimental settings. in section 6, we discuss the evaluation results from a diverse set of experiments. finally, we discuss the conclusion and the potential direction of future work in sections 7 and 8. ## related work classifying scholarly documents is an important task, whether for understanding the dynamics of scientific fields or simply for organizing scientific literature more effectively. in previous literature, it typically relies on textual features such as titles, author keywords, and abstracts, as well as the interrelationships between the documents (i.e., citations and co-authorship). full texts are frequently not available and processing a large amount of text can be computationally expensive. a wide variety of classification features have been proposed at different levels of granularity, e.g., themes, topics, and subjects. a large proportion of classification methods rely on semantic similarity @xcite @xcite @xcite . others include approaches for clustering documents based on keyword co-occurrence @xcite . further approaches leverage the relationship graph representation built from ci-tations and co-authorship @xcite @xcite . one promising but unexplored approach to theme classification is using information about argumentative zoning (az) @xcite . az refers to the examination of the argumentative status of sentences in scientific articles and their assignment to specific argumentative zones. its main goal is to collect sentences that belong to predefined zones, such as "claim" or "method". annotated az corpora has been created by @xcite @xcite with approaches to az identification reported in @xcite . in this work, we aim to test to what extent can the az signal support classification of scholarly documents into research themes. classification models previously appplied to this task include traditional machine learning models, such as k-nearest neighbours @xcite , k-means @xcite and naïve bayes @xcite . it has been reported that these models encounter performance challenges related to overly coarse classifications and low accuracy @xcite . there are applications of deep neural networks (nn) models as well, such as convolutional nn @xcite and recurrent nn @xcite . more recent deep learning approaches take advantage of pretrained language models @xcite . one of the common practices to evaluate approaches for classifying scientific text is to use classification systems from digital libraries @xcite @xcite , such as the acm computing classification system 2 , the web of science categories 3 and science-metrix 4 . other practices involve generating automatic annotations for scientific collections that can be completely synthetic @xcite or curated by experts @xcite @xcite @xcite . however, to date, there has been no established benchmark to evaluate these approaches. ## inital dataset creation as previously discussed, one of the significant challenges faced in the domain is the lack of largescale labelled data for research theme classification. for the shared task, a completely new goldstandard dataset was compiled using data drawn from the u.k.'s research excellence framework (ref) 2014 exercise @xcite . in total, 191,000 research outputs were submitted by 154 higher education and research institutions, and these were then peer-reviewed by experts from each domain. the ref divided research outputs into 36 'units of assessment' (uoa) or domain areas. the institutions themselves selected to which unit of assessment each output was submitted. clinical medicine business and management studies allied health professions, dentistry, nursing and pharmacy psychology, psychiatry and neuroscience biological sciences general engineering computer science and informatics mathematical sciences physics earth systems and environmental sciences chemistry public health, health services and primary care geography, environmental studies and archaeology agriculture, veterinary and food science education aeronautical, mechanical, chemical and manufacturing engineering electrical and electronic engineering, metallurgy and materials social work and social policy politics and international studies architecture, built environment and planning economics and econometrics sport and exercise sciences, leisure and tourism history law sociology english language and literature communication, cultural and media studies, library and information management modern languages and linguistics civil and construction engineering anthropology and development studies philosophy art and design: history, practice and theory music, drama, dance and performing arts area studies theology and religious studies classics research theme 0 1000 2000 3000 4000 5000 number of papers the data from the ref exercise, therefore, provides a near-perfect starting point for the task of automatically identifying research themes as the uoa labels were manually assigned to each output by the expert academics responsible for its production. for each output, the following were available from the ref data; publication title, publication year, publication venue, name of institution, and unit of assessment. these fields were fully populated for 190,628 out of 190,963 submissions to the outputs category of the ref process. we further enriched each record with the doi, core id, and abstract (where available). the core id is used to identify the actual research article held by the core service foot_1 . not all papers in the dataset are open access, therefore the full-text content of all papers is not available. for non-open access papers, core often still has the metadata for these articles. for the data used in this shared task, separate test and train datasets were generated. from the full ref dataset, 51,560 randomly selected records were used for the train set, and a separate 10,000 were selected for the test set. the datasets were then verified to ensure that there was no overlap between the two sets. figure 1 shows the crossdomain (theme) breakdown of all records used for this task. ## classification ensemble this section depicts the approach we used to estimate probabilities of academic publications belonging to specific theme and the heuristics we follow for classification. in general, we want to exploit all the information available for the scholarly documents that need to be classified. academic publications are typically well-structured documents with multiple textual fields and metadata. we rely on open-access platforms to enrich the data with additional information (section 4.2). currently, transformer-based contextual language models like elmo @xcite or bert @xcite outperform most feature-based representation methods. we use a classifier based on contextual word embeddings to evaluate the utility of individual textual fields in the classification of academic publications. ## discussion in this work, we first released a new gold-standard human-annotated dataset of over 60k papers com- second, we use this dataset to establish a new benchmark for research theme classification, testing a range of classic machine learning models under the same laboratory conditions. unsurprisingly, our results confirm that models trained with both titles and abstracts as input features consistently achieve higher results than when using titles alone. these results hold both for baseline models and our newly introduced ensemble bert model. while the results confirm that the bert-based ensemble model outperforms traditional models, the performance of svms is only marginally worse. it is interesting to note that using all available features for training (run 5) decreases the score compared to the model trained on titles and abstracts only. we hypothesise that a large proportion of false negatives can be attributed to noise introduced by reference sections within the full texts, especially for closely aligned domains. the confusion matrix (figure 3 ) shows that many of the incorrect classifications happened in closely related domains (clinical medicine / biological science for example). this is indicative of the difficulty of this task, particularly when presented with closely matched or overlapping domains. indeed, one limitation of our approach may be the classification of each paper into a single research field. in real-world examples, a paper could often be classified into multiple domains. another limitation is that our ensemble model requires the availability of both title and abstract, which are necessary for the az approach, which we have seen contributes to the performance. assigning research themes to scholarly documents has wide-ranging applications. these include enhanced domain-specific search, for in-stance search in chemistry is a complex task due to the need to index chemical compounds, and identifying emerging research trends. further, a significant problem with current bibliometric methodologies is accounting for cross-disciplinary differences in both publishing and citation practices. identifying the research theme enables accounting for disciplinary differences by, for instance, calculating normalised citation counts. in future work, we would like to measure the importance of weight assignments for augmented predictions and consider the overlap between disci-plines to evaluate ways of disambiguating predictions falling into related themes. ## conclusion we have introduced a new large human annotated gold-standard dataset and a benchmark for research theme classification of scholarly documents. the work was conducted in the context of the extracting research themes task from the 2022 edition of the scholarly knowledge graph generation shared task. the task was to identify the main research theme from a taxonomy of 36 classes, introduced by the uk research excellence framework. our experiments addressed the effect of using a variety of textual fields on the prediction performance. enriching the supplied training and testing data with external textual information (e.g., pdf source, full-text article, references) using openaccess sources improved the results of our models. however, we have demonstrated that this enrichment might also introduce additional noise. we presented a new transformer-based classifier model based on bert and used it to obtain multiple predictions for a given research article for each textual field. we experimented with a variety of aggregation functions to produce the final prediction. despite incomplete and noisy data, the results show that our ensemble model has a small positive impact on the classification performance.