M u LMS - AZ : An Argumentative Zoning Dataset for the Materials Science Domain (2023)

Abstract: Scientific publications follow conventionalized rhetorical structures. Classifying the Argumentative Zone (AZ), e.g., identifying whether a sentence states a Motivation, a Result or Background information, has been proposed to improve processing of scholarly documents. In this work, we adapt and extend this idea to the domain of materials science research. We present and release a new dataset of 50 manually annotated research articles. The dataset spans seven sub-topics and is annotated with a materials-science focused multi-label annotation scheme for AZ. We detail corpus statistics and demonstrate high inter-annotator agreement. Our computational experiments show that using domain-specific pre-trained transformer-based text encoders is key to high classification performance. We also find that AZ categories from existing datasets in other domains are transferable to varying degrees.

## introduction in academic writing, it is custom to adhere to a rhetorical argumentation structure to convince readers of the relevance of the work to the field @xcite . for example, authors typically first indicate a gap in prior work before stating the goal of their own research. argumentative zoning (az) is a natural language processing (nlp) task in which sentences are classified according to their argumentative roles with varying granularity @xcite . az information can then be used for summarization @xcite , improved citation indexing @xcite , or writing assistance @xcite . manually annotated az datasets @xcite @xcite only exist for few domains and employ differing annotation schemes. the resulting models are not directly applicable to our domain of interest, materials science research, which presents a challenging domain for current nlp methods (e.g., @xcite @xcite . in this paper, we present mulms-az, the first dataset annotated for az in this domain. working together with domain experts, we derive a hierarchical multi-label annotation scheme (see this resource paper makes the following contributions: in this section, we describe related work on az. ## data sources and annotated corpus in this section, we present our new dataset. ## modeling we model az as a multi-label classification problem, using bert @xcite as the underlying text encoder. we also test domain-specific pre-trained variants of bert. scibert @xcite has been pre-trained on articles in the scientific domain. @xcite is a version of scibert further pre-trained on materials science articles. we use the cls embedding as input to a linear layer, transform logits using a sigmoid function and choose labels if their respective score exceeds 0.5. for multi-task experiments with other datasets, we use a single shared language model and one linear output layer per dataset. for hyperparameters, see appendix a. as shown in ## experimental results we here detail our experimental results. settings. we split our corpus into train, dev, and test sets of 36, 7, and 7 documents. for all experiments and for hyperparameter tuning, we always train five models. the training data is split into five folds. similar to cross-validation, we train on four folds and use the fifth fold for model selection (cf. van der goot, 2021), repeating this process five times (also for hyperparameter tuning). the dev set is only used for tuning, and we report scores for the five models on test. in this setting, deviations are naturally higher than when reporting results for the same training data. for hyperparameters and implementation details, see appendix a. to evaluate our experiments, we use hierarchical @xcite . these scores operate on the sets of labels per instance, always including the respective supertypes of gold or predicted labels. results. as a first step to explaining what part of rhetoriusing these mapped labels, we train binary classifiers that aim to detect the presence of a particular pm label. as training data, we use art, dri, and a selection of documents from the pm dataset by de moura and feltrim (2018) that were published in materials science journals (see appendix c). we add az-cl to the training data only if an unambiguous mapping of its categories to the pm label in question is possible. here, we use the dev set of mulms-az for model selection and hyperparameter tuning. results for running the resulting classifiers on mulms-az are reported in ## conclusion and outlook we have presented a new az corpus in the field of materials science annotated by domain experts with high agreement. our experimental results demonstrate that strong classifiers can be learned on the data and that az labels can be transferred from related datasets only to a limited extent. our new dataset opens up new research opportunities on cross-domain az, class imbalance scenarios, and integrating az information in information extraction tasks in materials science.