Semi-Automated Labeling of Requirement Datasets for Relation Extraction (2021)

Abstract: Creating datasets manually by human annotators is a laborious task that can lead to biased and inhomogeneous labels. We propose a flexible, semi-automatic framework for labeling data for relation extraction. Furthermore, we provide a dataset of preprocessed sentences from the requirements engineering domain, including a set of automatically created as well as hand-crafted labels. In our case study, we compare the human and automatic labels and show that there is a substantial overlap between both annotations.

## introduction while recent advances in natural language processing have yielded high-quality language models such as bert @xcite , @xcite and electra @xcite which are able to continue sentences, fill in masked words and correctly parse human language, using these models for most use-case scenarios still requires them to be trained on a down-stream task using labeled data. for some tasks, e.g. sentiment analysis of reviews, creating datasets is relatively easy as large databases with annotations already exist (such as the imdb movie review dataset @xcite ). however, training a model on niche tasks often demands hand-crafting new datasets from spread-out documents. this is usually done by humans who collect, preprocess, and annotate sentences which is a laborious task and can result in biased and/or inhomogeneous labeling, e.g. if annotation instructions were not understood correctly or left room for subjective interpretation. this becomes especially apparent if multiple, non-expert individuals are involved in this process. in requirements engineering, we usually work with large documents written in natural language @xcite which describe the specifications of a software project, usually classified as either functional requirements, specifying what functionality the system should provide, and non-functional requirements, specifying in what way the system should implement those functions. however, these documents are often updated during the life cycle of the project and span up to multiple hundreds of pages, depending on the project size. keeping track of all the changes and maintaining the software based on the requirement document can soon become a challenge @xcite which is why an automatic conversion to, e.g., uml diagrams can come in handy. to do so, it is necessary to parse the relations between entities from the written text into a structured format, thus creating a comparable corpus of requirements in natural language and the same relation in a formal language. in this paper, we propose a semi-automatic approach that, given a clean, grammatically correct sentence stating a software requirement, outputs a labeling corresponding to the relation the requirement describes based on a small set of pre-defined rules of word dependency relations. this should reduce human bias manifesting in labels as the annotator does not actively choose the labels for each word anymore but instead defines abstract rules which provide for homogeneous, deterministic labeling and reduce the amount of labor for creating such datasets. this automatically annotated data can then be used for training a more powerful model, as shown by @xcite . we summarize our main contributions as follows: ## evaluation given our automatically labeled data, we evaluate the quality of the labels by comparing its output to human annotations. to do so, we randomly sample 199 sentences (10.77%) from the 1848 sentences which were automatically labeled. two of the authors then annotated these sentences manually. the annotators were given the descriptions of each label type, but had no access to the actual labeling from the algorithm. annotators collaboratively labeled the data, discussing the labeling for each sentence and agreeing upon a single valid labeling. we then calculate inter-rater reliability with the cohen's κ between the human annotators and the automatic annotator, once over all labels and once as average inter-reliability per sentence (i.e., we calculate one cohen's κ score per sentence and average over all sentences -this considers each sentence equally while the overall score puts more weight on longer sentences). the results can be found in while one could split the cond label into multiple different labels, this would increase the number of patterns required a lot. alternatively, one might reduce the coverage of the labeling in general but we focused on including as much information as possible. the relatively low score for ent2 mainly arises from sentences containing multiple relations where many words describe a passive entity for other relations than the one of the main sentence. our approach currently is not able to effectively extract multiple relations from a single sentence yet. this is also the reason why the score rel is lower than the one for ent1. ## limitations while our approach works well for requirements documents -after all, relations between software entities and modifications of these relations can be extracted well by syntactically parsing the sentence structure -this does not apply to word labels which require a semantic understanding of the input. for example, if we were to create labels for named entity recognition, our algorithm would fail as ## conclusion & outlook in this paper, we present a novel approach for data labeling which allows users to annotate sentences for relation extraction within a shorter time period compared to manual annotation while at the same time having a consistent labeling scheme for the entire dataset. our approach exploits syntactic features which are the integral foundation of most relation extraction tasks. for the future, it would be helpful to implement an automatic extraction of requirement sentences by, e.g., training a classifier to identify relevant sen-tences in plain text or .pdf documents as well as a semi-automatic approach with human validation for preprocessing sentences into grammatically and orthographically sound ones. we plan on extending the pattern engine our algorithm relies on, e.g., allowing for recursive patterns to parse nested sentences and to extract multiple relations from one sentence as well as optional pattern parts to reduce redundancy (e.g., a sentence where the active entity is the nominal subject, the relation the dependency tree root and the passive entity the direct object may have a relation modifier in an adverbial clause. as of the current state, this requires two patterns (exponentially increasing with the number of optional dependencies) while with a pattern where this adverbial clause is considered optional, we only need a single pattern).