Reading between the Lines: Information Extraction from Industry Requirements (2023)

Abstract: Industry requirements describe the qualities that a project or a service must provide. Most requirements are, however, only available in natural language format and are embedded in textual documents. To be machine-understandable, a requirement needs to be represented in a logical format. We consider that a requirement consists of a scope, which is the requirement’s subject matter, a condition, which is any condition that must be fulfilled for the requirement to be relevant, and a demand, which is what is required. We introduce a novel task, the identification of the semantic components scope, condition, and demand in a requirement sentence, and establish baselines using sequence labelling and few-shot learning. One major challenge with this task is the implicit nature of the scope, often not stated in the sentence. By including document context information, we improved the average performance for scope detection. Our study provides insights into the difficulty of machine understanding of industry requirements and suggests strategies for addressing this challenge.

## introduction requirements are a critical part of the development process for products and services. they are documented descriptions of the physical or functional qualities that a product or a service must have. in industry, requirements serve as a means of communication between contractors and manufacturers, defining what is expected to be built and the quality standards to be met. governments and international organizations may also impose requirements to ensure compliance with rules, regulations and standards. requirements are included as part of the contract between two parties, making adherence to them a legal obligation. violating the requirements can lead to legal implications and financial losses, underscoring the importance of careful specification and adherence to requirements throughout the development process. a requirement is typically associated with a specific piece of equipment that needs to be built which is referred to as the scope of the requirement. however, a requirement may only be relevant if certain conditions are met, which will be referred to as condition. the demand of the requirement is a feature or quality the scope must possess. as an example, consider the requirement equipment with a weight of more than 1000 kg shall have a weight certificate. here, the scope is equipment, while the condition is with a weight of more than 1000 kg, and the demand is to have a weight certificate. most requirements are expressed as natural language text and are embedded in documents. these documents often have a hierarchical structure with chapters, sections, and other headings, which provide important context for understanding the requirements. when the number of requirements documented in this way increases, managing and maintaining these documents becomes a significant challenge. in addition, checking for consistency in a set of requirements and ensuring compliance of project descriptions with a set of requirements are time-consuming tasks that ideally should be automated. to overcome these challenges, computer systems could be used to automatically identify relevant requirements, check consistency and ensure compliance. this could be achieved by creating all new requirements in a machine-understandable format. however, the industry is often bound by existing requirements in their current form. therefore, extracting information from existing documents is essential for enabling automated systems. we have found that identifying the scope of a requirement can be particularly challenging since this information is often not explicitly stated in the sentence. context and additional information is often required to make accurate predictions. the document's title, section headings and domain knowledge can provide valuable context for identifying the scope of a requirement. recent studies have shown that adding contextual information can improve the performance of nlp tasks that typically focus on one sentence at a time. for instance, in named entity recognition (ner), @xcite used unstructured text as context, while @xcite incorporated image-based information. similarly, context has been found to be beneficial in relation extraction. e.g., @xcite improved performance using information from knowledge graphs. our work makes three key contributions. first, it introduces a novel task: identifying three semantic components scope, condition, and demand of a requirement sentence. second, it establishes baselines for this task. third, it investigates the extent to which including context information from the document can improve the quality of identifying these semantic components. ## related work related work can be grouped into information extraction from requirements or legal text, and using context information to improve sequence labelling. ## evaluation to assess the generalization capabilities of our approaches across different domains and evaluate their performance on unseen documents, we extracted and labelled an additional 400 sentences extracted from four other documents. two documents were selected from the same domain (high speed and light craft), while the other two were from different domains: floating fish farming units and installation and drilling facilities. consistent with our previous datasets, we labelled the sentences and created two versions of each dataset: one with context (os-e101 c , ru-hslc-pt5 c , ru-hslc-pt6 c , ou-0503 c ), and one without context (os-e101, ru-hslc-pt5, ru-hslc-pt6, ou-0503). this differentiation is necessary because the model trained with context expects an input format where the sentence has headers and noun chunks appended, while the model trained without context expects the sentence only. to evaluate the performance of the models on scope, condition, and demand detection, we annotated each sentence in the corpus with the textual representation of scope, condition, and demand. we used the original documents as a guide to ensure accurate annotations. the gold scope comprises a set of normalized noun chunks, which we combined into a set as described in section 4. note that the gold scope, condition, and demand are the same both for the dataset with context and the dataset without context. we evaluate the performance using three different metrics used in the text generation literature: rouge-l @xcite , bleu unigrams @xcite , and a language-model-based measure (lbm). to use similarity measures from text generation literature, we created a single string from the sets of extracted scope, condition, and demand spans by joining items with the word "and." some of the metrics do not handle empty strings, therefore, if the predicted string or the gold string is empty, we replace it with a dummy string "empty." we then compare the predicted and gold strings using the respective similarity scores. for the lbm metric, we utilized sentence embeddings generated by the all-roberta-large-v1 model from the sentence-transformers library @xcite . the embedding captures the semantic meaning of sentences and enables us to estimate the semantic similarity between predicted and gold strings. the lbm score is the cosine similarity between the predicted and gold strings. ## discussion the sequence labelling model achieves a rouge-l f1 score of 0.45 on scope detection without access to context information. considering that only half of the sentences in the train dataset have scope labels, the performance is promising. gpt-3, being much larger, is able to leverage information learned during pre-training and generate scopes that are not explicitly mentioned in the text, allowing improved performance compared to the sequence labelling model. the sequence labelling approach demonstrates strong performance in the detection of the condition and demand. the results suggest that with a larger training corpus, the accuracy is likely to be suitable for practical applications. the challenge with scope detection lies in the need to infer implicit scopes by "reading between the lines." to address this challenge, our study proposes the explicit inclusion of context information to enhance performance. by incorporating document context, for most sentences, we have scope labels. either, the labels come from the sentence itself or from the context, as seen in on scope detection with context, we observe a general improvement over the results without context. despite the increased number of sentences with scope labels in the training data, the improvement achieved by the models does not align proportionally. in particular, gpt-3 does not effectively leverage context information. it is possible that presenting the examples differently or refining the prompt could lead to improved results. the observed improvement with context information is most prominent on the test data and on ou-0503. performance improvements on test data may in part be explained by the model learning relevant terms used in headers in the documents used for training. however, the improvement on ou-0503 demonstrates that the model is still able to generalize, and not only memorise scope labels. including context from the document did not result in improvements for condition and demand detection. the difference between the experiments with and without context is consistently small. ## conclusion and future work we have introduced the novel task of identifying the semantic components scope, condition, and demand in a requirement sentence. we have established baselines by casting the task as a sequence labelling problem and a few-shot learning problem. we have also highlighted the particular challenge of identifying the scope which is often not explicitly given and proposed including context information explicitly to improve scope detection. including context information in the text is helpful for identifying the scope of a requirement sentence in all the requirements documents in this experiment. in addition, this work establishes that the detection of scope is very different from the detection of a condition and the demand, and that different approaches work differently for scope detection than for condition and demand detection. it may thus be useful to consider them as different tasks, requiring different tools and strategies. in future work one could i) investigate when adding context helps, ii) investigate what kind of context helps, or iii) investigate other types of context information and how to present the context information to a language model. furthermore, iv) matching the scopes to concepts in a knowledge graph would be interesting as thereby it could be possible to resolve similar textual representations of the same ontological concept. finally, v) more research is needed to see if our results also apply to requirements from other sources.