UPB at S em E val-2021 Task 8: Extracting Semantic Information on Measurements as Multi-Turn Question Answering (2021)

Abstract: Extracting semantic information on measurements and counts is an important topic in terms of analyzing scientific discourses. The 8th task of SemEval-2021: Counts and Measurements (MeasEval) aimed to boost research in this direction by providing a new dataset on which participants train their models to extract meaningful information on measurements from scientific texts. The competition is composed of five subtasks that build on top of each other: (1) quantity span identification, (2) unit extraction from the identified quantities and their value modifier classification, (3) span identification for measured entities and measured properties, (4) qualifier span identification, and (5) relation extraction between the identified quantities, measured entities, measured properties, and qualifiers. We approached these challenges by first identifying the quantities, extracting their units of measurement, classifying them with corresponding modifiers, and afterwards using them to jointly solve the last three subtasks in a multi-turn question answering manner. Our best performing model obtained an overlapping F1-score of 36.91% on the test set.

## introduction our world revolves around quantities and units of measurement present in all texts, ranging from scientific texts to recipes. nevertheless, the process of automatically extracting measurements is not trivial, considering that, in most situations, the quantitative structures are ambiguous and are not present in the same area within the text. therefore, parsing the semantic relations becomes a ubiquitous task, since proper quantity identification leads to transformations towards an easy to follow quantitative summary. advantages of the previously mentioned process can be found in medical prescriptions @xcite . as such, a system that can robustly and confidently identify medication quantities, measurement units, as well as the medication itself has the potential to become a breakthrough for computer-based medicine and consultations. another use case resides in erp systems where proper parsing of resource descriptions facilitates the identification of similar or duplicate items. the measeval -counts and measurements competition @xcite organized by the 15th international workshop on semantic evaluation (semeval-2021) creates a new challenge in the area of natural language processing, proposing five subtasks related to span identification, classification, as well as relation extraction, that aim to improve the state of the art for the current field of measurement information extraction. we created a cascaded system to solve the stated problem that is composed of: (1) a subsystem that identifies quantities in the input text; (2) a subsystem that classifies their value modifiers; (3) a subsystem that extracts their measurement unit; and (4) a subsystem that then finds the appropriate measured entities, measured properties, and qualifiers by asking questions related to entity-relations. three pretrained transformer-based @xcite language models are experimented for subsystems (1) and (4) of the cascaded system by fine-tuning them on the specific task: bidirectional encoder representations from transformers (bert) @xcite , robustly optimized bert pretraining approach (roberta) @xcite , and science bert (scibert) @xcite . a character-level bidirectional long short-term memory (bilstm) @xcite architecture was considered for subsystems (2) and ( 3 ). the rest of the paper is structured as follows. the next section presents a series of solutions associated with relation extraction, span identification, and measurement unit identification. the third section outlines our approaches related to the subtasks proposed by the competition. the fourth section presents a performance evaluation of our systems together with an error analysis, while the final section concludes our work and outlines potential future improvements. measurement unit identification. @xcite proposed a two-step system for search space size reduction, followed by unit extraction from the previously obtained textual fragments. also, hundman and mattmann (2017) presented a hybrid system composed of a crf that identifies quantities values and their units, followed by a rule-based model to detect their corresponding entities. relation extraction. zhang and wang (2015) adopted a model based on recurrent neural networks (rnn) @xcite composed of three main elements: an embedding layer, a bidirectional recurrent layer, followed by a max pooling layer that produces the feature vector used for relation classification. rnn-based models were also applied by @xcite who adopted bil-stms, or by @xcite who proposed an architecture based on hierarchical rnns alongside an attention mechanism. furthermore, several convolutional neural network-based models with various approaches were proposed, for example: multi-level attention @xcite , attentionbased context vectors @xcite , or multi-level features (word, lexical, sentence) @xcite . bilstms are also present in the work of @xcite who implemented a mechanism based on entity-aware attention using latent entity typing. @xcite approached the relation extraction task by employing a graph neural network system that modeled each relation as a node and learned the dependencies between the nodes. ## method our approach on measeval consisted of a cascade system composed of individual subsystems for each of the problems in the first two subtasks, and then jointly solving the last three subtasks with a single subsystem. ## conclusions and future work this paper introduces our approach that solves all the five subtasks of the 8th task of semeval-2021 competition in a cascaded manner. first, quantities are identified as a sequence tagging task by using a pretrained language model with a crf layer. then, the measurement units are extracted and the modifiers are classified using bilstms at character level on the identified quantities. finally, the measured entities, measured properties, and qualifiers are jointly identified, together with their relations, by using a multi-turn question answering approach with hand-crafted questions specific to each relation type. our best model obtained an f1-score of 36.91% on the test set. we further emphasized several limitations of our approach and showed that the overall performance was highly sensitive to the quality of the identified quantities. a possible direction for future work is to test the system using language models that can process longer sequences, such as longformer @xcite or bigbird @xcite , in order to reduce the effect of missing entities simply due to the sequence length. we also consider creating an ensemble model using several pretrained language models to boost the overall performance, as reported by @xcite .