KGP at S em E val-2021 Task 8: Leveraging Multi-Staged Language Models for Extracting Measurements, their Attributes and Relations (2021)

Abstract: SemEval-2021 Task 8: MeasEval aims at improving the machine understanding of measurements in scientific texts through a set of entity and semantic relation extraction sub-tasks on identifying quantity spans along with various attributes and relationships. This paper describes our system, consisting of a three-stage pipeline, that leverages pre-trained language models to extract the quantity spans in the text, followed by intelligent templates to identify units and modifiers. Finally, it identifies the quantity attributes and their relations using language models boosted with a feature re-using hierarchical architecture and multi-task learning. Our submission significantly outperforms the baseline, with the best model from the post-evaluation phase delivering more than 100% increase on F1 (Overall) from the baseline.

## introduction most scientific experiments are accompanied by relevant measurements, which help researchers to quantify their observations and qualitative arguments. measurements also play a pivotal role in summarizing large experiments, and provide a brief idea of the results obtained. it is customary for scientists to present their research in the form of scientific papers. nowadays, with thousands of papers being published digitally every year, it is extremely difficult to go through every single paper in order to get the desired data. the most popular electronic open-access repository of e-prints, arxiv, currently has 1,867,929 articles 1 . the sheer vastness of this number suggests just how important it is for us to automate the task of extracting measurementrelated information from research papers @xcite . a thorough understanding of the measurements not only requires the numerals, but also the context in which the quantities occur. moreover, the entities and the properties measured along with the qualifiers that condition the measurements are crucial for understanding the measurement. measeval @xcite is a semantic relation extraction task focused on obtaining 9 different entities pertaining to counts, measurements and qualifying attributes of these quantities in a collection of excerpts from research papers in english. figure 1 shows an example of a quantity along with its attributes and relations from this dataset. we propose a three-stage pipeline to address this task. the first stage uses a pre-trained bert model @xcite to detect quantity spans from sentences. receiving the detected spans as inputs, the second stage obtains the units and modifiers using extracted units and modifier keywords. finally, the third stage receives the quantity spans from the first stage and uses another pre-trained language model over each quantity-span-conditioned sentence to obtain quantity-span-aware contextualized representations for each sub-token in the sentence. these representations are then used to detect the measured entity corresponding to each quantity (if any). the predictions from the measured entity task are then fused with the individual representations for each sub-token. these representations are used to detect the measured property and the qualifiers in a multi-task learning setting @xcite . our submission surpassed the baseline by a significant margin and ranked 3 rd for the unit task. our current best model delivers 516.7%, 436.8%, and 296.4% f1 (overlap) @xcite gains for measured entity, measured property and qualifier tasks respectively, over the baseline. we showed that co-deposition of blended mixtures leads to 60% higher photocurrents than in thickness-optimized pth/c60 heterojunction counterparts @xcite . ## related works understanding and extracting information from scientific documents has been receiving increasing interest @xcite . extracting units of measurement from scientific documents was previously studied via regular expressions and supervised classifiers @xcite . in the orthogonal directional, there has been rapid progress in understanding natural language using deep pre-trained language models @xcite @xcite , which has lead to a general improvement across multiple tasks. the sequence labelling @xcite and span prediction @xcite tasks for natural language have also received great interest recently. we build upon these systems. ## problem statement we are given a set of documents @xmath0. every document d @xmath1, consists of various quantity (q) spans q @xmath2. every q j @xmath3, can have a unit of measurement (e.g. cm, ml) associated with it. also every q j @xmath4.g. whether it is a range of values, whether it denotes the median of a set of values, etc.) foot_0 . for every q j @xmath5, there can exist a corresponding measured entity (me) e j i . some qs do not have any me, e.g. in '3413 women', the measurement is 3413 and 'women' is the 'unit' of 3413 and not its me (according to "s0006322312001096-1177.tsv"). similarly in 'three occasions', the measurement is 'three' and 'occasions' is its 'unit' and not its me (according to "s0165587612003680-1078.tsv"). if a q j i has a corresponding me e j i , it can also have an associated measured property (mp) p j i . finally, the qs, mes and mps can have a number of qualifiers (qual) qual j i providing additional information about them. the problem also introduces three relations, namely qualifies (qs), hasproperty (hp), and hasquantity (hq). these relations are defined between qs, mes, mps and quals as binary classification functions (f (x, y) → (0, 1)): the problem statement consists of 5 sub-tasks. we deal with identifying all q spans in the documents in sub-task 1, followed by detecting the units and mods for each identified q in sub-task 2. in sub-tasks 3 and 4, we identify the me, mp, and qual spans, corresponding to the extracted qs. finally in sub-task 5, we identify the relationships hq, hp, and qs between the detected q, me, mp, and qual spans. figure 1 shows the annotation procedure to be followed @xcite . ## system overview we model all the previously described sub-tasks as supervised learning problems. firstly, we perform a minimal pre-processing of sentence segmentation and number normalization on the documents. then, stage 1 handles sub-task 1 and the stage 2 handles sub-tasks 2 respectively, and the remaining ones are handled by stage 3 of our pipeline. before proceeding to describe our approach, we describe the baseline model, provided by the task organizers. the baseline treats the detection of q, me, mp and qual spans all as sequence labeling problems. it uses the spacy entity tagger model @xcite to extract all these four spans. the units for these qs are obtained by stage 2 stage 1 stage 3 modifier: none unit: nm training data extracted units and keywords these observations were followed by moutou et al. (2003) who measured the transit depth in the he 1083 nm line that was predicted significant by seager et al. (2000). segment embedding token embedding bert tokenizer these [cls] ... the he in 108 ##3 nm line ... [sep] bert model dropout + linear ... nq nq nq q q q nq ... nq quantity span ... ... bert tokenizer these [cls] ... <@xmath6... [sep] bert model nm + + + + + + + + + + e1 e1 e0 e0 e 0 e1 e1 e1 e0 e0 ... ... ... ... dropout + linear ... ne e ... matching the largest units in these predicted spans with those from the train dataset. ## experiments and discussion all experiments were performed using pytorch @xcite and huggingface's transformers @xcite . optimization was done using adam @xcite . we include the complete set of experimental parameters in §d. ## conclusion in this paper, we present our system details for the semeval 2021 task-8: measeval which is aimed at extracting entity and semantic relations pertaining to counts and measurements. we use a multi-staged approach where we first identify the quantity spans using bert, then the units and modifiers for these predicted quantity spans by intelligent templates that leverage extracted units and modifier keywords. finally we input the quantity-aware sentences to another bert model to predict me, mp, and qual in a multi-task learning settings with feature re-use. our submission achieved the second runner up position on the leaderboard for the unit-identification sub-task and it showed the highest improvement in the post-evaluation phase, with an f1 (overall) score only 0.063 lower than the highest score across both the phases.