S em E val-2021 Task 8: M eas E val – Extracting Counts and Measurements and their Related Contexts (2021)

Abstract: We describe MeasEval, a SemEval task of extracting counts, measurements, and related context from scientific documents, which is of significant importance to the creation of Knowledge Graphs that distill information from the scientific literature. This is a new task in 2021, for which over 75 submissions from 25 participants were received. We expect the data developed for this task and the findings reported to be valuable to the scientific knowledge extraction, metrology, and automated knowledge base construction communities.

## introduction counts and measurements are an important part of scientific discourse @xcite . it is relatively easy to find measurements in text @xcite , but a bare measurement like 17mg is not informative without knowing what it is referring to. for example, it is important to know whether a quantity is 17mg of a medicine dosage or 17mg of concrete additive. only recently have attempts been made to identify the named entity and property being measured @xcite . extracting such information is challenging because the way scientists write can be ambiguous and inconsistent. furthermore, the location of this information relative to the measurement can vary greatly, and might even be in a different sentence. being able to extract measurement information automatically can enable the construction of databases of measured properties. such databases are important in biomedicine @xcite , engineering @xcite , and other scientific disciplines @xcite , but the approaches used for populating these databases do not generalize widely. furthermore, knowledge graphs @xcite frequently aggregate quantitative data reported in the literature and are often built through a largely manual curation process. examples include: litterbase foot_0 @xcite , which aggregates observations of marine litter distribution; neuroelectro foot_1 @xcite , which collects information on electrophysiological properties of neurons; and various model organism databases like the zebrafish information network foot_2 @xcite , which provide summaries of gene information. beyond knowledge graphs and curated databases, clinical health contexts often require extraction of measured values for lab results and patient observations. moreover, scientific research frequently relies on precise measurements for reproducibility of experimental methods @xcite . measured property extraction could be of value in many other contexts, such as fact checking and news validation or in statistical analysis for public policy @xcite . research in information extraction and knowledge graph creation has concentrated on forming triples by extracting entities and relations @xcite . little attention has been paid to the extraction of measured properties, entities, and conditions or contexts, yet these elements are needed to place measurements into a database and for their subsequent use in comparison and calculation. units and measures are an important part of the semantic web, though research has largely been focused on ontology design @xcite . there is, thus, a need for understanding the state of the art on this important task. the aim of this paper is to introduce the mea-seval shared task for the extraction of counts, measurements, and related context from englishlanguage scientific documents, as well as to present an analysis of the results of participant systems on the task. the rest of this paper is organized as follows. we begin with a description of related work. this is followed by the description of the task itself (section 3) and the associated data and annotation procedure (section 4). the evaluation regime is detailed in section 5 including baselines. subsequently, we present an analysis of the results of the systems on the task. finally, we summarize the various participating systems approaches and conclude. ## related work there is a substantial body of work discussing units of measurement, ontologies to describe them, systems designed to extract them, as well as related work on knowledge graphs of numerical attributes. automated extraction of measured quantities, such as 520 +/-8 items/kg, is straightforward and many tools exist to perform this task @xcite @xcite . to build a knowledge graph, we must put these measurements in context. we need to determine the properties being measured (e.g. abundances), the entities that exhibit those properties (e.g. the maowei sea), and possible qualifying conditions under which measurements are obtained (e.g. the date and depth of the sampling). these properties, entities, and conditions can then be mapped to those that are used in the knowledge graph, so that the measurements can be normalized into a common system. there are a number of ontologies that cover units of measurement, such as quantities, units, dimensions, and types ontologies (qudt) 4 and the ontology of units of measure and related concepts (om) @xcite . these and others are discussed in a survey paper by @xcite . most of these ontologies focus on conversion between different systems of measurement, and on classifying types of measurement or domain of application, but do not necessarily address the "thing" being measured. the joint committee for guides in metrology's (jcgm) international vocabulary of metrology covers this in slightly more depth, discussing measurement units and quantity values, then talking about quantities themselves, which it defines as a "property of a phenomenon, body, or substance" (joint committee for guides in @xcite . we find that this nomenclature, while precise, is likely to be con-fusing to non-metrologists from both an evaluation and annotation perspective, so to support the data annotation process for this task we use a simplified nomenclature. metrology research in the semantic web community is often focused on ontology alignment for units of measurement ontologies. @xcite look at aligning unit ontologies to support merging data across many weather information systems, while @xcite more generally look at using mathml for aligning unit ontologies. efforts around designing linked data models for semantic sensor streams for the internet of things also utilize the units of measurement ontology for representing measurement information @xcite . none of this work addresses extraction of measurements and their contexts nor building knowledge graphs from such information. other research explores creating databases of numeric attributes. @xcite infer new values using linear regression for neighboring entities in a knowledge graph. @xcite use a logistic regression with distributional vectors. @xcite use a system of averages and boundary values to infer an estimated numeric attribute value. rather than imputing new values from related entities, measeval starts from a value and puts it into the context of measured entities and measured properties, working toward a knowledge representation of numeric data. ## task description measeval is an entity recognition and semantic relation extraction task focused on finding counts and measurements, attributes of those quantities, and additional information including measured entities, properties, and measurement contexts. measeval is composed of five sub-tasks that cover span extraction, classification, and relation extraction, including cross-sentence relations. given a paragraph from a scientific text: more detailed definitions can be found be reviewing the measeval annotation guidelines. 5 we describe each of the elements to be extracted in more detail in the next section. ## results and discussion during the 21-day evaluation period (january 10 through 31, 2021), 26 codalab users submitted a total of 89 submissions, of which 77 passed validation and were successfully scored by the evaluation script. given the complexity of the task, we opted to allow for five submissions total during the evaluation, although some collaboration between users meant that some teams were able to effectively submit more than five times. we note that submissions did not calculate scores on sub-tasks, thus making it difficult to overly optimize models using just the overall score. the relatively generous submission allowance does not seem to have presented too much of an over-fitting problem, as scores remain relatively low on all tasks, although the collaboration could have given some participants a slight advantage in the rankings. in addition to the overall f1 scores, the overall top-performing model was at least tied for top performance in five out of 6 of the component scores in figure 4 provides a visualization of the distribution of scores for 9 of the 10 subject areas in the corpus. the mathematics subject area has been omitted from this graphic due to under-representation in both the training and evaluation datasets. ## summary of participating systems the measeval track at semeval-2021 received nine system description paper submissions, eight of which are represented in the analysis in there are several points of similarity between the eight main submissions. all but one of the systems are based on the bert architecture @xcite or a derivative, such as scibert @xcite , biobert @xcite , or roberta @xcite . all but one used a pipeline architecture, starting with quantity extraction. all but one used sequence tagging with a bio encoding scheme, and four followed the sequence tagger by a conditional random fields (crf) model to assemble tokens into spans and improve accuracy over simple token-level classifiers. unit and modifier extraction was either done using a character-level bilstms, another bert model, or a rule-based approach. finally, it was common to see measuredentity, measuredproperty, and qualifier, and sometimes the relation extraction components, stacked together in a multi-task sequence tagging model as a final stage, taking both the original sentences and quantity spans as input. one system diverged from the sequence tagging tagging approach and used templated question answering techniques to handle the relation extraction along with related spans.