Measurement Extraction with Natural Language Processing: A Review (2022)

Abstract: Quantitative data is important in many domains. Information extraction methods draw structured data from documents. However, the extraction of quantities and their contexts has received little attention in the history of information extraction. In this review, an overview of prior work on measurement extraction is presented. We describe different approaches to measurement extraction and outline the challenges posed by this task. The review concludes with an outline of potential future research. Research strains in measurement extraction tend to be isolated and lack a common terminology. Improvements in numerical reasoning, more extensive datasets, and the consideration of wider contexts may lead to significant improvements in measurement extraction.

## introduction humanity is accumulating more and more knowledge at an ever faster pace. cast into large amounts of documents, relevant knowledge is no longer graspable by a few individuals. information extraction (ie) is a task in natural language processing (nlp) and assists in managing the amount of information hidden in documents by automatically extracting and organizing information from semi-and unstructured sources (e.g., populating a database from information conveyed in natural language). in the early 1990s, the message understanding conferences (muc) fostered research in ie trough challenges in template filling @xcite . later mucs and the automatic content extraction (ace) program split ie into several sub-challenges, helping named entity recognition (ner), relation extraction, event extraction and coreference resolution to emerge as individual research subjects @xcite @xcite . today, ie is applied in many domains, such as biomedicine @xcite , 2018), chemistry and materials science @xcite . however, measurements and their contexts have received little attention in the history of ie @xcite @xcite @xcite . numbers form a cornerstone of our society, on which science, engineering, trade and much more is built. numerical reasoning is therefore an essential, albeit underexplored, problem in nlp @xcite , the addressing of which seems to even enhance the general literacy of language models @xcite . the task of measurement extraction is to identify quantities and related information in texts, tables and figures. in this review, we focus on measurement extraction from text (cf. figure 1 ). when specifying measurements, the transition from natural to mathematical language is seamless, making measurement extraction a special task within nlp. the variety of relevant problems to which measurement extraction is applied further highlights its importance. research on measurement extraction is focused on highly quantitative domains, in particular clinical, biomedical, chemistry, and materials research. accordingly, most systems are applied to scientific publications and clinical documents. within the medical domain, more frequent applications include the extraction of eligibility criteria from clinical trials @xcite , the extraction of lab test information @xcite , the extraction of measurements from narrative radiology reports @xcite @xcite and the extraction of the left ventricular ejection fractions of hearts @xcite @xcite . within chemistry and materials science, information on experiments @xcite @xcite , materials synthesis @xcite and nanoscience @xcite @xcite @xcite is extracted. beyond that, application domains include patent analysis @xcite , automated chart generation @xcite , fact-checking @xcite , earth science @xcite , engineering design @xcite , information retrieval @xcite @xcite , automated compliance checking @xcite , and more. however, research strains tend to be isolated within these domains (see figure b1 ), indicating a lack of an overview. in this paper, we define measurement extraction (section 2) and survey prior research (section 3). subsequently, we highlight special challenges (section 4) and provide several recommendations for future research (section 5). section 6 describes the limitations of this review. to the best of our knowledge, the present review is the first that focuses on measurement extraction. ## task definition the language around measurement extraction lacks standardization (see section 5). likewise, the scope of measurement extraction is not welldefined. we define it as follows: quantity extraction is the task of identifying quantities. a quantity (e.g., '1 kg') is composed of a numeric value and, if applicable, a unit. the meaning of a quantity is often altered by modifiers such as 'average', 'approx.' or 'above'. modifiers adjacent to numeric values are sometimes included in the quantity spans @xcite . a quantity might be given as a range, enumeration, with an uncertainty specification, or all together. numeric values might be expressed as numeric numbers (e.g., '27'), alphabetic numbers (e.g., 'twenty-seven'), combinations (e.g., '2 million'), imprecise quantities (e.g., 'a couple'; cf. @xcite or constants (e.g., 'room temperature' or 'speed of light'). within a quantity span, the unit might be identified. units are often abbreviated according to their symbol (e.g., 'j' for joule). note that nouns, such as in '9 family houses', are sometimes considered units @xcite . quantities can be normalized to base si units. as some unit symbols are ambiguous, the kind of quantity might be identified first (e.g., length for '1 µm'). furthermore, the notions of change (e.g., 'decreased') might be extracted for quantities that are given relative to another quantity (e.g., in "the gdp decreased by 4.6 %"). the boundary between quantities and equations is fuzzy. hence, formulaic expressions are considered to differing extents. measurement extraction adds to the identification of quantities by extracting their related measured properties and measured entities (cf. figure 1 ). a measured property might be given implicitly. measurement extraction can be generic or simplified by only targeting specific measured entities and properties (e.g., if particle sizes should be identified, only length units must be considered). furthermore, additional qualifiers such as constraints, measuring methods or references that qualify a quantitative statement might be extracted. measured entities, properties, units and relevant context might be disambiguated against a knowledge base (that is, entity linking). related tasks that involve numerical reasoning besides measurement extraction from other modalities are, amongst others, product attribute value extraction @xcite , equation parsing @xcite , solving math word problems @xcite , quantity entailment @xcite , number sense disambiguation @xcite , numeral attachment @xcite , and masked measurement prediction @xcite (see appendix a). the interested reader is directed to the surveys of @xcite and @xcite , which provide extensive overviews of various nlp tasks involving numeracy. ## prior work on measurement extraction various systems for measurement extraction have been proposed. the first research efforts focusing on measurement extraction date back to at least 2006 @xcite . we identified 80 publi-cations describing one or more systems for measurement extraction. this section summarizes their approaches according to the following subtasks: varying scopes. most systems do not cover all subtasks and the respective concept types of the general pipeline depicted above, which fails to reflect the large variations in their scopes. only a few systems cover the identification of measured entities, properties, further context, and their relations in addition to quantity extraction (see table b1 ). many of those are submissions to measeval (task 8 at semeval 2021; @xcite . frequently, the other systems do not distinguish between measured entities and properties. the rule-based (symbolic) systems tend to have a narrower scope than the learning-based systems; that is, instead of identifying measurement concepts generically, only particular concepts, which are specific to the domain and use case, are identified. covering only a small set of concepts facilitates normalization and entity linking. in fact, with symbolic approaches, this information is often already evident from the matching patterns. many systems do not approach the extraction of quantity modifiers and qualifiers. all systems that approach quantity modifier extraction only consider a small set of modifier classes. only a few systems consider the notions of change (e.g., 'increased') for relative quantities @xcite @xcite . in measeval, phrases that indicate change are regarded as qualifiers. 1 only a few articles explicitly state that co-references @xcite @xcite and negations are considered. @xcite @xcite . ## challenges quantities are easy to identify in text, both numbers and units, which facilitates anchoring semantic role labeling schemata @xcite . in addition, many numerical relations are accompanied by only a few keywords @xcite and values of numerical attributes "can be estimated even if they are not explicitly mentioned in the text" @xcite . nonetheless, measurement extraction poses various challenges: measurements are diversely expressed. quantities can be expressed in a myriad of different surface forms, yet alone by different levels of rounding and combinations of units. in addition, different writing styles for decimal and thousands separators exist. also, complex patterns involving multiple quantities such as "group 1, 2 and 3 were given 4, 5 and 6 µg ml , respectively" are common @xcite and measurement extraction might include parsing formulaic expressions (e.g., "t(29) = -1.85, @xmath0.074"; @xcite . ## a vision for the future having arranged and summarized the prior work in measurement extraction, we now provide several recommendations that might positively shape future research. these go beyond addressing the aforementioned challenges, which must inevitably be dealt with, in that they are concrete recommendations for action. a common terminology is what language around quantitative information extraction is lacking. although standardization efforts exist @xcite , different terms are used for the same concept and the same terms are used for different concepts. for example, the terms measurement entity @xcite , numeric property @xcite , and value @xcite are all used for referring to a quantity. adding to the confusion, in @xcite a quantity denotes a measured property. aiming to end this confusion, we propose to adopt the terminology of measeval, which defines the terms quantity, measured entity, measured property, quantity modifiers, and qualifiers @xcite . in line with the unit ontology qudt @xcite , a quantity is composed of a numeric value and a unit. more extensive datasets that cover quantities as well as their contexts could greatly improve results in measurement extraction. the dataset used in measeval, for example, consists of only 428 paragraphs @xcite , limiting the performance of the learning-based methods @xcite . more generic annotations could render datasets for measurement extraction more sus-tainable. we argue that the reuse of datasets is hindered by incompatible annotations. for example, the sets of quantity modifier classes in the datasets of measeval and grobid-quantities do not match. also, the sets of modifiers are selective, not considering all occurring modifiers and combinations. therefore, we propose annotating the quantity spans with pseudo-mathematical representations that can be parsed into classes depending on the task or directly used for sequence-to-sequence approaches. improving the numerical reasoning capabilities of the models may well improve the performance of measurement extraction systems. character-level embeddings, for example, outperform word-and subword-level methods @xcite . altering the surface form of all numbers during pre-processing can improve model performance @xcite @xcite . furthermore, extending language models with special representations of numbers improves numerical reasoning capabilities @xcite . @xcite propose the extension of language models with a set of executable programs for symbolic reasoning. still, recent advances in numerical reasoning have been barely considered in the literature on measurement extraction. document understanding remains an ambitious objective. systems for measurement extraction that consider document context and incorporate information from other modalities are rare @xcite @xcite . in fact, many systems operate on a sentencelevel or truncate the processed text after a fixed token limit. we argue that both context from other modalities (e.g., joint inference from text and tables) and distant context should be considered. ## limitations relevant literature has been iteratively identified using different academic search engines, foremost semantic scholar 6 , and by tracing the references in already identified publications. publications disclosing systems whose scope is too narrow or offset, that target figures or tables, or that lack detailed information are dropped. related work that was not deemed relevant is listed in appendix a. that said, many systems extract quantities, amongst other concepts, but do not elaborate on it. it is likely that additional systems exist that identify quantities and their contexts, but which are not included in this review. it was decided against a quantitative assessment of the systems' performance, as both their scopes and evaluations differ from each other, making a fair comparison difficult. research papers. in proceedings of the 4th international workshop on computational terminology (computerm), pages 77-85, dublin, ireland. association for computational linguistics and dublin city university. george doddington, alexis mitchell, mark przybocki, lance ramshaw, stephanie strassel, and ralph weischedel. 2004. the automatic content extraction (ace) program -tasks, data, and evaluation. in proceedings of the fourth international conference on language resources and evaluation (lrec'04), lisbon, portugal. european language resources association (elra). xin luna dong, xiang he, andrey kan, xian li, yan liang, jun ma, yifan ethan xu, chenwei zhang, tong zhao, gabriel blanco saldana, saurabh deshpande, alexandre michetti manduca, jay ren, surender pal singh, fan xiao, haw-shiuan chang, giannis karamanolakis, yuning mao, yaqing wang, christos faloutsos, andrew mccallum, and jiawei han. 2020. autoknow: self-driving knowledge collection for products of thousands of types. proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining, pages 2724-2734. dheeru dua, yizhong wang, pradeep dasigi, gabriel stanovsky, sameer singh, and matt gardner. 2019. drop: a reading comprehension benchmark requiring discrete reasoning over paragraphs. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 2368-2378, minneapolis, minnesota. association for computational linguistics. tracy edinger, aaron m. cohen, steven bedrick, kyle ambert, and william hersh. 2012. barriers to retrieving patient information from electronic health record data: failure analysis from the trec medical records track. amia annual symposium proceedings, 2012:180-188. steffen epp, marcel hoffmann, nicolas lell, michael mohr, and ansgar scherp. 2021. a machine learning pipeline for automatic extraction of statistic reports and experimental conditions from scientific papers. a related work that is not considered systems whose scope is too narrow or offset are not considered in this review. the pure identification of units is not considered measurement extraction. @xcite parses unit strings into a structured semantic representation using the qudt ontology @xcite and thereby allows the transformation of (compound) units. @xcite perform entity linking of units in text against a knowledge graph. they also extract numbers but do not elaborate on it. furthermore, the identification and parsing of numerals (e.g., @xcite is not considered. not all numerals are part of measurements, for example, ordinal numbers (e.g.,. 'fig. 1 '), or nominal numbers (e.g., postal codes). pouran ben veyseh et al. (2021) approaches only the relation extraction subtask of measeval and is therefore not considered. moreover, systems targeting other modalities than text are not considered. subercaze (2017), for example, extracts measurements from wikipedia infoboxes. furthermore, systems that, alongside other information, extract quantities, but do not elaborate on it, are not covered. for example, gate @xcite has a plugin for tagging measurements, but further information is missing. @xcite extract information, including quantities, but without addressing these specifically. @xcite aggregate numeric results for web search queries without sharing details on their ie system. quantalyze foot_7 is a product from max.recall information systems gmbh. it seemed to have poor recall @xcite and supported only a small set of units @xcite . quantulum foot_8 is a python library for the extraction of quantities from text, which is able to perform unit disambiguation based on wikipedia and glove vector representations. other ie systems extract quantitative information from data sheets but do not elaborate on it @xcite . many systems have been proposed that extract attribute values from product profile pages, but do so without specifically discussing numeric values @xcite @xcite . lastly, we did not consider systems that target related but distinct tasks from measurement extraction such as the identification of text fragments that contain measurements and their contexts @xcite , the extraction of molar ratios of material compositions @xcite , the prediction if a numeral in a sentence is a claim or fact @xcite , the classification of numerals in financial tweets into different categories @xcite , financial numeral attachment @xcite , extracting relation cardinalities (e.g., the cardinality @xmath1, haschil-@xmath2, as he has two children; @xcite and generating descriptions of quantities that put them into relation with other quantities (e.g., "about twice the median income for a year" given a sentence that contains the quantity '100 000 $'; @xcite . @xcite argue that language models should jointly reason about numbers and units to learn good representations of measurements and propose the task of masked measurement prediction.