S tanford ML ab at S em E val-2021 Task 8: 48 Hours Is All You Need (2021)

Abstract: This paper presents our system for the Quantity span identification, Unit of measurement identification and Value modifier classification subtasks of the MeasEval 2021 task. The purpose of the Quantity span identification task was to locate spans of text that contain a count or measurement, consisting of a value, usually followed by a unit and occasionally additional modifiers. The goal of the modifier classification task was to determine whether an associated text fragment served to indicate range, tolerance, mean value, etc. of a quantity. The developed systems used pre-trained BERT models which were fine-tuned for the task at hand. We present our system, investigate how architectural decisions affected model predictions, and conduct an error analysis. Overall, our system placed 12 / 19 in the shared task and in the 2nd place for the Unit subcategory.

## introduction the growing ease of access to large bodies of scientific information and research has not been accompanied by an improvement in ease of analysis or understanding of scientific text. while the performance of natural language processing tasks such as part-of-speech (pos) tagging and dependency parsing have seen improvements, analyzing counts and measurements in scientific texts has remained a largely unaddressed problem, though some work has been done by @xcite . measurements involve quantification (in units such as litres or kilograms), entities and the measured properties (e.g. growth rate of a fungus) and value modifiers (e.g. mean, range, tolerance). extracting semantic relations between quantities, entities, value modifiers, and units of measurement and deriving meaning from those components is challenging because scientific communication can be ambiguous or inconsistent. in addition, the loca-tion of this information relative to the measurement tends to vary. the precise extraction and contextualization of measurements could enable accurate summarization of large bodies of scientific literature. additionally, this could allow for unordered measurement and numeric data from scientific texts to be transformed into standardized ordered data for easier analysis. in this paper, we describe a rapidly implemented, lightweight model that extracts measurement context in natural language. our pipeline was implemented in approximately 48 hours by the first three authors, who had minimal formal neural nlp experience. despite this, we were able to achieve strong results in the two categories (quantity and unit) that we entered, achieving second place in the unit category and 12 / 19 overall. ## experimental setup for all of the subtasks, we used the provided traintrial split of 2531 annotations across 249 texts versus 832 annotations across 66 texts. besides tokenizing the text and applying io labels (see section 3.1) to get the data into a format fit for the bert token classifier, we did not preprocess the data significantly. we tested filtering unrecognizable and irrelevant characters using regex but found no meaningful improvement in performance. we thus decided to omit preprocessing to keep with the simplistic theme of this paper. during the hyperparameter tuning phase, we compared models based on token-level f1 score, implemented using the classification report from the metrics module of pytorch, yielding results consistent with the provided testing script, which evaluated on the f1 overlap score. due to the limit on the total number of iterations we could go through due to the deadline, we tuned hyperparameters by hand. we found that out of the hyperparameters we tuned, the scheduler hyperparameters were irrelevant because the scheduler was rarely used at all. the number of training epochs likewise did not come into play so long as we let the training continue until trial loss stopped decreasing; around 15 epochs was almost always enough. the only hyperparameters that seemed to significantly impact training results were the learning rate and the weight difference between quantity and non-quantity labels. after some manual tuning, we found that a learning rate of around 10 -5 and no weight-difference adjustment performed best on the trial set for this task, resulting in our final token-level quantity identification f1 score of 0.85 on the trial set. for the quantity identification subtask, several post-processing steps were implemented to correct simple errors in our model's predictions, based on common errors detected via manual error analysis in the model's predictions. first, commas and spaces were removed from the ends of the predicted quantities to correct for potential small character offsets on the edges of predictions, from either model error or the tokenization-detokenization process. furthermore, all quantities that did not contain either numeric characters or strings that commonly occur in numbers (such as two or -teen) were discarded from the prediction pool. additionally, all prepositions from a hand-picked set like beyond and at were cut from the front of each input string in order to better isolate numeric quantities from relational terms. the trial set results with and without post-processing are shown in ## results our model's overall precision, recall, and f1 score on overlap on the trial set are as shown in these results are promising given the speed of implementation of our system. across all 5 subtasks, our final f1 score was 0.272. overall, the model ranked 12th out of 19 teams in the evaluation phase of measeval: counts and measurements. ## conclusion we propose a bert-based model for information extraction of measurements and their contextual qualifiers from text. as scientific texts become increasingly open for public consumption, we hope that such systems will help present findings to broader audiences in an accessible manner. going forward, possible areas of exploration include replacing bert with more powerful models and augmenting training data that the model frequently makes erroneous predictions on.