Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction (2023)

Abstract: We present a cross-domain approach for automated measurement and context extraction based on pre-trained language models. We construct a multi-source, multi-domain corpus and train an end-to-end extraction pipeline. We then apply multi-source task-adaptive pre-training and fine-tuning to benchmark the cross-domain generalization capability of our model. Further, we conceptualize and apply a task-specific error analysis and derive insights for future work. Our results suggest that multi-source training leads to the best overall results, while single-source training yields the best results for the respective individual domain. While our setup is successful at extracting quantity values and units, more research is needed to improve the extraction of contextual entities. We make the cross-domain corpus used in this work available online.

## introduction numeric components such as counts, measurements and are crucial information for researchers across various disciplines. an automatic system for the extraction of numerical measurements (e.g. 10 %) and related "contextual" information such as measurands (e.g., concentration) and measured entities (e.g., chemical solution) for scientific literature can aid in the efficient construction of knowledge bases @xcite for downstream research tasks. ideally, the system should be able to handle multiple subject domains or even unseen domains, as relying on multiple specialized systems is inefficient and sometimes infeasible: for instance, each specialized model requires dedicated training and deployment resources. further, the target-domain cannot always be known at inference time. ## a new multi-domain corpus for measurement extraction in this section we describe the creation of a multidomain corpus for measurement and context extraction. this will enable the investigation of crossdomain prediction performance. ## extraction architecture we now describe our model setup which is designed to extract the entity and relation types described in the previous section. we model the extraction as a two-step pipeline made up of two token-classification models, which we coin as task 1 and task 2. we first extract all quantities (task 1) and then simultaneously predict u, me and mp (task 2) based on each extracted quantity. this cascading setup resolves the relation extraction problem of assigning the context entities to the correct quantity span, as the data model allows for a deterministic, rule-based assignment of the relations between u, me and mp (see gangwar et al. (2021); davletov et al. (2021)). figure 3 shows the extraction flow based on an example sentence: the information from the first task is input into the second task through special tokens [q] and [/q] which we wrap around the identified q spans (see also @xcite ; @xcite ). for each identified q, an enriched prediction sample is created, thereby allowing for overlapping entities and conditioning the unit and context entity extractor on one q at a time. for q extraction we use binary io-tags @xcite . for task 2 we use the bio-tagging scheme. to accommodate the tokens [q] and [/q] which signal the identified q spans from task 1, we add them to the models' vocabulary as special tokens, extending the embedding size by two. for training, we use cross-entropy loss over all classes and train task 1 and task 2 separately. a drawback of this simple architecture is the fact that it cannot enforce the 1:1 relationships prescribed by the data model, since it is possible to predict more than one me or mp. further, we set the input sequence to the size of a single sentence to account for the one-sentence annotation window of the msp and battery materials dataset. ## domain adaption and generalization we experiment with a) adaptive pre-training and b) multi-source fine-tuning. figure 4 summarizes the applied methods and resulting model configurations. with the exception of the training setting with all sources, all shown configurations are applied to the models of both tasks. adaptive pre-training. this setup comprises a combination of pre-trained base models and intermediate pre-training: we use bert base foot_2 @xcite as the baseline model representing the canonical text domain, and scibert foot_3 @xcite , which we expect to be more closely related to the domains of our measurement extraction corpus, because it was pre-trained from scratch on scientific articles. we create intermediately pre-trained variants for each of the two bert-models using task-adaptive pre-training (tapt) @xcite : we continue pre-training of the models on the unlabeled (training) data of our measurement extraction corpus. thereby we aim to bring the models closer to the target domains of the task and induce increased task performance compared to the base models. we also apply adapter-based @xcite intermediate pretraining to compare full tapt against to a more parameter efficient approach @xcite . as pre-training data, we create a "curated" (gururangan et al., 2020) multi-source corpus comprised of the pooled data from all three training datasets. we enhance it with unlabled data from the same datasets to further increase the corpus size. as such we add all articles of the oa-stm dataset (elsevier labs, 2015), the source on which the measeval annotations are based. we remove paragraphs which appear in the test and validation splits of the measeval data through fuzzy string matching foot_4 . further, we add 1,128 battery materials claims which were excluded from the measurement extraction corpus due to lack of quantity spans, and include the rest of the msp data that was not re-annotated. this resulted in a pre-training corpus of approximately 630k words. ## experiments we perform various experiments investigating the generalization capabilities of our system depending on data selection and domain adaption techniques. the implementation details can be found in appendix g. ## error analysis to better understand the challenges of the task and deficiencies of our system, we perform an indepth error analysis. we analyze error sources on a fine-grained entity level. ## conclusion we have applied pre-trained language models to end-to-end measurement, unit and context extraction. while our setup exhibits good extraction performance for quantities and units, more research has to be done to improve the extraction of contextual entities. we have identified long-range dependencies of mes as a particular error source. in terms of cross-domain generalization and multi-source training, multi-source training produced the best overall results, while single-source training often yielded the best results for the respective target domain. an exception to this was the small bm dataset, for which we observed the best unit and context extraction performance in the cross-domain prediction setting. this is an indicator for domain generalization, especially for low-resource domains. however, this needs to be confirmed in additional experiments with a dataset comprising even more domains. when comparing adaptive pre-training methods, the most consistent performance driver was the use of the scibert base model instead of the bert base model. further, we found adapter-based intermediate pre-training to be worse in most cases for both model types and tasks, which may be due to the task complexity. this theory is affirmed by the fact that we saw better adapter-based pre-training results for the simpler quantity extraction. finally, we found non-conclusive results for the comparison of no pre-training versus full pretraining. the instability of results may be due to the limited size of our pre-training data, or the effect of catastrophic forgetting. future work with a larger pre-training corpus may give clearer insights into this case.