CL a C - BP at S em E val-2021 Task 8: S ci BERT Plus Rules for M eas E val (2021)

Abstract: This paper explains the design of a heterogeneous system that ranked eighth in competition in SemEval2021 Task 8. We analyze ablation experiments and demonstrate how the system components, namely tokenizer, unit identifier, modifier classifier, and language model, affect the overall score. We compare our results to similar experiments from the literature and introduce a grouping algorithm developed in the post-evaluation phase that increased our system’s overall score, hypothetically elevating our competition rank from eight to six.

## introduction the measeval 2021 shared task involves identifying related groups of the four annotation types seen in ## data the data comprises excerpts from scientific articles of different disciplines: astronomy, engineering, medicine, agriculture, biology, chemistry, earth science, computer science, and mathematics. the limited data (see all complete data-points contain one quantity and at most one annotation span of each other type. a summary of complete data-point frequencies is provided in annotations present frequency qa 11 me,qa 361 me,mp,qa 512 me,ql,qa 50 me,mp,ql,qa 225 mp,qa 4 mp,ql,qa 1 all of the system's deep learning components are implemented using pytorch @xcite . our models use simple (linear + softmax) classification on top of bert's implementation from the huggingface library. 3 pre-trained weights are obtained from huggingface's model hub. 4 our competition scibert models are optmized by adam (kingma and ba, 2015) using a constant learning rate of 5e-6 (the rest of adam's parameters are set to pytorch's defaults). the competition system's token classifier is fine-tuned on all the training data for 5 epochs, while the modifier classifier is finetuned on all the training data for 3 epochs. ## results our submission ranked eighth in competition (listed as rank 7 on the codalab leaderboard). our system is robustly above average on all categories (see our competition system had a non-zero dropout probability, therefore, we cannot recreate its exact performance for the benchmarking of our ablations. the system labelled full was run with zero dropout probability and is otherwise identical to our competition system. all ablated systems are compared to this baseline. noticeable differences the first comparison in row 2 substitutes our fine-tuned scibert model for a fine-tuned bert base model for token classification. scibert outperforms bert for every category except quantity and unit. in row 3 we assess our modified tokenizer, by substituting it with spacy's default tokenizer. the performance is near identical to our baseline system in every category. the third and fourth systems were trained without modifiers and without units respectively. overall, the system without units performed worse than the system without modifiers. this is, however, to be expected as there are 905 units and only 552 modifiers, i.e. units account for more overlap score. finally, the last comparison showcases the improvements when using span++. ## discussion our system's modular pipeline allows us to assess the components in ablation studies and the various experiments we perform are informative about the task. compared to bert base, scibert pretraining gives a solid advantage to all categories, except quantities and units. not surprisingly, comparing scibert's and bert's performance on the token level objective ( our system might benefit from a more precise token level classifier due to its grouping algorithm: it is reasonable to assume that it performs better when more of the detected spans are correct (high precision system) and performs worse with a greater number of incorrect spans (high recall system). substituting spacy's tokenizer into our system does not make much difference because scibert's wordpiece tokenizer will break any unknown to- system lm em p r f o qa me mp ql unit mod hqa hp quals full scibert .346 .641 .516 .572 .382 .848 .249 .308 .111 .667 .549 .296 .136 .061 full bert base .311 .591 .496 .539 .349 .853 .239 .24 .063 .697 .537 .253 .107 .038 full no m.t * scibert .341 .644 .513 .571 .381 .848 .258 .306 .103 .653 .522 .297 .148 .056 full no mod scibert .309 .63 .464 .534 .347 .848 .249 .308 .111 .667 .0 .296 .136 .061 full no unit scibert .277 .608 .422 .498 .314 .848 .249 .308 .111 .0 .567 .296 .136 .061 full with span++ scibert .38 .687 .551 .612 .421 .848 .292 .367 .151 .667 .549 .371 .162 .081 clac-bp (rank 7) scibert ----.389 .855 .251 .318 .107 .677 .546 .308 .147 .058 top ranked -----.519 .861 .437 .467 .163 .722 .642 .482 .318 .092 compet. mean -----.323 .741 .200 .196 .021 .602 .364 .205 .114 .012 compet. median -----.369 .818 .251 .245 .000 .661 .375 .308 .147 .000 * m.t: modified tokenizer em: exact match hqa: hasquantity hp: hasproperty quals: qualifies addressing the modifier categories with a separate scibert model is successful and allows our system to benefit from several rule-based transition phases. while the task is not well understood (see the overall low performances and the high interannotator disagreement), we believe this type of architecture to be more beneficial than end-to-end systems, as we can pinpoint weaknesses and experiment more easily. this is demonstrated by the sizeable improvement of results when implementing a simple distance measure constraint for the grouping step. once such features have been identified as effective, their encoding in the classifier becomes possible. ## summary measurements are ubiquitous in scientific articles, yet have so far not been addressed. the measeval task is an opportunity to combine different subtasks and experiment how best to combine them. our system approaches the measeval task in a modular fashion: preprocessing, two classifications, and postprocessing. it breaks the task into two, first a classification of the relations for measured entities, measured properties, and qualifiers and second a classification of modifiers. scibert, trained on scientific articles, yields better performance than bert in our experiments, mainly due to improvements in precision. the differences, however are small. before, between, and after the two classification steps are a number of rule-based modules that create the classifier input and piece together the classifier output for submission. several experiments on variations in these rule-based accessories suggest their usefulness and ways to improve our results.