JBNU - CCL ab at S em E val-2022 Task 12: Machine Reading Comprehension and Span Pair Classification for Linking Mathematical Symbols to Their Descriptions (2022)

Abstract: This paper describes our system in the SemEval-2022 Task 12: ‘linking mathematical symbols to their descriptions’, achieving first on the leaderboard for all the subtasks comprising named entity extraction (NER) and relation extraction (RE). Our system is a two-stage pipeline model based on SciBERT that detects symbols, descriptions, and their relationships in scientific documents. The system consists of 1) machine reading comprehension(MRC)-based NER model, where each entity type is represented as a question and its entity mention span is extracted as an answer using an MRC model, and 2) span pair classification for RE, where two entity mentions and their type markers are encoded into span representations that are then fed to a Softmax classifier. In addition, we deploy a rule-based symbol tokenizer to improve the detection of the exact boundary of symbol entities. Regularization and ensemble methods are further explored to improve the RE model.

## introduction mathematical symbols and descriptions appear in various forms across document section boundaries without explicit markups, and mathematical symbols appear in the form of long texts. thus, linking mathematical symbols and their descriptions is challenging. semeval 2022 task 12: 'linking mathematical symbols to their descriptions @xcite ', is a relation extraction task targeted at scientific documents divided into two sub-tasks: sub-task a is a named entity recognition (ner) task that aims to predict the span of symbols and descriptions, and sub-task b is a relation extraction (re) task that aims to predict relations between symbols and descriptions. extracting these entities and relations is done to discover relational facts from unstructured texts. this problem can be decomposed into ner @xcite and @xcite . early works employed a twostage relation extraction system, training one model to extract entities @xcite and another model to classify relations between these entities @xcite . to reduce the error propagation of ner or better capture the interactions between ner and re, joint models have been proposed as a promising approach that are based on an end-to-end method or on the setting of multi-task learning using shared representations @xcite @xcite . recently, it has been observed that re based on the shared encoder is suboptimal, but the use of separated encoders for ner and re has shown improved performance compared to shared encoders, reexamining the effectiveness of the simple pipelined two-stage approach @xcite . from these results, we hypothesize that whereas separated encoders for ner and re can learn customized representations useful for each task, joint models may include irrelevant information in the learned representation for ner or re tasks, lowering the performance of the model. these results of using distinct encoders (zhong and chen, 2021) encourage us to adopt the aforementioned two-stage approach for ner and re tasks, consisting of 1) mrc-based ner and 2) span pair classification for re, as follows: unlike the pure system of (zhong and chen, 2021) that exploits the standard span-based ner of @xcite , our ner model is based on an mrc-based model @xcite , which treats ner as ## system overview in this section, we first describe the models of the proposed system for each sub-task. ## experimental results as shown in overall, the recall is relatively higher than the precision for the re model. in our preliminary experiments, we observed a similar tendency with high recall for the ensemble method, despite the fact that the ensemble method was shown to be effective in terms of f1 score. we use a voting threshold to increase the pre-cision of the ensemble re model and adjust the ensemble inference so that a nil-type class is assigned either when the size of the majority votes from the models does not exceed the voting threshold or when the class from the majority votes is nil-type. figure 4 shows the performance of the ensemble method across different voting threshold values. in this case, we observe that as the voting threshold is raised, the f1 score gradually increases, while precision increases and recall decreases. as a result, when voting threshold is 10, the best performance of the f1 score is obtained. this run was finally submitted. ## analysis in this section, we examine the effects of some of the components of our system as well as additional trials. ## conclusion our system shows first for all subtasks of semeval-2022 task 12: 'linking mathematical symbols to their descriptions'. mrc-based ner and span pair classification for ner are part of our system that uses scibert as a backbone encoder. to improve the performance, the symbol tokenizer for ner model, regularization, and ensemble methods, for re model are used. to improve the performance further, future work should look into data augmentation and mathematical symbol and description-aware pretraining.