LIORI at S em E val-2021 Task 8: Ask Transformer for measurements (2021)

Abstract: This work describes our approach for subtasks of SemEval-2021 Task 8: MeasEval: Counts and Measurements which took the official first place in the competition. To solve all subtasks we use multi-task learning in a question-answering-like manner. We also use learnable scalar weights to weight subtasks’ contribution to the final loss in multi-task training. We fine-tune LUKE to extract quantity spans and we fine-tune RoBERTa to extract everything related to found quantities, including quantities themselves.

## introduction semeval-2021 task 8 consisted of five subtasks that covered span extraction, classification, and relation extraction tasks. this paper presents solutions to all five of them which showed the best results in the competition 1 . in the subtask 1(a) participants were asked to retrieve quantity (q) spans from texts. for example, in the following text "the soda can's volume was 355 ml.", the system should retrieve "355 ml" as q span. the rest of the subtasks were to extract information related to retrieved quantities (qs) from subtask a. the subtask 2(b) was to extract the unit of measurement (uom) of the extracted q and also to classify it into 10 classes: hastolerance, isapproximate, iscount, islist, ismean, ismeanhas-tolerance, ismeanisrange, ismedian, isrange, is-rangehastolerance. it should be noted that some 1 https://github.com/davletov-aa/meas-eval qs could be related to more than one type and there were ones which didn't belong to any type. the subtask 3(c) was to extract measured entity (me) and measured property (mp) spans. in the subtask 4(d) additional qualifier (qlfr) spans, which helped to validate or understand the extracted q, were asked to be extracted. and finally, subtask 5(e) was to extract relations between qs, mes, mps and qlfrs. more detailed information about the competition could be found in the @xcite 's shared task description paper. ## related work span extraction and classification problems have a long history of studies and are often studied as a part of named entity recognition (ner). for example, the ner dataset ontonotes v5 @xcite contains such entities as "quantity", which also includes measurements, and "money". however, the general ner approach used in ontonotes or @xcite datasets is not so fine-grained as the one that is used in the task under study. most state-of-the-art models for named entity recognition and relation extraction are based on transformer architecture by @xcite . for example, the top three best models for ontonotes v5 according to paperswithcode.com use bert foot_0 . bert is a large pre-trained language model based on attention @xcite . bert has a unique training procedure where the model is trained using masked language objective, where some tokens are replaced with a special '[mask]' token and the model should predict the original token. bert also had an additional training objective -the model had to predict whether a sentence was random or it followed the first sentence. however, some papers have investigated that bert is undertrained and that training bert on more data and for a longer time might increase model performance. roberta was one of the first and more influential papers of such kind @xcite . roberta modifies bert's pretraining procedure. the roberta model is trained longer, with bigger batches over more data and on longer sequences. roberta's authors have also found that removing the next sentence prediction objective from bert matches or slightly improves bert performance. researchers have also suggested ways of leveraging the nature of the task and adding some problem bias to named entity recognition. among such works, which is currently the best performing for ontonotes v5 and conll 2003 according to paperswithcode.com, is luke @xcite . the authors of luke have added a new language modeling task that consists of predicting randomly masked words and entities in an entity-annotated corpus retrieved from wikipedia. the authors have also expanded the selfattention mechanism to entity types and consider entity types when computing attention scores. the proposed approach allowed the authors to achieve state-of-the-art results not only for named entity recognition but for a bunch of other unrelated tasks such as squad1.1 question answering. for relation extraction, transformer-based models also outperform other approaches. a promising approach is treating relation extraction as a question answering problem. among works implementing this approach, we can mention @xcite where the authors restructured relation classification as a question answering (qa) like span prediction problem. it allowed them to get stateof-the-art results for tacred and semeval 2010 task 8 datasets. ## conclusion in this paper, we introduced our solution to semeval-2021 task 8: measeval: counts and measurements. our approach was based on roberta and luke models. we show that extracting mea-surements from a text can be treated as a questionanswering task. in this work, we tried a set of different models, hyperparameters, and weighting schemes and present their effect on the final result.