CL a C -np at S em E val-2021 Task 8: Dependency DGCNN (2021)

Abstract: MeasEval aims at identifying quantities along with the entities that are measured with additional properties within English scientific documents. The variety of styles used makes measurements, a most crucial aspect of scientific writing, challenging to extract. This paper presents ablation studies making the case for several preprocessing steps such as specialized tokenization rules. For linguistic structure, we encode dependency trees in a Deep Graph Convolution Network (DGCNN) for multi-task classification.

## introduction scientific articles contain many quantities which have to be linked to their measured entities. identifying quantities may seem as simple as digit recognition, but numbers alone are not informative. the entities and properties being measured, while crucial information, are difficult to extract. semeval 2021 task 8 @xcite is a semantic relation extraction task consisting of 5 subtasks: identifying quantities and their modifying attributes, identifying measured entities and their properties as well as qualifying attributes, if specified. recent reports on the strong performance of purely neural models for nlp tasks often underreport the data preprocessing and postprocessing steps that accompany them. preprocessing significantly influences overall performance. typical nlp preprocessing steps include sentence splitting and tokenization, sometimes followed by task relevant gazetteer annotation, possibly named entity recognition (ner), part-of-speech (pos) tagging and dependency parsing. these preprocessing steps are so common that many different packages perform them, such as stanford corenlp @xcite , spacy 1 , and nltk @xcite . we approach the measeval task as a multi-class classification task using a deep graph convolution neural network (dgcnn) @xcite , treating the dependency parse tree as a graph to convolve over. we explore tokenization variants, as well as encodings of the dependency relations using node2vec @xcite and umap @xcite techniques. ## problem statement the measeval @xcite task consists of 5 (not independent) sub-tasks covering span detection, classification and relation extraction across multiple sentences foot_0 . given a paragraph of scientific content in english, a system should: 1) label quantity spans (q) where q can be simple count or a numerical value with a unit. 2) if there is a unit it should labelled as unit(u), and a q should be classified into one of the types (count, approximate, range, list, mean, median, medianhassd, mean-hastolerance, rangehastolerance, hastolerance) as modifier(mod). 3) for each q, systems should identify the span of a measured entity (me) if one exists and also any measured properties (mp). 4) identify any spans of qualifiers (ql) that record additional detail related to q, me or mp. 5) label the relationships between q, me, mp and ql spans using hasquantity(hq), hasproperty(hp) and qualifies relation types. ## system overview motivation: unlike named entity detection tasks, measeval's me or mp detection depends on quan-tities and their relation to other tokens within a sentence. since dependency parse trees are capable of providing approximations of semantic relationships between predicates and their arguments, we opted to generalize over different dependency parse trees to obtain latent path representations to distinguish between different semantic connections that quantities have with mes or mps. to encode this path representation we use a graph convolutional networks (gcn) @xcite in the form of a dgcnn @xcite , to operate directly on the dependency graph to capture higher order neighborhood information in the form of embeddings. this embedding is used for classifying the relationship type between the tokens to detect one of the 6 classes explained in section 3.2.2 our system has 3 main phases: preprocessing, input creation and training a gcn model and post processing respectively. each phase communicates with the next through conll format @xcite files. ## results and analysis we split the combined training and trial datasets randomly into 75% training and 25% validation set resulting in 233 and 80 documents in our training and validation set respectively. we experimented with different preprocessing and feature representations using the official meas-eval evaluation script. dgcnn training parameters were fixed for all the experiments. we experiment with different path length limits (column 4: h). while a length limit of 8 showed better results on the development data than our competition limit of 5, the same is not true for the test data (see t: s: h: p r f1 em o c s u 5 .586 .466 .520 .281 .328 c s u 8 .563 .503 .532 .288 .336 n s u 5 .567 .485 .522 .291 .334 n s u 6 .591 .487 .535 .301 .344 n s u 8 .654 .529 .585 .337 .387 hasquantity and hasproperty received no attention during development and consistently scored 0 for runs with umap, see combined tokens: when cd(,cd),* and cd is followed by respectively, each list item corresponds to a different entity/property (e.g. this compares to signatures of accelerated electron precipitation from peaked electrons and "inverted-v" electrons, which occur on 9.8 and 3.4% of mex orbits, respectively.) as per gold annotation, 9.8 and 3.4% are 2 different qs both with "mex orbits" as me and "signatures of accelerated electron precipitation" as mps. we generate 9.8 and 3.4% as a single q and "signatures" as me, leading to several false negatives. documents s0032063312003054-2458, s0016236113008041-3257, s0378112713005288-1916 caused this error. math equalities our system does not protect math environments such as ...tetragonal unit cell with @xmath0.1816(4) å and @xmath1.0322(6) å... in document s0022459611006116-1351. consequently, a is labelled as a determiner (dt). as determiners are not part of gold labels, a is eliminated in postprocessing when it should have been labelled as an mp in this case and also in document s0022459611006116-1257. calculations with measurements our custom tokenizer does not handle calculations, as in "...re-vised average base reaction rate (from @xmath2..." in document s0019103512003533-3908. the gold annotation for q is (@xmath3, where we return "2×10" as a q with "average" as me, "1×10" as q with "reaction" as me and "9" (split by symbol "-") as seperate quantities with me as "reaction", incurring false positive errors. duplication of quantities lists of quantities including units are not combined with our custom tokenizer, thus for for 4.5 kg and 6 kg samples in document s0016236113008041-3257, we stipulate two different quantities. this results in duplication of quantities in our submission. 11 units the gold standard annotates for instance thin shale barriers (s1750583613004192-1126), das (s037842901300244x-1654), %∆e/e (s0301010413004096-767), and kloc (s016412121300188x-3207) as units. these are not included in our gazetteer list of 4028 units, incurring false negative errors. ## conclusions the measeval task is a challenging task that can benefit from a variety of tools in a well integrated system. the small data size limits a true appreciation of the challenges involved but ablation studies suggest that tokenization variations influence precision and recall differently and should be carefully considered in application systems. also, a umap reduction suggested a ca 1% performance boost on validation data, but incurs a ca 5% loss on test data, showing signs of overfitting. the subgraph classifier proved effective only for quantity and unit prediction. ablations show that larger subgraphs and longer paths lead to performance degradation, making a case for more task oriented locality features. duplicates have to be removed. while the unusual complexity of the classification task and the limited size of the dataset prohibits very general conclusions, we showed that dcgnns offer an interesting way to encode dependency information but that it has to be supported by several domain inspired contributions to work for all task components effectively.