S em E val 2022 Task 12: Symlink - Linking Mathematical Symbols to their Descriptions (2022)

Abstract: We describe Symlink, a SemEval shared task of extracting mathematical symbols and their descriptions from LaTeX source of scientific documents. This is a new task in SemEval 2022, which attracted 180 individual registrations and 59 final submissions from 7 participant teams. We expect the data developed for this task and the findings reported to be valuable for the scientific knowledge extraction and automated knowledge base construction communities. The data used in this task is publicly accessible at https://github.com/nlp-oregon/symlink .

## introduction the exponential growth of published articles may exceeds many readers' ability to keep track of the development of their field of interest. hence, automatic reading comprehension of scientific documents has attracted the attention of researchers across various domains such as drug discovery, knowledge base construction, and natural language processing. a crucial aspect of understanding scientific literature is understanding terminologies and formulae because they offer an explicit and precise interface to present the relation between scientific concepts @xcite . as such, a reading comprehension machine needs to (i) identify their descriptions and formulae, (ii) segment them into primitive terms and symbols, and (iii) link the associated terms and corresponding symbols. working with mathematical formulae is arduous due to two fundamental reasons. first, common text encodings such as ascii and unicode do not fully support typing mathematical symbols. as a result, complex mathematical formulae are rarely written using either ascii or unicode. rather, a higher level encoding (or typesetting) is often used to encode the content of scientific documents, in particular latex. second, most scientific documents are stored in one of two forms: photos or portable document format (pdf). scientific documents that were published prior to the graphical computer era are printed and now scanned and distributed as photos. nowadays, scientific documents are often composed in some text editors or word processing software, then exported and shared a pdf file. unfortunately, analyzing textual information in photo images or pdf files is extremely difficult, and most of the natural language processing tools are not developed to handle this format. as such, to facilitate the understanding of scientific literature, documents should be stored using a universal easy-to-process text-like encoding. in this paper, we use latex as the typesetting to facilitate document analysis. thanks to recent advances in text processing and image recognition, a latex document can often be restored to some extent from either a photo or a pdf file @xcite . this paper introduces the symlink shared task for the extraction of mathematical symbols and their descriptions from english scientific documents using their latex source. figure 1 visualizes an example of the task. this paper also presents an analysis of the results of participant systems on the task. the rest of the paper is organized as follows. section 2 presents related work in extracting formulae and their related information from scientific documents. section 3 describes the subtasks of this symlink shared task. section 4 presents the data creation process including data sources, preprocessing, annotation guidelines, annotation, and data format. an analysis of the created data set is provided in section 5. the evaluation method is presented in section 6, while the descriptions of the submitted systems are presented in section 7. ## related work early studies for scientific literature link formulae to wikipedia page @xcite . even though this can provide additional information regarding the mathematical expression, a reader might find it harder to understand the wikipedia page as it is presented in many unrelated forms. linking to the description in the same document is more practical @xcite as the descriptions are dedicated to the symbols and the context presented in the document. previous studies on symbol-description extraction rely on pattern matching @xcite and rule-based algorithms @xcite . these methods might work for observed patterns with an assumption of close proximity between symbol and description. they may fail to capture distant symboldescription pairs and symbols in very complex structures such as algorithms in computer science literature. most of the previous studies have attempted to extract and link at formula level @xcite @xcite . in reality, understanding mathematical formulae requires details of atomic symbols e.g. superscript, subscript, function arguments. we believe that addressing the problem at this fine-grain level is crucial to drive future research toward a better understanding of the complex symbol-description extraction task. prior to this shared task, some studies have created datasets for similar tasks @xcite @xcite . however, one of them is created for publications written in japanese @xcite , making it nearly impossible to transfer to english literature. while two other datasets @xcite ) only annotate small-scale golden datasets for evaluation purposes. as the result, no training data is available for training deep neural network models. in this shared task, we provide a large-scale dataset for english literature that we believe will provide enough supervision for the promising deep neural network-based models. definition extraction from scientific document is close to the task presented in semeval task 12. the scientific document understanding workshop has hosted the acronym extraction and acronym disambiguation shared tasks, namely acronym extraction and acronym disambiguation shared tasks @xcite . the prior studies in this research direction considers extracting definitions from the text @xcite @xcite , or together with acronyms, and acronyms sense disambiguation (pouran ben veyseh et al., 2020, 2021). ## task description the ultimate goal of symlink shared task is to extract pairs of mathematical symbols and descriptions from scientific documents. as such, symlink shared task is a combination of an entity recognition and an entity linking task. given a latex source of a paragraph from a scientific document: ## data analysis figure 3 presents the distribution of the span lengths of both symbols and descriptions of up to 15 tokens. as can be seen from the figure, the majority of entities have a length of 1-3 tokens. however, overall, the span lengths of both symbols and descriptions vary significantly from 1 up to 47 tokens (note that figure 3 only illustrates the spans { " i d " : " 1 5 0 3 . 0 1 1 5 8 v2 . . . " , " p h a s e " : " t e s t " , " t o p i c " : " c s . a i " , " document " : " 1 5 0 3 . 0 1 1 5 8 v2 . . . " , " p a r a g r a p h " : " p a r a g r a p h 4 8 " , " t e x t " : " . . . w i t h a c o v a r i a n c e m a t r i x o f @xmath0 ; t h a t i s , . . . " , " e n t i t y " : { " t1 " : { " e i d " : " t1 " , " l a b e l " : "symbol" , " s t a r t " : 3 2 5 , " end " : 3 2 6 , " t e x t " : " i " } , " t2 " : { " e i d " : " t2 " , " l a b e l " : "primary" , " s t a r t " : 3 0 3 , " end " : 3 2 0 , " t e x t " : " c o v a r i a n c e m a t r i x " } } , " r e l a t i o n " : { "r1 " : { " r i d " : "r1 " , " l a b e l " : " d i r e c t " , " a r g 0 " : " t2 " , " a r g 1 " : " t1 " } } } with up to 15 tokens). this demonstrates a key challenge of the symbol-description linking task in this paper where symbols and descriptions with long spans might introduce confusion for extraction models. to further understand the dataset, we present the distances between the entities and relations annotated in symlink by different relation types in figure 4 . the distributions can be grouped into two categories. the first category involves the symboldescription relations while the second group involves the coreference relations. the distributions of symbol-description relations have long tails, indicating that symbols and descriptions tend to ap- ## evaluation the results are evaluated separately for the named entity recognition (ner) task and the relation extraction (re) task. for ner, we use the entitybased partial/type from semeval 2013 task 9.1. for re, we use standard precision, recall, f-score metrics. relations output by the participating system is correct if the prediction label strictly matches the gold standard. during the 21-day evaluation period (january 10 through 31, 2022), 7 codalab users submitted a total of 59 submissions with 37 submissions passing the validation and being scored. given the complexity of the task, we allow unlimited submissions during the evaluation. as such the top submitter tried up to 18 times. ## summary of participating systems the symlink track at semeval-2022 received 4 system description paper submissions presented in