Extracting Mathematical Concepts from Text (2022)

Abstract: We investigate different systems for extracting mathematical entities from English texts in the mathematical field of category theory as a first step for constructing a mathematical knowledge graph. We consider four different term extractors and compare their results. This small experiment showcases some of the issues with the construction and evaluation of terms extracted from noisy domain text. We also make available two open corpora in research mathematics, in particular in category theory: a small corpus of 755 abstracts from the journal TAC (3188 sentences), and a larger corpus from the nLab community wiki (15,000 sentences)

## introduction the majority of scientific research is communicated using natural language, often in the form of papers like this one. however, the volume of scientific literature in any given field is too large to be completely understood by any one individual. so how can expert researchers, let alone newcomers or outsiders, come to terms with the breadth of scientific knowledge in their field? recently, nlp tools have become stunningly effective at making information that is relevant to everyday concerns more accessible. tools for search, question answering, and summarization have improved significantly on various general benchmarks. to make research more effective and accessible, similar tools are needed for specialized domains. some research communities might number only in the thousands of researchers, and have specialized vocabulary and language usage, including heavy use of symbols, diagrams and/or markup 1 certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. such identification is not intended to imply recommendation or endorsement by the national institute of standards and technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose. we define the notion of a torsor for an inverse semigroup , which is based on semigroup actions , and prove that this is precisely the structure classified by the topos associated with an inverse semigroup . unlike in the group case, not all set-theoretic torsors are isomorphic : we shall give a complete description of the category of torsors ... language, as in mathematics. these smaller communities require a general methodology for constructing specialized tools themselves. knowledge graphs-networks of concepts and their relations in a particular domain of knowledge-have become the preferred technology for representing, sharing, and adding knowledge to modern ai applications @xcite . the construction of such a graph begins with the identification of central concepts in the domain in question. given a corpus of text, such as a collection of papers, the task of identifying these central concepts is sometimes known as term extraction, and there are many generic toolkits for performing this task. in this paper we study four examples: textrank @xcite , dygie++ @xcite , opentapioca @xcite , and parmenides @xcite . a potential methodology to construct specialized, domain-specific knowledge management tools would begin by running a generic term extractor over a suitable corpus of domain-specific text and assuming that it extracts a reliable set of terms. however, each research community may wish to evaluate these terms to test whether they meet the community's specific needs. this evaluation must determine how well the underlying terms reflect important concepts in the domain. ideally, such an evaluation would be made against a corpus annotated by human experts, which would provide a gold standard reference for a representative sample of the domain. such a corpus would ideally capture all and only the relevant concepts present in the corpus, allowing evaluation based on both precision and recall. however, obtaining a hand-annotated reference corpus is not always practical, especially with noisy data. first, hand annotation is time-consuming, and may be infeasible for certain research communities. second, the specialized nature of the text means that the annotators will need to be experts in the domain. this makes hand annotation potentially very expensive for highly specialized domains. in particular, we are also seeking a methodology that can be undertaken with little to no additional direct effort from domain experts, and hand annotation does not meet this criterion. what, then, does a methodology for constructing and evaluating extracted terms look like for specialized research domains? in this paper we propose an evaluation methodology that combines information from different 'silver standard' sources. in our case, we study author-selected keywords from paper abstracts, titles from a community-managed wiki, and linguistically identified noun phrases. we argue that, in the case that traditional f 1 scores are not informative enough when drawn from any individual source, the evaluation of several sources nonetheless gives us valuable information about the properties of terms extracted. we apply this methodology to evaluate lists of terms extracted from text in the mathematical field of category theory. by analyzing the results, we see that generic tools do not have their full efficacy on the specialized domain of category theory, and we have the grounds to infer some reasons why. nonetheless, this amalgamated evaluation method provides a path forward for constructing and maintaining a high quality list of domain-specific concepts in category theory. a key result of this paper is also the groundwork we lay, including two small corpora and some basic experiments, for understanding how nlp tools can be used to build a knowledge graph for mathematics. ## category theory as a case study although we seek to develop a generic methodology, we have chosen to ground these investigations in the specific field of category theory. category theory is a branch of mathematics focused on relationships and composition. it is often seen as a way to organize mathematics as a whole @xcite . while this choice is largely dictated by the interests of the authors, category theory presents a number of features which reflect the challenges and potential of automatically constructing domainspecific knowledge management tools. category theory as a field dates back to the 1940s @xcite . while the field is well established, the volume of text available remains small compared to the corpora used in other nlp applications. a leading journal in the field, theory and applications of categories (tac), published 55 papers in 2021, and a total of 845 papers since its first issue in 1995. this is small compared to, for example, the 3.27 million materials science abstracts used to train the nlp backend for the materials science search engine matscholar @xcite . most of category theory research is described in natural language, especially english. however, the language is specialized in ways that may pose challenges to automatic systems: though the category theory community is relatively small, it has a large online presence, which has supported the creation of community-oriented websites and blogs, including the nlab, a wiki for notes, expositions, and collaborative work, with a focus on category theory. the nlab was started in 2008, and as of may 2022, has over 16000 articles. the authors' own interest and expertise in category theory also allows us to quickly analyze the results of any experiments from the perspective of a potential user. ## automatic term extraction algorithms we run a number of experiments to test four different automatic terminology extraction methods: opentapioca (a simple entity linking system designed specifically for category theory), dygie++ (a neural ner system that has been trained to extract scientific terms), textrank (a graph-based algorithm originally designed for key phrase extraction, but adapted to ate), and parmenides (a linguistically-motivated phrase extraction system that combines symbolic processing and neural parsing). opentapioca: opentapioca (delpeuch, 2020) is a simple named entity linking system that links phrases of natural language text to entities in wiki-data @xcite . it cannot identify new concepts-only those already represented in wikidata. opentapioca is a simple baseline system that uses basic string matching to identify relevant phrases, built on the recognition that powerful knowledge bases like wikidata has led to recent success in other systems. opentapioca is of particular interest, because it is designed to link entities that are not just locations, dates, or the names of people and organizations, but a variety of technical concepts. opentapioca also provides a filter that allows it to limit results to entities that appear in nlab, effectively filtering out concepts that are not related to category theory. dygie++: dygie++ @xcite is a span-based neural scientific entity extractor. the system builds upon the older dygie @xcite . both systems were developed in collaboration with the allen institute for artificial intelligence, and use supervised methods to identify relevant spans of text. dygie++ has been trained on a variety of different corpora and subtasks, including the identification of chemical compounds, drug names, and mechanisms. though dygie++ has not been trained or tested directly on category theory, the similarities between the domains it has been trained on and ct, as well as its overall strong performance, make it a good candidate to test for extracting ct concepts. textrank: textrank @xcite ) is a graph-based ranking algorithm based on pagerank, which has been applied to keyword extraction and text summarization as well as automatic terminology extraction. though textrank is a somewhat older algorithm, it is still a common algorithm that has been implemented many times. we use a modern python implementation, pytextrank foot_3 . parmenides: parmenides @xcite takes a linguistic approach to terminology extraction. it uses spacy foot_4 to identify syntactic structures, then normalizes the syntactic structure and identifies phrases for extraction. parmenides is highly customizable, but is designed primarily for linguistic analysis and not for terminology extraction. nevertheless, it can be used to identify key linguistic phrases as an initial step for ate. ## test corpus automatic terminology extraction takes a corpus of natural language text and produces a list of relevant terms. to produce a list of terms for category theory, we need to supply a corpus of category theory text. to create such a corpus, we take abstracts from theory and applications of categories (tac). this is the primary corpus that we use for our experiments. we also provide a second corpus, using a subset of the nlab wiki foot_5 . these corpora will be made publicly available. we remove markup, section headings, and l a t e x expressions from the text to create a cleaned version of the corpus. both corpora are written in english. after cleaning the corpora, we run spacy to produce automatic annotations in the style of conll-u. spacy is a free open-source library for natural language processing in python distributed since 2015. it features named entity recognition (ner), part-of-speech (pos) tagging, dependency parsing, and word vectors. note that these are the first publicly available category theory corpora, and we are not aware of any ## evaluation methodology the ate systems described in section 3, combined with the tac corpus described in 4, allow us to construct candidate lists of category theory concepts, which could be used as the basis for a knowledge graph. we now arrive at the central question of this paper: how do we assess the quality of such lists? again, our goal is not to assess the quality of the extraction algorithms as generic tools, but rather to assess the quality of the lists of category theory concepts they produce. this is a key distinction: our goal is not generality, but the evaluation of data in a particular context. more precisely, the usual methodology @xcite would be to construct an expertly annotated corpus, labeling all the category theory concepts contained within it. we could then compare the list of terms produced by the term extractors against the gold standard, to produce standard metrics such as precision, recall, and f 1 score. as described above, this methodology can be expensive and impractical for small, highly technical research communities. instead, we seek to evaluate against multiple, imperfect sources of truth to discern different properties of the data. to compensate for the imperfect nature of our reference lists, we must pair each one with a qualitative description of the properties it can reveal. this allows us to use the list to shed light on the nature of the concepts under evaluation, even if a single, representative score cannot be constructed. the reference lists we consider for this paper are described in ## reference lists we now discuss in more detail the properties of the reference lists we have chosen for category theory. figure 2 shows the overlap of terms found between the three reference lists. ## discussion overall, the f 1 scores presented here are very low when compared to the results of semeval 2017 @xcite . dygie++ also reports higher numbers on the datasets it has been trained on @xcite . our results are, however, similar to the results of @xcite , which considers the problems of terminology extraction using papers indexed in citeseerx, which reports f 1 scores of 0.33. parmenides always outperforms the other models we consider on recall, but generally performs poorly on precision. conversely, opentapioca has relatively high precision scores, resulting in the highest f 1 score for both author keywords and nlab page titles. parmenides was designed as a linguistic analysis tool; it extracts all possible phrases, with only limited power to rank those phrases by relevance. as a result, it extracts almost all of the linguistic units that are available, including large amounts of irrelevant text. opentapioca, on the other hand, is designed to pull out only category theory concepts, but is limited in its ability to extract novel terms and those not described in nlab. the terms extracted by dygie++ are reasonable in terms of f 1 score. for author-selected keywords, dygie++ performs the best, and it has the secondhighest f 1 score for nlab page titles. however, it is not enough to just consider f 1 scores in this case. the reference lists that we consider have limitations, and we cannot rely on them all to be both complete and precise. as described above, the author keywords and nlab titles have limited recall-they do not contain all of the possible category theory terms in the text, because they are designed for other purposes. for these reference lists, we can only rely on the recall of the extracted terms. low recall on the author keywords indicates that a list does not contain many of the advanced concepts from category theory, while low recall on the nlab titles indicates that a list does not contain many of the basic concepts from category theory. low precision on these, however, may indicate that a list contains terms which may still be valid, but which do not appear in these reference lists. the proper conclusion, then, should not be that opentapioca is the best option because it has the best overall f 1 score. nor is dygie++ necessarily ideal just because of its high performance on author- ## conclusions we present the first computational work extracting mathematical concepts from abstracts. we investigated four different term extractors, previously described for other domains, and evaluated the results against the limited annotated data we had for category theory. the results are somewhat limited as well, compared to previous results on more generic domains. however, other domain-specific analyses have some of the same problems, which suggests that our results are still promising. we also provide insight into the evaluation of automatically-generated terminologies for limitedresource domains. the usual f 1 scores are not entirely reliable unless the gold standard can be assumed to include both all and only the relevant terms, but partially-correct 'silver standards' may still provide useful insight into the data. in our case, we can draw some important con-clusions about the terminology lists that we extract. because author keywords and nlab titles are most reliable for recall, we can determine that tools such as parmenides and textrank are able to extract large quantities of both advanced and basic category theory terms. however, the large number of other terms extracted by parmenides suggests that it may need additional filtering to be useful for automatic terminology extraction for our usecase. our evaluation can also be further improved. the low relative recall of the noun phrase reference list itself suggests that additional phrase types are common in our data. adding verb phrases and more complex noun phrases could help us identify high-precision terminologies, as well as high-recall ones. another possibility in our case is to continue working toward our use-case. since we have further downstream uses of the terminology-namely, the creation of a knowledge graph-we can use this to further our evaluation. by extracting relations between terms, we can identify which terms are the most connected and which are isolated, under the assumption that isolated terms are less likely to be part of domain-specific language. we have also constructed two publicly-available corpora that can be developed into more sophisticated datasets. though there are still many limitations to both evaluation and ate in mathematics, we hope that our work provides a basis for future developments in the area, and that our insights on evaluation and domain-specific research can be ap-plied more generally.