What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation (2020)

Abstract: Acronyms are the short forms of phrases that facilitate conveying lengthy sentences in documents and serve as one of the mainstays of writing. Due to their importance, identifying acronyms and corresponding phrases (i.e., acronym identification (AI)) and finding the correct meaning of each acronym (i.e., acronym disambiguation (AD)) are crucial for text understanding. Despite the recent progress on this task, there are some limitations in the existing datasets which hinder further improvement. More specifically, limited size of manually annotated AI datasets or noises in the automatically created acronym identification datasets obstruct designing advanced high-performing acronym identification models. Moreover, the existing datasets are mostly limited to the medical domain and ignore other domains. In order to address these two limitations, we first create a manually annotated large AI dataset for scientific domain. This dataset contains 17,506 sentences which is substantially larger than previous scientific AI datasets. Next, we prepare an AD dataset for scientific domain with 62,441 samples which is significantly larger than previous scientific AD dataset. Our experiments show that the existing state-of-the-art models fall far behind human-level performance on both datasets proposed by this work. In addition, we propose a new deep learning model which utilizes the syntactical structure of the sentence to expand an ambiguous acronym in a sentence. The proposed model outperforms the state-of-the-art models on the new AD dataset, providing a strong baseline for future research on this dataset.

## introduction acronyms are shortened forms of a longer phrase. as a running example, in the sentence "the main key performance indicator, herein referred to as kpi, is the e2e throughput" there are two acronyms kpi and e2e. also, the acronym kpi refers to the phrase key performance indicator (a.k.a. the long form of the acronym kpi). in written language, acronyms are prevalent in technical documents that helps to avoid the repetition of long and cumbersome terms, thus saving text space. for instance, about 15% of pubmed queries include abbreviations, and about 14.8% of all tokens in a clinical note dataset are abbreviations @xcite @xcite . considering the widespread use of acronyms in texts, a text processing application, such as question answering or document retrieval, should be able to correctly process the acronyms in the text and find their meanings. to this end, two sub-tasks should be solved: acronym identification (ai): to find the acronyms and the phrases that have been abbreviated by the acronyms in the document. in the running example, the acronyms kpi and e2e and the phrase key performance indicator should be extracted. acronym disambiguation (ad): to find the right meaning for a given acronym in text. in the running example, the systems should be able to find the right meanings of the two acronyms kpi and e2e. note that while the meaning of kpi is found in the sentence, the meaning of e2e should be inferred from its occurrences either in the previous text of the same document or external resources (e.g., dictionaries). acronym identification and disambiguation can be used in many downstream applications including slotfilling @xcite , definition extraction @xcite , and question answering @xcite over the past two decades, several approaches and resources have been proposed to solve the two subtasks for acronyms. these approaches extend from rule-based methods for ai and feature-based models for ad (i.e., svm and naive bayes) @xcite @xcite to the recent deep learning methods @xcite @xcite @xcite . while the prior work has made substantial progress on this task by providing new approaches and datasets, there are some limitations in the existing datasets which hinder further improvement. first, most of the existing datasets for ai are either limited in their sizes or created using simple rule-based methods (i.e., not humnanannotated). for instance, @xcite exploits the rules proposed by @xcite to generate a corpus for acronym disambiguation from wikipedia. this is unfortunate as rules are in general not able to capture all the diverse forms to express acronyms and their long forms in text @xcite . this limitation not only restricts the coverage of the previous methods and datasets for ai, but also imposes some bias to evaluation of ai using these datasets. second, most of the existing datasets are in the medical domain, ignoring the challenges in other scientific domains. while there are a few datasets for general domain (e.g., wikipedia), online forums, news and scientific documents @xcite @xcite @xcite , they still suffer from either noisy examples inevitable in the heuristically generated datasets @xcite @xcite or their small size, which makes them inappropriate for training advanced methods (e.g., deep neural networks) @xcite @xcite . in order to address the limitations of the existing resources and push forward the research on acronym identification and disambiguation, especially in scientific domain, this paper introduces two new datasets for ai and ad. notably, our datasets are annotated by human to achieve high quality and have substantially larger numbers of examples than the existing ai datasets in the non-medical domain (see appendix b.3). moreover, based on acronyms and long forms obtained by ai annotations, we introduce a new ad dataset that is also substantially larger than the existing ad datasets for the scientific domain. for more details about the datasets, see appendices b.2 and b.3. finally, we conduct extensive experiments to evaluate the state-of-the-art methods for ai and ad on the proposed datasets. the experiments show that the existing models fail to match the human-level performance on the proposed datasets. motivated by the unsatisfactory performance of the existing ad models, we introduce a new deep learning model for acronym disambiguation. in particular, one of the limitations of the prior ad models is that they fail to efficiently encode the context of the ambiguous acronyms in the given sentences. on the one hand, the traditional feature-based models for ad are limited in their ability to effective representations for the contexts due to the use of hand-designed features. on the other hand, the current deep learning models for ad, based on either language models or sentence encoders (e.g., lstms) to capture the contextual information, cannot effectively encode the long dependencies between words in the sentences. these long dependencies are important for acronym disambiguation as the meanings of the acronyms might depend on some words that are sequentially far from the acronyms of interest. to address the issue of long dependencies in sentences, prior work on other natural language processing (nlp) applications has shown that dependency trees can be employed to capture dependencies between words that are sequentially far from each other @xcite . as none of the recent ad models employs the dependency trees to encode the sentence context for acronyms, we propose a novel deep learning model for ad that exploits the syntactic dependency structures of the sentences using graph convolutional neural networks (gcn) @xcite . in this work, the structure information induced by the gcn model would be employed as the structural context for acronym disambiguation. our experiments show that the proposed model outperforms the existing state-of-the-art ad models, providing a strong baseline for the future work on the proposed dataset for ad. in summary, in this paper, we make the following contributions: ## data collection and annotation in order to prepare a corpus for acronym annotation, we collect a corpus of 6,786 english papers from arxiv. these papers consist of 2,031,592 sentences that would be used for data annotation for ai and ad in this work. ## model in this section we describe the details of the proposed model for ad. as mentioned in section 2.2, we formulate this task as sequence classification problem. formally, given an input sentence @xmath0, w 2 , ..., w n and the position of the acronym, i.e., p, the goal is to disambiguate the acronym w p , that is, predicting the long form l from all the possible long forms of w p specified in the dictionary d. the proposed model consists of three major components: (1) sentence encoder: this component encodes the input sentence into a sequence of representation vectors for the words, (2) context encoder: in order to augment the representation of the acronym w p , the context encoder component is designed to leverage the syntactical structure of the sentence to generate the context representation vector for w p , and (3) prediction: this component consumes the representations obtained from previous components to predict the long form l for the acronym w p in the sentence. in this section, we describes these components in details. ## experiments datasets & evaluation metrics: we evaluate the performance of the state-of-the-art acronym identification and disambiguation models on sciai and sciad, respectively. for ad, in addition to sciad, we evaluate the performance of the models on the uad wikipedia dataset proposed by @xcite . uad consists of sentences in general domain (i.e., wikipedia). as the domain difference between sciad and uad (i.e., scientific papers vs wikipedia articles) could effect on the direct comparison between the performances of the models, we also prepare sciuad dataset for acronym disambiguation foot_2 . sciuad employs the same corpus as we use to create sciad but the acronyms, long forms, their mappings, and the ambiguous use of acronyms in corpus are all extracted using the unsupervised method proposed by uad @xcite . in sciad and sciuad indicates that our dataset preparation is more effective to capture acronym ambiguity in the given corpus than uad. second, comparing the ratio of sentences that have overlap with the long forms emphasizes the difference between scientific domain and general domain. the higher overlap ratio of sciad and sciuad compared to uad could be attributed to two characteristics of the scientific domain: 1) sentences in scientific papers are normally longer than sentences of general articles and it is corroborated by comparison between average sentence length in the three datasets shown in to create training and testing splits of the datasets, we randomly divide sciai, sciad and uad into training, development and test data using 80:10:10 ratio. regarding the evaluation metrics, for ai, a prediction of acronym or long form is counted as true if the boundaries of the prediction matches with the boundary of the ground-truth acronym or long form in the sentence, respectively. we report the macro-averaged precision, recall and f1 score computed for acronym and long form prediction. for ad, similar to prior work @xcite , we report the performance of the models using macroaveraged precision, recall and f1 score computed for each long form. baselines: for acronym identification, we compare the performance of the following baselines on sciai: (1) rule-based methods: these models employ manually designed rules and regular expressions to extract acronyms and long forms in text; namely we report the performance of noa @xcite , uad (ciosici et al., 2019) foot_3 and ade @xcite , (2) feature-based models: these models define a set of features for acronym and long form predictions, then they train a classifier (e.g., svm or crf) using these features, namely we report the performance of bioadi @xcite and lncrf @xcite , (3) deep learning models: since to the best of our knowledge, none of the existing work leverage pre-trained word embeddings with deep architectures for ai, we implement an lstm-crf baseline for this task. for more details on the lstm-crf baseline and hyper parameters, see appendices a and b. for acronym disambiguation, we compare the performance of the proposed model, i.e., gad, with the following baselines: (1) non-deep learning models: this category includes two models: (a) "most frequent" (mf) which takes the long form with the highest frequency among all possible meanings of an acronym as the expanded form of the acronym, and (b) a feature-based model that employs hand crafted features from the context of the acronyms to train a disambiguation classifier; namely ade @xcite , (2) deep learning models: in this category, we report the performance of (1) language-modelbased baselines that train the word embeddings using the training corpus, namely noa @xcite and uad @xcite , and (2) models employing deep architectures (e.g., lstm), namely we report performance of decbae @xcite and bem (blevins and zettlemoyer, 2020) foot_4 . ## analysis this sections provides further analysis on the annotation process and the proposed datasets and model. we first discuss the common errors in annotations that result in conflicts. afterwards, we provide a sample complexity analysis of the ad task. case study is presented in the end. ## related work in the last two decades, several work has been proposed for acronym identification and disambiguation. for ai, most of the prior work employs rule-based methods @xcite @xcite @xcite @xcite or feature engineering @xcite . majority of the recent work leverages the rules proposed by @xcite to extract long forms from the sentences containing acronyms. recently, it has been shown that the rule-based methods fall far behind human level performance, especially in scientific domain @xcite . in addition to the rule-based methods, some work utilizes users' search experience to identify the meanings of acronyms @xcite @xcite . however, this method cannot be applied to non-web-based resources (e..g, scientific papers). for acronym disambiguation, prior work employs either feature-based models @xcite or deep learning models. while some of the deep learning models directly employ word embeddings for disambiguation @xcite @xcite , some of them employ deep architectures to encode the context of the acronym @xcite . moreover, acronym disambiguation has been also modeled as the more general tasks word sense disambiguation (wsd) @xcite or entity linking (el) @xcite . while the majority of the prior work studies ad in medical domain @xcite @xcite , recently some work proposes acronym disambiguation in general @xcite , enterprise @xcite , or scientific domain @xcite . ## conclusion we introduce new datasets and strong baselines for ai and ad sub-tasks. these datasets are created based on human annotation and shown to be more challenging than their previous counterparts. while the baselines achieve good performance, especially the proposed gad model which outperforms the prior models for ad, more research is required to reach to human level performance for the new datasets. dataset train dev. test sciai 14,006 1,750 1,750 sciad 49,788 6,233 6,225 uad 459,869 57,484 57,484 sciuad 15,982 1,997 1,999