GLADIS : A General and Large Acronym Disambiguation Benchmark (2023)

Abstract: Acronym Disambiguation (AD) is crucial for natural language understanding on various sources, including biomedical reports, scientific papers, and search engine queries. However, existing acronym disambiguationbenchmarks and tools are limited to specific domains, and the size of prior benchmarks is rather small. To accelerate the research on acronym disambiguation, we construct a new benchmark with three components: (1) a much larger acronym dictionary with 1.5M acronyms and 6.4M long forms; (2) a pre-training corpus with 160 million sentences;(3) three datasets that cover thegeneral, scientific, and biomedical domains. We then pre-train a language model, AcroBERT , on our constructed corpus for general acronym disambiguation, and show the challenges and values of our new benchmark.

## introduction an acronym is an abbreviation formed from the initial letters of a longer name. for instance, the following two sentences contain the acronym "ai": acronym disambiguation (ad) is the task of mapping a given acronym in a given sentence to the intended long form. acronym disambiguation is crucial for downstream tasks such as information extraction, machine translation, and query analysis in search engines @xcite . acronym disambiguation is also important for humans: acronyms may make a text more difficult to understand for readers who are not familiar with the specific domain. a study on a microsoft question answering forum found that only 7% of the acronyms co-occur with their corresponding long forms, which confuses the readers about the meaning of a text @xcite . acronym disambiguation has received more attention in the past few years. the first step in acronym disambiguation is usually the creation of a dictionary, i.e., a mapping of each acronym to one or more long forms. early systems extracted acronyms and their definitions automatically from texts by @xcite or supervised @xcite methods. once a dictionary is available, acronym disambiguation methods expand acronyms in a given text by capturing the contexts for specific domains, e.g., the enterprise domain @xcite , biomedical texts @xcite , and scientific papers @xcite . @xcite was the first general and web-based system, recognizing and expanding acronyms across multiple domains. several benchmarks have also been constructed, including for the biomedical area @xcite and the scientific area @xcite . several methods finetuned scibert @xcite on sciad to disambiguate acronyms in scientific documents @xcite @xcite . although these works have significantly advanced the progress of acronym disambiguation, they suffer from three main limitations. first, most existing dictionaries (and benchmarks) focus on one specific domain. in real-world applications, however, the input text may be general, cross-domain, or of an unspecified domain (as in search engine queries). second, existing dictionaries are limited in size. for example, there are only two long forms for the acronym "ai" in sciad ( with this work, we aim to improve acronym disambiguation along two dimensions: first, we automatically construct gladis, a general and large acronym disambiguation benchmark that includes a larger dictionary, a pre-training corpus and three datasets covering the general, biomedical, and scientific domains. our dictionary contains 1.5m acronyms and 6.4m long forms, which trumps existing dictionaries by a factor of 3. we complement this dictionary by three domain-specific datasets for acronym disambiguation, which are adapted from three existing human-annotated and crowd-sourced datasets @xcite @xcite . the pre-training corpus has 160 million sentences with acronyms, collected from the pile dataset @xcite with a rule-based algorithm @xcite . second, we propose acrobert, the first pre-trained language model for general acronym disambiguation. our experiments show that this model outperforms existing systems across multiple domains. our code and data are available at https://github.com/tigerchen52/gladis . ## constructing gladis our gladis benchmark consists of three components: a dictionary, a pre-training corpus, and three domain-specific datasets. ## acrobert we can now capitalize on our dictionary and pre-training corpus to propose a new method for acronym disambiguation. it takes as input (1) a dictionary of acronyms with their long form(s), and (2) a large-scale corpus that contains acronyms (we assume that the boundaries of the acronym have already been recognized). our goal is to pretrain a language model for acronym disambiguation, which has a strong generalizability across multiple domains. the pre-training strategy of bert. we adapt the bert model for our purpose. bert is pretrained by using two unsupervised tasks, masked language model (mlm) and next sentence prediction (nsp). the masked language model task randomly masks some percentage of the input tokens, and then forces the model to predict the masked tokens, similar to a cloze task. the next sentence prediction task asks the model to predict whether one sentence follows the other. the next sentence prediction task can be used to predict, from the input text (e.g., "this is the product's first true ai version, and it understands your voice instantly."), the correct long form ("artificial intelligence"). here, the model learns to judge whether the input context that contains the acronym "ai" is coherent with the long form "artificial intelligence". the masked language model task can memorize the correlation of tokens between the context sequence and long form. thus, the model learns that the phrase "artificial intelligence" often co-occurs with "product" or "understand". however, we find that this naive technique does not perform well (see the ablation studies in table a4 ). we believe that the reason is that the acronym is usually ambiguous with many candidates (as shown in acrobert. to mitigate the weaknesses of the original bert, we pre-train an adapted bert, called acrobert, by slightly adjusting the next sentence prediction task. the framework is shown in figure 2 . it aims to bring the positive sample pairs closer together, and to push apart the negative sample pairs. we find that already such a simple model can perform very well. for each pair of a candidate long form and a sentence with an acronym, we compose an input for the next sentence prediction task as "[cls] long form [sep] sentence [sep]". then, we obtain representations of this sequence by applying bert to this new input. the final hidden vector e [cls] ∈ r h of the first input token ([cls]) is used as the aggregate representation, where h is the dimension of the hidden vector. next, the scores for the binary classification are: the negatives in this triplet framework can be randomly sampled from the dictionary. however, we observe that such random negatives contribute less to the training and result in slower convergence because the initial model can easily distinguish these triplets. therefore, it is crucial to select harder triplets that are active and beneficial to the training. for this purpose, we introduce a certain number of ambiguous negatives to each mini-batch, e.g., "artificial intelligence" can be added to the input context as an ambiguous negative sample for the positive pair "adequate intake [sep] in the united states, the ai for potassium for adults is 4.7 grams." through the pre-training step, acrobert is able to identify the correct long form with the most consistent theme from numerous candidates based on the input context. ## experiments in this section, we compare acrobert empirically to other acronym disambiguation approaches. ## conclusion in this paper, we have presented gladis, a challenging benchmark for acronym disambiguation, which includes a larger dictionary, three datasets from the general, scientific, and biomedical domains, and a large-scale pre-training corpus. we have also proposed acrobert, a bert-based model that is pre-trained on our collected acronym documents, which can significantly outperform other baselines across multiple domains, and which is more robust in the presence of very ambiguous acronyms and overshadowed samples. for future work, we aim to enhance the performance of ac-robert on the overshadowed cases, which is crucial for the acronym disambiguation task.