m -Networks: Adapting the Triplet Networks for Acronym Disambiguation (2022)

Abstract: Acronym disambiguation (AD) is the process of identifying the correct expansion of the acronyms in text. AD is crucial in natural language understanding of scientific and medical documents due to the high prevalence of technical acronyms and the possible expansions. Given that natural language is often ambiguous with more than one meaning for words, identifying the correct expansion for acronyms requires learning of effective representations for words, phrases, acronyms, and abbreviations based on their context. In this paper, we proposed an approach to leverage the triplet networks and triplet loss which learns better representations of text through distance comparisons of embeddings. We tested both the triplet network-based method and the modified triplet network-based method with m networks on the AD dataset from the SDU@AAAI-21 AD task, CASI dataset, and MeDAL dataset. F scores of 87.31%, 70.67%, and 75.75% were achieved by the m network-based approach for SDU, CASI, and MeDAL datasets respectively indicating that triplet network-based methods have comparable performance but with only 12% of the number of parameters in the baseline method. This effective implementation is available at https://github.com/sandaruSen/m_networks under the MIT license.

## introduction natural language is often ambiguous and contains phrases, words, acronyms, and abbreviations which have more than one meaning @xcite . the complexity of natural language is further augmented based on which context these words are being used @xcite . scientific and medical communities use domain specific technical terms, which are often shorthanded for ease of use. this has resulted in the prevalence of acronyms in scientific and medical documents @xcite . to understand these expert texts, it is important to disambiguate the meaning of their acronyms. for example, given a sentence with the acronym rnn, the possible expansion for the acronym can be recurrent neural network, random neural network, recursive neural network, reverse nearest neighbour, etc. out of these expansions, the one corresponding to the meaning of the sentence should be identified in order to correctly understand the sentence. the task of identifying the correct expansion of acronyms from possible expansions is called acronym disambiguation (ad). methods of pattern matching, language modeling, and machine/deep learning have shown promising results in ad. early systems for ad used pattern matching @xcite together with approaches based on word embeddings and machine learning @xcite where the ad task is considered as a classification problem. recent efforts in ad mainly include the use of deep learning-based models @xcite and pre-trained language models @xcite . however, identifying the correct expansion of an acronym calls for better representation of text. in this study, we approached the problem of ad with the aim of learning effective text representations towards better disambiguation of acronyms. we derived our approach from siamese networks @xcite and triplet networks (tns) @xcite . tns, inspired by siamese networks, aim to learn the information of inputs based on one or a few samples of training data using a triplet loss to provide better representations for data. the main contributions of this paper were as follows: we leveraged the triplet loss and tns @xcite for ad with the aim of learning sentence embeddings, which can capture the semantic differences of the different expansions of the same acronym. we extended the tn architecture further to include m networks and mapped the figure 1 : triplet network architecture and modified triplet network architecture. the triplet network architecture (left, formula (1)) considers the anchor sentence x a i , positive sentence x p i , and negative sentence x n i for a sample when computing the triplet loss. modified architecture (right, formula (2)) considers the anchor sentence, positive sentence, and all the possible negative sentences for a sample. this includes m number of similar architectures. ad task as a binary classification problem, which predicts if the suggested expansion for an acronym is correct or not. to the best of our knowledge this is the first attempt of adapting the tn-based methods and triplet loss for disambiguating the acronyms. we evaluated and verified the proposed approach on the aaai-21 scientific document understanding ad task dataset (sdu dataset) (veyseh et al., 2020), sense inventory for clinical abbreviations and acronym dataset (casi dataset) @xcite , and on a sample of the medical abbreviation disambiguation dataset (medal) @xcite . we made our implementation available at https://github.com/sandarusen/m_networks under the mit license. ## related work extensive body of prior research for ad in scientific and medical domains exists because understanding scientific and medical text requires both ad and domain knowledge. earliest approaches for ad included the use of a number of rules and patterns @xcite , training of classifiers based on a set of features which represent the context of the input like, part-of-speech tags, case representation of the words, or word stems @xcite , and computation of the cosine similarity between the text with the acronym and the possible expansions based on word embeddings @xcite . recent efforts in ad include the use of deep learning-based methods and pre-trained language models @xcite @xcite . with the introduction of transformers, the transformer-based pre-trained language models have been extensively used for the ad task. bert (bidirectional encoder representations from transformers) models such as @xcite , scibert (bert-based language model for performing scientific tasks) @xcite , and roberta (robustly optimized bert pretraining approach) @xcite are the language models that are exploited to formulate the problem of ad as a classification task for ad. the sdu@aaai-21 ad task consisted of systems with transformer-based language models, which differed based on how the inputs and the outputs to the systems were defined @xcite . in our work, we explored triplet loss and tns for ad using pre-trained language models. tns and triplet loss have been effectively used for representation learning by distance comparisons among pairs of examples. they were initially introduced for computer vision related tasks @xcite and are now used in many natural language processing (nlp) tasks @xcite @xcite . we believe that through the triplet loss, the models will be able to learn subtle yet complex differences among the different expansions of the same acronym. ## methods the goal of ad was to identify the correct expansion for a given acronym in text. considering a dictionary of acronyms d with acronyms as keys [a 1 , a 2 , ..., a j ] where j is the number of acronyms. for each acronym a i , the m possible expansions were represented as [e 1 , e 2 , ..., e m ]. given a sentence x i with an acronym a i , the correct expansion should be obtained from d out of the expansion list of the corresponding a i . we modeled the ad task based on a tn as well as a modified version of the tn architecture with the triplet loss. the tn allowed the ad task to be expressed as a binary classification problem to predict which expansion is the most relevant to the given acronym based on the context it appears (appendix a). for the modified version of the tn, we included m number of architectures considering the possible negatives for a sample at once. this resulted in an anchor sentence, a positive sentence, and a list of negative sentences as inputs to the architectures (figure 1 ). denoting anchor, positive, and negative embeddings as x a i , x p i , and x n i , respectively, where @xmath0, 2, . . . , k, and considering a d-dimensional embedding in the vector space f (x) ∈ r d and α a margin that is enforced between positive and negative pairs, the loss for the tn was defined as follows using the l 2 distances for the tn: the triplet selection ensured effective training of the models. hence, it is advised to consider triplets, which violate the triplet constraint (formula (1)). in our approach, we considered the same positive sentence with the respective acronym replaced by other expansions of the acronym as negatives. even though the text in the sentences was very much similar to each other, replacing the acronym with possible expansions resulted in a change in the semantic meaning of the overall sentences. hence, we believe considering sentences with other possible expansions as negative sentences satisfied the necessity of having hard negatives, which were difficult to discriminate from the correct expansion. ## experiments we used the sdu dataset (veyseh et al., 2020), casi dataset @xcite , and medal dataset @xcite (see appendix b for further information). the sdu dataset contained data from 6, 786 english scientific papers published at arxiv and consisted of 62, 441 sentences. the dataset also consisted of a dictionary of acronyms and their possible expansions. we used the publicly available training and development data of the sdu dataset for our experiments. casi dataset was created using admission notes, consultation notes, and discharge summaries from hospitals affiliated with the university of minnesota. 37, 500 samples from casi dataset was split into train, validation, and test subsets and a dictionary with the acronyms was created for the experiments. the medal dataset was created from 14, 393, 619 articles in pubmed. we created a sample dataset and a dictionary of acronyms from medal dataset for experiments ( we performed a basic preprocessing on the sentences, which were quite long, by sampling tokens in the sentences as proposed by @xcite . we used n/2 tokens to the left and right of the acronym for sentences with length of more than 120, considering @xmath1. as a baseline model, we experimented with the system proposed by singh and kumar, 2021 which modeled the ad task as a span prediction task. the proposed system fine-tuned the complete scibert model with 12 layers to predict the start and end indices of the correct expansion of an acronym given all the possible expansions, leveraging the scib-ert's ability to encode pair of sequences together. we used the pre-trained scibert model architecture as the base model for experiments on sdu dataset and the pre-trained biobert (bert-based language model for performing biomedical tasks) @xcite model as the base model for experiments on the casi and the medal datasets with their first 11 encoder layers frozen followed by dropout of 0.5 to avoid over-fitting and a dense layer to map the feature embeddings output by the base models with dimensions of 768 to 64 (appendix c). these 64 dimensional embeddings were used to compute the triplet loss. we trained the models using a learning rate of 5 × 10 -4 with the adam optimizer @xcite . the best model over 10 epochs with a batch size of 32 was chosen as the final model. to evaluate the performance of the proposed architecture in the training set, we computed the macro-averaged f1 score. if the distance between the anchor and the positive sentence is less than the distance between the anchor and negative sentences, the prediction of the model was considered correct. we used f1 also in evaluation. we computed the distances between the anchor and possible sentences from which the sentence with the minimum distance to the anchor was considered the sentence with the correct expansion. ## results and analysis by comparing the proposed methods with the baseline system on the three datasets, we observed that the methods based on tns learnt to discriminate among the different expansions of an acronym. compared to the tn-based method, the m network-based method has comparable performance as the baseline for all the datasets. both the proposed methods outperformed the baseline on sdu and medal datasets. the m network-based method gave an f1 score of 87.31% on sdu dataset, 70.67% on casi dataset, and 75.75% on medal dataset ( to investigate the semantic similarity and the representation of the output embeddings in the vector space, we visualized output representations obtained by the m network-based architecture for the sdu, casi, and medal datasets by reducing the dimensions using principal component analysis (pca) (figure 3 of appendix d). for the sdu dataset, we used the acronym rl with reinforcement learning to obtain the positive and respective negative sentences. similarly, for the casi dataset the acronym dm with diabetes mellitus expansion and for the medal dataset the acronym rsm with respiratory muscle strength expansion were used. ## discussion in this paper, we have suggested a new approach for disambiguating the acronyms to effectively identify the correct expansion through better representation learning using tns by creating high quality sentence embeddings, which can capture the semantic differences among the different expansions of the same acronym. namely, we have presented how methods based on tns and triplet loss can be used for ad. to address the effective learning of context representations for identifying the correct expansion of acronyms, our methods leverage the contextual information of text and semantic similarity among expansions. in particular, our paper has introduced m networks inspired by tns. our experiments have demonstrated that methods based on tns have comparable performance on both scientific and medical domains. however, the applicability of the proposed methods on casi dataset should be further investigated. finally, the number of parameters in tn-based methods is only 12% of the number of parameters in the baseline method resulting in smaller size of the models ( we have tested the proposed methods on the sdu, casi, and medal datasets. the tn-based method for ad can be used for data augmentation when the training data is limited. given that the original tn architecture only considers one negative sample at a time, considering all the possible expansions of each acronym one at a time can be used to augment the training data size. this addresses the issue of limited training data for deep learning architectures. however, in the modified tn-based architecture with m networks, at the training stage all the possible negatives are considered for a sample at once. therefore, data augmentation is not possible in this case. in this paper, our main goal was to approach the ad problem as an effective representation learning problem to discriminate among the possible expansions of an acronym based on the context it appears. earliest approaches on ad relied on rules and patterns @xcite to identify the correct expansion of an acronym which evolved to use of machine learning-based approaches with different features @xcite and computing of semantic similarity between the text with acronym and the possible expansions. recent efforts involved pre-trained language models for the ad task. most of these systems were validated on one domain of focus (i.e., scientific text, medical text, or general text). we approached the problem focusing on learning better representations for text through tns and triplet loss using pretrained language models. furthermore, we tested the proposed approaches on both the scientific and medical domains. as future work, we intend to experiment with different constrastive losses @xcite . specifically, our aspiration is to compare and contrast the proposed approach with infonce ( @xcite , a popular contrastive loss which includes multiple negatives and normalises across examples in a mini batch. ## ethical considerations we have proposed an approach for ad using tnbased methods with the aim of learning effective representations for data. we have used scibert trained on scientific publications and biobert trained on biomedical domain corpora (pubmed abstracts and pmc full-text articles) for our experiments. instead of finetuning all the layers in the pre-trained language models, we have finetuned only the last encoder layer by freezing the first 11 encoder layers thereby bringing the latest deep learning advances to ad in a computationally efficient way. however, the m network architecture despite its smaller number of parameters has m architectures. this has resulted in more updates in the parameters increasing the computational time in the training stage. the proposed approaches have been tested and validated on three datasets: sdu dataset, casi dataset, and medal dataset. according to the national statement on ethical conduct in human research (2007) -updated 2018 @xcite , a new ethics approval is not required for our experiments and, to the best of our knowledge, the three original datasets have been created ethically. all the three datasets are publicly available (see appendix b). identifying the correct expansion of acronyms is important in improving the understandability of scientific/medical text due to the prevalence of technical acronyms which are shorthanded for ease of use. for people with limited expertise knowledge, understanding scientific/medical documents can be difficult, stressful and cause misunderstandings. the proposed methods can be used in scientific/medical text simplification tasks to provide lay people with better understanding of text through the disambiguation of acronyms.