MACRONYM : A Large-Scale Dataset for Multilingual and Multi-Domain Acronym Extraction (2022)

Abstract: Acronym extraction is the task of identifying acronyms and their expanded forms in texts that is necessary for various NLP applications. Despite major progress for this task in recent years, one limitation of existing AE research is that they are limited to the English language and certain domains (i.e., scientific and biomedical). Challenges of AE in other languages and domains are mainly unexplored. As such, lacking annotated datasets in multiple languages and domains has been a major issue to prevent research in this direction. To address this limitation, we propose a new dataset for multilingual and multi-domain AE. Specifically, 27,200 sentences in 6 different languages and 2 new domains, i.e., legal and scientific, are manually annotated for AE. Our experiments on the dataset show that AE in different languages and learning settings has unique challenges, emphasizing the necessity of further research on multilingual and multi-domain AE.

## introduction acronyms are short forms of longer phrases that are often constructed using a few letters selected from the long phrases. due to their functionality, acronyms are common in many languages and domains. for instance, 73% of abstracts of scientific papers contain at least an acronym @xcite . as such, in text processing applications, e.g., question answering and machine translation, it is necessary to correctly identify the acronyms and their meanings. toward this goal, our work focus on the task of acronym extraction (ae), aiming to recognize acronyms and their definitions/long forms in text. for instance, in the sentence "they will meet in the conference of the world trade organization (wto)", an ae system should identify "wto" and "world trade organization" as the acronym and long form, respectively. despite all progress in recent years, prior work on ae has mainly limited to specific domains and languages. specifically, biomedical and scientific texts in english have been the main focus in prior work. however, recognition of acronyms in other languages and domains is also important and might involve challenges not reflected in english biomedical/scientific texts. for instance, many existing ae methods for english employ uppercase letters to identify acronyms @xcite . however, in non-case sensitive languages, e.g., arabic or persian, uppercase letter concept does not exist, thus causing a failure of existing ae systems. moreover, in each domain or language, different styles might be exerted to shorten a longer phrase to produce acronyms. for instance, initial letters of the words in the phrases are commonly used to form acronyms in scientific english; however, in legal english or danish documents, the use of initial letters for acronym detection is less effective (see section 3). as such, it is desirable to study ae in more diverse domains and languages to better support multi-domain and multilingual applications. unfortunately, to the best of our knowledge, there is no existing dataset for multilingual and multi-domain ae, thus impeding research effort in this area. to this end, our work addresses this issue by introducing a new manually labeled dataset for ae. in particular, based on two different domains of scientific and legal texts, our dataset annotates ae data for sentences in six different languages: english, danish, spanish, french, persian, and vietnamese. as such, legal texts, danish, spanish, french, persian, and vietnamese are not explored for ae in prior work. in addition, our dataset is large-scale, providing 27,200 annotated sentences for ae to support advanced model development (e.g., with data-hungry deep learning models). finally, we conduct extensive experiments to understand the challenges of ae in the created dataset. our experiments show that the ae task in our dataset presents significant challenges for existing models in different domains and languages. this is even more pronounced in the cross-lingual and cross-domain transfer settings where existing models perform poorly on our ae dataset. as such, more research effort is needed to address the challenges of acronym understanding in different settings. we will publicly release the dataset to foster research in this area. ## data annotation data collection: we collect data in two domains of legal and scientific documents for ae annotation. for each domain, documents in different languages are required. as such, for the legal domain, we employ the united nations parallel corpus (unpc) @xcite and the europarl corpus @xcite . the unpc corpus contains official records in 6 languages while the europarl corpus consists of the proceedings of the european parliament in european languages. to accommodate our annotation budget and diversify the resulting dataset, we choose documents from four languages in the two corpora (i.e., english, french, and spanish in unpc, and danish in europarl) for our ae annotation. in addition, for the scientific domain, we employ the publicly available papers and m.s./ph.d. theses in the field of computer science for ae annotation. specifically, we collect the papers published in the acl anthology of natural language processing research for english. also, for typologically different languages, we crawl public computer science thesis in persian and vietnamese. following @xcite , we split the selected documents into sentences that will be annotated separately by annotators. in addition, to optimize the annotation cost with greater numbers of acronyms, we apply the same procedure in (veyseh et al., 2020) to filter out sentences that has low chance to contain acronyms or long forms. in particular, the procedure only retains sentences that involve at least one acronym candidate (i.e., a word with more than a half of characters as capital letters) and a sub-sequence of words to match the acronym candidate (i.e., concatenating the initials of the words can form the candidate) @xcite . here, we only apply this procedure for english, french, spanish, and danish as our persian and vietnamese data is small and the sentence filtering procedure will leave less sentences for annotation. finally, given the retained sentences for each language, we randomly sample a subset of sentences for manual ae annotation. the numbers of annotated sentences are presented in to study the challenges of ae in each language, following (veyseh et al., 2020), we compute the inter-annotator agreement (iaa) scores using krippendorff's alpha @xcite with the masi distance metric @xcite for the initial independent annotations of the two annotators, i.e., before resolving the conflicts. ## experiments this section studies the challenges of the multilingual and multi-domain ae task in macronym. in particular, for each pair of available languages and domains (we have 7 pairs in total), we first prepare the data by randomly splitting the corresponding set of annotated sentences into separate training/development/test portions with the ratios of 80/10/10 (respectively). afterward, we report the performance of the representative ae models on the test set for each possible pair of languages and domains under different learning settings. ## related work early attempts for ae have employed rule-based methods @xcite @xcite @xcite or feature engineering models @xcite @xcite . recently, deep learning methods have delivered sota performance for ae @xcite @xcite @xcite @xcite . despite such progress, prior ae research and datasets have mainly focused on english biomedical and scientific texts, leaving non-english languages and other domains less explored. here, we note that there exist some acronym glossaries for non-english languages @xcite . however, such resources do not annotate sentences/texts in multiple languages and domains for ae as we do in macronym. ## conclusion we present the first multilingual and multi-domain dataset for ae, involving annotation for 6 languages and 2 domains. our experiments show that the proposed dataset presents significant challenges for ae methods in different learning settings and languages. in the future, we will expand the dataset to include more domains and languages for ae. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of aro, odni, iarpa, the department of defense, or the u.s. government. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. this document does not contain technology or technical data controlled under either the u.s. international traffic in arms regulations or the u.s. export administration regulations.