CL a CL ab at S ocial D is NER : Using Medical Gazetteers for Named-Entity Recognition of Disease Mentions in S panish Tweets (2022)

Abstract: This paper summarizes the CLaC submission for SMM4H 2022 Task 10 which concerns the recognition of diseases mentioned in Spanish tweets. Before classifying each token, we encode each token with a transformer encoder using features from Multilingual RoBERTa Large, UMLS gazetteer, and DISTEMIST gazetteer, among others. We obtain a strict F1 score of 0.869, with competition mean of 0.675, standard deviation of 0.245, and median of 0.761.

## motivation finding mentions of diseases in tweets in all languages has become an important tool for epidemiologists, especially in times of a pandemic. so-cialdisner @xcite at smm4h 2022 concerns the recognition of disease mentions in spanish tweets. a disease mention can include both lay and professional language and may be located in hashtags or usernames as well as the tweet text. disease entities are underlined in example 1: ## system our contribution consists of a simple pipeline incorporating four main components: a standard tokenizer for splitting hashtags and username tokens with an ad hoc extension for disease recognition, word embeddings for spanish and english from roberta large, four gazetteer lists extracted from relevant domain resources and represented as one hot vectors, and a linear classifier that produces bio (beginning, inside, outside) tags for each token for recognition of multi-token disease mentions. the simplicity of this pipeline and its knowledge injection from readily available domain resources rather than training purely from training data make our system's strength. tokenization sequence analysis classifying each token into bio tags requires good tokenization. we pre-process the data using annie @xcite and the twitter english tokenizer @xcite . hashtags and usernames usually include several words strung together without separating characters, e.g. #investigaepilepsia. since disease mentions can occur in these word composites, their individual word components have to be separated. camel casing makes splitting this example into investiga and epilepsia easy, but for hashtags like #nosolohaycovid only knowledge of words in spanish leads to the desired split into no, solo, hay and covid. because disease names are not always part of the general vocabulary of a language, disease gazetteers are a lightweight means to inject such domain knowledge into a general machine learning system. a gazetteer lists entities of a particular type. for example, a disease gazetteer would contain a list of disease names like covid, hepatitis, epilepsia etc. ad hoc gazetteer lists that are written for a specific task often suffer from limited coverage. we circumvent our own limitations by using four gazetteer lists compiled from extant resources: goldgaz: disease names compiled from the gold annotations of socialdisner training and validation data distemistgaz: (miranda-escalada et al., 2022) spanish disease gazetteer compiled from snomed-ct @xcite silvergaz: disease gazetteer compiled from silver standard data made available for socialdis-ner @xcite umlsgaz: disease gazetteer of spanish and english disease terms we compiled from umls @xcite . semantic type t047 represents disease or syndrome, identifying all umls concepts representing diseases. twitter tokenizer extension the gate twitter english tokenizer @xcite identifies hashtags and usernames, but doesn't split them. to split usernames and hashtags, we identify the longest substring in a hashtag/username that matches an entity in our gazetteer, then treat that substring as a separate token. the extension matches first against goldgaz, then distemistgaz, followed by umlsgaz, and finally silvergaz. classification we frame the named entity recognition (ner) task as a sequence labeling task with the bio (beginning, inside, outside) tagging scheme. in other words, we classify each token as either b (token is the beginning of a disease name), i (token lies inside a disease name) or o (token lies outside a disease name). model preprocessing produces s, a list of t tokens. s is fed into a pretrained xlm roberta large @xcite ) model, which returns the matrix @xmath0, with @xmath1. the i th row of h is a vector of size d and corresponds to the contextualized embedding of the i th token. we use the gate annie gazetteer plugin to find tokens that match terms in umlsgaz exactly (case-insensitively), creating g @xmath2, a matrix where each row is a one hot vector indicating whether the i th token matches with umlsgaz or not. similarly, we create g silver and g distemist corresponding to silvergaz and distemistgaz respectively. then, h is row-wise concatenated with g umls , g distemist , and g silver to produce @xmath3. z is then fed into a 6-layer transformer encoder @xcite with positional encoding and 10 attention heads per layer. the transformer encoder outputs @xmath4. y is fed into a linear classifier which produces @xmath5, classifying each token into one of the 3 classes corresponding to the bio tagging scheme. this represents our submission system m sub . training xlm roberta large is fine-tuned on the training data using the adam optimizer (kingma and ba, 2015) with a learning rate of 1e-5 and early stopping. training for 10 epochs takes 2.5 hours on one nvidia rtx 3090 gpu. ## ablation and test results the evaluation is by strict f1 score, requiring exactly matching gold spans without partial credit. our submission model m sub was described above in section 2.2. let m no-gaz be the same model as m sub but without the one-hot features generated from gazetteer matching. let m no-tok be the same model as m sub but without the tokenizer extension using disease gazetteers (hashtags and usernames are tokenized with the gate hashtag tokenizer only). finally, let m roberta be the baseline model without one-hot features, custom tokenization, and transformer encoder -the output of the language model h is fed directly into the classifier. we see that m sub improves on m roberta by only 1.2%. omitting gazetteers or the tokenizer extension each looses only 0.4% over m sub . m no-gaz looses precision while m no-tok looses recall against the submission system. ## conclusion for the task of detecting disease mentions, a general large, pre-trained language model (xlm roberta large) was enhanced with task-oriented preprocessing (splitting hashtags into component words) and lookup in available quality word lists. this combination was effective and in competition placed our system nearly 20% above the mean.