Data Augmentation for Rare Symptoms in Vaccine Side-Effect Detection (2022)

Abstract: We study the problem of entity detection and normalization applied to patient self-reports of symptoms that arise as side-effects of vaccines. Our application domain presents unique challenges that render traditional classification methods ineffective: the number of entity types is large; and many symptoms are rare, resulting in a long-tail distribution of training examples per entity type. We tackle these challenges with an autoregressive model that generates standardized names of symptoms. We introduce a data augmentation technique to increase the number of training examples for rare symptoms. Experiments on real-life patient vaccine symptom self-reports show that our approach outperforms strong baselines, and that additional examples improve performance on the long-tail entities.

## introduction motivation. outside of clinical trials of vaccines on a small part of the population, it is important to study symptoms that arise as side effects of vaccines in the broader population. this is particularly crucial when the vaccines have only been granted emergency use permission, as has been the case for the covid-19 vaccines such as the pfizer-biontech mrna vaccine, the oxford-astrazeneca adenovirus-vectored vaccine, and others. in the united states, the vaccine adverse event reporting system (vaers) 1 , co-managed by the centers for disease control and prevention (cdc) and the u.s. food and drug administration (fda), is a national system that collects and analyzes reports from patients, about possible side effects after taking a vaccine. vaers presents a rich source of data for researchers to analyze. a challenge that arises when trying to analyze patient self-reports such as those in vaers is that patients are free to use their challenges. our application setting presents unique challenges : 1) entity names can be long and contain a lot of common nouns; 2) the number of entity types is large; 3) the number of labels in each example varies widely, e.g., patient reports contain anywhere from a minimum of 1 to a maximum of 131 symptoms; and 4) while a few symptoms are common, many are rare, resulting in a long-tail distribution of labels per entity type. contributions. to tackle these challenges, we frame the problem as an entity retrieval (er) task. we leverage an autoregressive entity retrieval model @xcite that generates standardized names of symptoms from patient self-reports, as opposed to a classification model such as a pre-trained model @xcite or biobert @xcite fine-tuned with a classification layer on top. to tackle data sparsity problems of rare symptoms, we propose a data augmentation method that generates training data points through the definition of symptoms. we then obtain symptom definitions in two ways: i) pre-trained language models: it has been shown that pre-trained language models are good at generating definitions @xcite , we therefore use @xcite to generate symptom definitions. and ii) umls: for additional definitions, we consult a medical knowledge graph, the unified medical language system (umls) @xcite . umls is the largest and most authoritative knowledge graph of the biomedical domain with over 3 million entities. our experiments on the vaers dataset show that our approach outperforms strong baselines, and that additional examples improve performance on long-tail entities. ## autoregressive entity retrieval model the goal of symptom entity detection is to predict symptom entities @xmath0, ..., e n } corresponding to the input description x. each example is a pair of (x, e) and the number of entities n varies over the dataset. as shown in figure 2 (a), multi-label classification approaches are trained to minimize cross entropy loss over all symptom classes. in the autoregressive entity retrieval, figure 2 (b) , the model generates a sequence of symptom names as a target sentence instead of classifying each entity class. we adopt genre's @xcite architecture that consists of transformer-based encoder and decoder. however, to retrieve multiple symptoms, genre requires annotated spans that refer to each symptom. for example, the source and target sequences should be ("i have muscle pain and fever", "i have [muscle pain] (myalgia) and [fever] (pyrexia)"). in our setting, a key difference is that the vaers dataset is not annotated with the mention spans of entities, only whether or not a particular symptom was mentioned by the patient. therefore, we generate the target sequence as a comma separated list, i.e., the pair of source and target sequences is ("i have muscle pain and fever", "myalgia, pyrexia"). then the model is trained to maximize the probability ## data augmentation while the data of common symptoms, such as headache and pyrexia, are abundant to train the model, examples of long-tail symptoms are rare, and therefore have fewer reported instances in the dataset. the median of the number of symptoms in our train set is 5 and over 80% of entities occur less than 50 times while headache and pyrexia have over 100k examples, see figure 3 . to overcome the problem posed by this very skewed training data distribution, we propose to generate additional labeled data in the form of definitions. the idea is that we can treat a symptom definition as a synthetic patient report (input sequence), and the symptom name as the corresponding label. we obtained definitions of symptoms in two ways: using a pre-trained language model, and using the umls biomedical dictionary. pre-trained language model. we use @xcite to generate definitions of longtail symptoms. we use the prompt: "the definition of [symptom name] is". we then add the generated sentence as a synthetic patient report and the symptom name as a label, to our augmented data. ## experiments dataset. from vaers, we consider data from the last three years (2019 to 2021), and randomly split it into train, validation, and test sets of 534, 516; 66, 814; and 66, 814 (80%/10%/10%). long-tail symptoms. the vaers dataset contains 10, 507 symptom entities. we define the symptoms with a frequency of less than 50 as longtail entities. as a result, 8, 755 entities are classified as long-tail, which are 83.3% of the total entity set. data augmentation. we obtained 10, 507 generated definitions from gpt-3 and 3, 480 definitions through the umls dictionary api. for the experiments, we used a single definition to mimic a patient with a single symptom, in addition, we cre-melaena gpt-3 blood in the stool, typically caused by gastrointestinal bleeding. ## related work in biomedical entity retrieval or entity linking, bert-based models, such as biobert @xcite or enrudr-bert @xcite , are often used to classify or re-rank candidate entities @xcite @xcite . in contrast to previous work, we took a generative approach. the social media mining for health applications (smm4h) workshop @xcite has introduced various shared tasks including normalization of adverse drug effects @xcite and detection of disease mentions in social media. approaches to overcome the problem of data sparsity and long-tail training data distributions include: data sampling @xcite , cost-sensitive loss function @xcite , regularization @xcite , semisupervised learning @xcite , and word/sentence level attention mechanism @xcite . the success of the few-shot generation demonstrated by @xcite has resulted in several studies that leverage gpt-3 for this purpose @xcite . @xcite explores ways of leveraging external resources such as dictionaries or medical documents. we use both a language model, in addition to a dictionary whose coverage is limited. ## conclusion we studied the problem of vaccine side-effect detection on real-world patient data. the characteristics of this domain render traditional classification approaches ineffective. our experiments demonstrated that combining a generative approach with synthetic data from symptom definitions obtained from a pre-trained lm and a medical dictionary can help improve performance on rare symptoms. exploring other approaches for learning with limited data, is an avenue for future work.