SYMPTOMIFY : Transforming Symptom Annotations with Language Model Knowledge Harvesting (2023)

Abstract: Given the high-stakes nature of healthcare decision-making, we aim to improve the efficiency of human annotators rather than replacing them with fully automated solutions. We introduce a new comprehensive resource, SYMPTOMIFY, a dataset of annotated vaccine adverse reaction reports detailing individual vaccine reactions. The dataset, consisting of over 800k reports, surpasses previous datasets in size. Notably, it features reasoning-based explanations alongside background knowledge obtained via language model knowledge harvesting. We evaluate performance across various methods and learning paradigms, paving the way for future comparisons and benchmarking.

## introduction drug safety monitoring systems like vigibase @xcite and vaers foot_1 are essential for evaluating the safety of medications and vaccines. these systems enable anyone to report drug reactions, facilitating early detection of vaccine safety issues. for example, if a mother observes vomiting and fever in her two-year-old after the chickenpox vaccine, she may be curious about the frequency of similar reactions reported by others. vaers' online browser can provide the answer. being able to provide such answers involves normalizing the raw patient reports in order to address variations caused by polysemy, abbreviations, and other factors present in these informal and potentially noisy reports. vaers employs trained staff to review and "code" each submitted report, assigning formal medical terms to the symptoms experienced by the patient. vaers terms come from the standardized medical terminology known as the medical ## related work and existing datasets our work is related to the old task of named entity recognition (ner), which is the task of mapping mentions of entities in text to standardized entity names @xcite @xcite @xcite . our work goes beyond recognizing entities as we wish to support human decision making. additionally, our work is complementary to work on explainable artificial intelligence (xai) @xcite , and can be used to build more explainable models. furthermore, there are efforts to enhance open-source llms by extracting instruction-output pairs from massive, paid-api llms. the goal is to gather valuable data that can be utilized for instruction tuning of open-source models. this method is particularly beneficial for projects that lack the financial resources to hire people for human feedback, thereby democratizing access to advanced llm capabilities @xcite . closer to our work are prior efforts on creating datasets for medical entity recognition from diverse sources, including death certificates @xcite , scientific publications @xcite , and electronic health records (ehrs) @xcite . while these corpora contribute to medical entity recognition, they dataset source # symptoms size symptomify (ours) vaers 11,472 871,373 cometa @xcite reddit 7,645 20,000 medred @xcite reddit 18 2,000 redmed (lavertu and altman, 2019) reddit 2,978 n/a twitter twimed @xcite twitter 3,144 2,000 cadec @xcite askapatient 6,754 1,253 twitter adr @xcite twitter 1,280 1,784 ## lm knowledge harvesting access to the most powerful lms is often limited to paid apis with query restrictions. in dynamic settings, such as continuous meddra report annotation, rapid query response is crucial. therefore, we propose to extract a portion of lm knowledge to enrich an existing dataset. this approach allows greater control, transparency, and balance between performance, cost-efficiency, and autonomy. enriching existing datasets meets a widespread need as under-specified datasets are common. a prominent example comes from question answering tasks where only few datasets are fully labeled with question, answer, passage, and span. in our work, key steps to upgrade a dataset via lm knowledge harvesting include: (i) identifying shortcomings: assess the dataset to determine missing elements and areas needing improvement. (ii) creating good prompts: design prompts using existing dataset examples to minimize potential ambiguities. (iii) checking data quality: implement strategies to assess llm output quality, ensuring it is accurate, useful, and safe. (iv) maintaining detailed records: document the harvesting process, including prompts, llm configurations, and dates, for improved transparency and reproducibility. ## the symptomify dataset with vaers 3 as the starting dataset, we used lm knowledge harvesting to create symptomify. the dataset spans three years (2019-2021) and includes 839, 215 reports, incorporating symptom texts, relevant meddra terms, and associated metadata like age, sex, and vaccine type. ## conclusion we introduced symptomify, a large-scale dataset of annotated reports, reflecting reactions to medications and vaccines. it includes meddra symptoms, annotation explanations, and background knowledge about symptoms, designed to facilitate development of systems that can aid human annotators operating in the critically important public health domain. despite current performance on rare symptoms being low across all baselines, the continual evolution in language models and strides in few-shot learning offer promise for improvement. there are several avenues for improving symp-tomify and expanding its utility. integrating the explanations into a wider range of models may lead to the better generation of explained predictions with open source models such as falcon. expanding background knowledge to include relational knowledge, such as inter-symptom relationships, could yield performance gains. further, focusing on laypeople's perspectives and acquiring more layman medical terms could facilitate a broader, more inclusive understanding of symptoms. looking further ahead, as patient reports can sometimes contain not only textual descriptions but also ref-erences to medical images, pictures submitted by patients, and lab test results, the incorporation of these non-textual elements using multi-modal language models, could be useful.