Condition-Treatment Relation Extraction on Disease-related Social Media Data (2022)

Abstract: Social media has become a popular platform where people share information about personal healthcare conditions, diagnostic histories, and medical plans. Analyzing posts on social media depicting such realistic information can help improve quality and clinical decision-making; however, the lack of structured resources in this genre limits us to build robust NLP models for meaningful analysis. This paper presents a new corpus annotating relations among many types of conditions, treatments, and their attributes illustrated in social media posts by patients and caregivers. For experiments, a transformer encoder is pretrained on 1M raw posts and used to train several document-level relation extraction models using our corpus. Our best-performing model achieves the F1 scores of 70.9 and 51.7 for Entity Recognition and Relation Extraction, respectively. These results are encouraging as it is the first neural model extracting complex relations of this kind on social media data.

## introduction there is an increasing number of disease-related posts published online every day. on social media platforms such as reddit and twitter, people discuss medical conditions and treatments they use to obtain insights from one another. capturing medical entities and their relations in these real-world data may significantly benefit tasks such as disease detection @xcite , adverse drug event @xcite , and pharmacovigilance @xcite . previous studies have established guidelines and corpora focusing on medical mention, chemicaldisease relations, and drug-drug interactions @xcite @xcite . one limitation of most existing corpora is that their data are collected from well-structured medical text, including electronic health records (ehrs), medical discharges, and clinical notes. models trained on the corpora of formal medical texts may not perform well on the social media data because social media data are noisy @xcite with poor sentence structures and spelling mistakes. an annotated corpus with carefully designed guidelines is necessary to take full advantage of the large-scale disease-related social media data. however, only a few research works contribute to medical text mining in the social media context @xcite @xcite , and no work has directly investigated the condition-treatment relation extraction @xcite on social media data. to bridge the research gap mentioned above, we develop annotation guidelines and address the automatic extraction of medical entities and conditiontreatment relations on social media data (section 2). our annotation scheme and the new corpus are illustrated in section 4. we then experiment with joint models between ner and re using our corpus (section 5). finally, a detailed error analysis of the experiment results is provided in section 6. the contributions of this paper are as follows: ## data our data is collected from various social media forums, using the keyword-based method to filter out disease-related posts. the source sites include online support groups, disease forums, message boards, etc. we obtain approximately one million unlabeled social media posts. ## experiments for automatic entity recognition and relation extraction, we adopt the state-of-art joint model for mention detection, coreference resolution, and relation extraction @xcite . focusing on task interactions between mention detection and relation extraction, the model incorporates graph propagation and graph compatibility, which improves decision-making. since our dataset does not include coreference annotation, the coreference evaluation is not performed in this paper. pretraining though there are existing pretrained language models for the medical domain @xcite , they are trained on biomedical literature, clinical notes, and discharge summaries. due to the novelty of the dataset, these language models may not provide good representations for online posts due to the different language styles. to take advantage of pre-trained language models, we continue to train 3 models, namely bert @xcite , roberta @xcite , and spanbert @xcite , on 1,068,330 disease-related social media posts. preprocessing the joint model requires input documents to be segmented into sentences. since our annotated dataset is not pre-segmented, one additional preprocessing step is necessary before experimenting with the model. initially, we utilize elit tokenizer @xcite to segment posts, followed by remapping all token index, offset and label index. however, the tokenizer fails to process some posts due to the following problems: (1) inappropriate spacing (e.g., lacking space between two sentences in '...vomiting.she...'); (2) unknown characters between entities (e.g., ' iterative stratify split results postprocessing for this task, higher precision scores are preferable to higher recall scores since less false positive output would benefit the subsequent medical decision-making processes. hence, we attempt to improve the precision score through postprocessing. first, we adjust the top span ratio for entity extraction, which controls the pruning rate of candidate entities according to their mention scores. top span ratios ranging from 0.4 to 0.1 are tested, which leads to an average of 1~2 percent increase in precision. then, we filter out singleton 2 attribute entities with no relation attached, which gives a precision increase of ~2 percent. ## error analysis further analysis is conducted based on the model prediction on the test dataset. one crucial problem that is observed from the predicted results is the spurious problem. in other words, the model predicts entities that do not exist in the gold annotation. there is a total of 178 spurious entities produced in 173 posts. the majority of predicted spurious entity types are condition labels (100 spurious entities detected) and treatment labels (35 spurious entities detected). the reasons for this problem are threefold: subsequently, relation extraction also suffers from the spurious problem. since the relation is generated based on the detected entities, the model would predict relations on spurious entities. moreover, long-distance relations pose challenges to the relation extraction task. for instance, when more than one condition are labeled in the post, the model is prone to attach the treatment to the closer condition rather than the corresponding one. ## conclusion to facilitate medical text mining in the social media context, we develop an annotation scheme of disease-related posts for the condition-relation extraction. following the guidelines, we present a reliable corpus foot_2 with 9,785 entities and 3,645 relations, which is a valuable addition to the limited corpora in this field. additionally, we experiment with automatic entity recognition and relation extraction, providing a promising model for mining online medical posts. we also conduct a detailed error analysis that may shed light on future work. the findings of our work suggest potential directions for further studies in this domain. possible progress could be made by increasing the corpus size since the current corpus is relatively small. also, the model structure could be designed to solve the spurious problem.