Annotating Patient Information Needs in Online Diabetes Forums (2020)

Abstract: Identifying patient information needs is an important issue for health care services and implementation of patient-centered care. A relevant number of people with diabetes mellitus experience a need for information during the course of the disease. Health-related online forums are a promising option for researching relevant information needs closely related to everyday life. In this paper, we present a novel data corpus comprising 4,664 contributions from an online diabetes forum in German language. Two annotation tasks were implemented. First, the contributions were categorised according to whether they contain a diabetes-specific information need or not, which might either be a non diabetes-specific information need or no information need at all, resulting in an agreement of 0.89 (Krippendorff’s α). Moreover, the textual content of diabetes-specific information needs was segmented and labeled using a well-founded definition of health-related information needs, which achieved a promising agreement of 0.82 (Krippendorff’s αu). We further report a baseline for two sub-tasks of the information extraction system planned for the long term: contribution categorization and segment classification.

## motivation diabetes mellitus (dm) is a chronic disease with an estimated global prevalence of about 10% and steadily rising @xcite . to reduce the risk of diabetes-related acute and long-term complications, continuous health care and support as well as a high level of self-management skills are required @xcite . a relevant number of people with dm experience a need for information during the course of the disease @xcite . according to @xcite , a patient information need is defined as "the recognition that their knowledge is inadequate to satisfy a goal, within the context or situation that they find themselves at a specific point in the time". for example, shortly after diagnosis, individuals with dm report a high need for information with regard to new treatment strategies that simplify their everyday life @xcite . to support individuals with dm adequately, identifying their information needs is an important issue in the implementation of patient-centered care @xcite . nevertheless, research in this area is still sparse. only a few studies consider information needs during the course of the disease or in everyday life @xcite . studies from the field of diabetes often rely on classic quantitative or qualitative survey methods, such as questionnaires and interviews. however, to find the information they need, people with dm use online health communities, for example online diabetes forums, to improve their disease management @xcite @xcite ). an online forum can be described as an "asynchronous communication with other community members (e.g., patients can put questions to professionals or peers)" (van der @xcite . we focus on the analysis of textual contributions from online diabetes forums, as personal experiences reported with classic methods may differ from personal experiences reported in online forums. an advantage compared to traditional methods is that the communication of people with dm in online diabetes forums can highlight information needs that are more closely related to everyday life @xcite . this might be due to the perceived anonymity in online forums and the absence of a researcher, which can result in an openness of the forum users @xcite . nevertheless, there are several challenges for researchers using online forums as data sources. these include identifying relevant passages in a mass of posts @xcite and consideration of the more spoken language used by users when writing their posts @xcite . to the best of our knowledge, there is only one previous study in the field of information needs in diabetes, which uses online forums as data sources @xcite : @xcite analysed posts in online forums from adolescents with type 1 dm. they found that adolescents with type 1 dm visit online forums primarily to seek for social support, to receive information or counseling, and to exchange experiences. these findings support our motivation to explore information needs in the context of online forums. machine learning and natural language processing have been increasingly applied for the research of health-related issues in social media, e.g. content analysis, opinion mining, and the development of domain-specific lexicons @xcite @xcite @xcite . there are two recent works focusing in particular on diabetes @xcite . @xcite built a system for the detection and extraction of adverse drug events for drug safety surveillance. the developed approaches are evaluated on a case study corpus from a diabetes patient forum in the united states. @xcite investigated the automatic detection of the risk of type 2 dm directly from the twitter activity of a person. in both cases english language data sets are used. in this paper we present a novel corpus of german-language forum data on diabetes in which information needs are coded. the corpus was created with two objectives in mind, namely (a) the recognition of diabetes-specific information needs at contribution level and (b) the identification of relevant text segments within contributions that contain a diabetes-specific information need. our intermediate research goal is the building of an information extraction (ie) system which consists of three steps. first, contributions with diabetes-specific information needs are filtered. secondly, relevant text segments within the contributions with diabetes-specific information needs are identified. third, the relevant text segments are classified according to the dimensions of patient information needs. the developed data corpus will serve as training set for the development of such a system. we also show a first approach to both classification tasks. the text segmentation step will be investigated in future work. in the long term, the ie system will allow for an automatic analysis with regard to information needs in online diabetes forums, e.g. which topics trigger increased information needs and which groups have a need, and to develop a search system that can help individuals finding relevant forum posts related to their diabetes-specific information needs. the remainder of the paper is structured as follows. section 2 introduces the definition of patient information needs on which we base our annotation schema. subsequently, section 3 outlines the data basis for the corpus, the annotation process is described in detail and the resulting corpus is analysed. finally, section 4 presents a basic approach for the contribution categorization and the segment classification tasks and section 5 concludes our work and highlights the future research. ## ormandy's definition of patient information needs ormandy (2011) defines a patient information need as the "recognition that their knowledge is inadequate to satisfy a goal, within the context/situation that they find themselves at a specific point in the time". within this definition, the following four concepts are addressed, which can either activate or influence information needs: (i) goal/purpose, (ii) context, (iii) situation and (iv) time. ## data corpus in the following, the data source and the annotation process are presented. in addition, the reliability of the annotations is measured and the resulting data set is analysed. ## baseline approaches in this section, we provide baselines (a) for the first step in the ie system, the contribution categorization problem, and (b) for the third step in the ie system, the classification of given text segments according to the dimensions of an information need. the intermediate segmentation step will be examined in future work. the baselines are intended to serve as a basis reference value for follow-up research. the method was the same for both tasks. texts (either entire contributions or text segments) were split into lower case lemma tokens using spacy foot_0 and iwnlp @xcite and stop words were removed. the classification was implemented with scikit-learn @xcite . as classifier we chose a svm, with fixed c of 1 and linear kernel, as svms are known to achieve good results for various text classification tasks. the evaluation was performed through a five-fold cross-validation. all unigrams that occurred in at least 10 documents and in at most 80 percent of the respective training data set were retained in the vocabulary, and the resulting representations were weighted with tf-idf. the class in oth is poorly hit, which is likely due to the class imbalance in the data set. while none and in ds are about the same size, the class size of in oth with a total of 208 instances makes up less than 5% of all contributions. analysis of wrong predictions between in ds and none shows that a frequent mistake occurs in the recognition of rather technical, often long reports describing and evaluating products (e.g. blood sugar measuring devices, pumps). these are often erroneously assigned to in ds although they do not contain information needs. this might be due to the subject matter, showing similar wording to some contributions with diabetes-specific information needs. the f 1 values of 0.75 for in ds and 0.71 for none nevertheless show that, despite a high degree of overlap in vocabulary, there are terms or term combinations that are class specific. another interesting aspect relates to questions. contrary to the expectation that explicit questions with a corresponding punctuation mark are a clear indication of the existence of an information need, this did not hold true. on the one hand, contributions with rhetorical or self-reflexive questions (primarily in narrative contributions), e.g. "how did i come across this again?", were recognised as in ds . on the other hand, quite unexpectedly, short contributions consisting mainly of goal/purpose questions were assigned none. this illustrates that the use of questions is not a specific evidence of a need for information in this data corpus. the error analysis showed that a svm based on unigrams is insufficient for the task. instead, a more sophisticated approach that involves the context of a contribution to decide whether words or sentences indicate information needs must be used. this difficulty becomes particularly evident in the case of questions which can take on different functions depending on the context in which they are found, e.g. expressing the goal/purpose of an information need or, conversely, giving no indication of an information need (rhetorical or self-reflexive questions). when classifying text segments according to the dimensions of an information need, good precision values of 0.75 and 0.80 were achieved, but the recall rate was noticeably weaker. nevertheless, the f 1 macro value of 0.64 shows that our model was able to learn at least some characteristics of both classes and goes beyond pure guessing. the results imply that the correct assignment of the two dimensions of an information need based on the wording of a single text segment is very difficult. the concepts of the single segments might only become clear from the overall context of all relevant segments of a contribution. ## conclusion and future work with the long-term goal of developing an ie system for diabetes-specific information needs from forums, we introduced a german-language online diabetes forum corpus annotated on different levels. 4,664 forum posts were coded for the identification of diabetes-specific information needs on document-level. 2,167 contributions that contain a diabetes-specific information need were segmented and labeled using a well-founded definition. in this context we discussed how ormandy's definition of patient information needs can be applied to the domain of online forums. the resulting agreement values of 0.86 κ, 0.89 α and 0.82 α u prove good reliability. further, a svm approach was applied. a promising f 1 of about 0.75 was achieved in the identification of diabetes-specific information needs at contribution level. for the finer classification of text segments according to the dimensions of an information need, the approach performed weaker, with an improvable f 1 macro of 0.64. the results offer a baseline for further work. following this work, we will improve the classification approaches and also work on text segmentation methods, in order to develop a complete information extraction system in the long run.