Mining Health-related Cause-Effect Statements with High Precision at Large Scale (2022)

Abstract: An efficient assessment of the health relatedness of text passages is important to mine the web at scale to conduct health sociological analyses or to develop a health search engine. We propose a new efficient and effective termhood score for predicting the health relatedness of phrases and sentences, which achieves 69% recall at over 90% precision on a web dataset with cause-effect statements. It is more effective than state-of-the-art medical entity linkers and as effective but much faster than BERT-based approaches. Using our method, we compile the Webis Medical CauseNet 2022, a new resource of 7.8 million health-related cause-effect statements such as “Studies show that stress induces insomnia” in which the cause (‘stress’) and effect (‘insomnia’) are labeled.

## introduction health sociology studies the interaction of society with health. an important subject is how consumers obtain and perceive health information. since the web and search engines are among the most important sources, the quality of online health information has been studied so frequently in recent decades that three systematic reviews have been conducted (see studying larger portions of the online health domain requires the automation of various acquisition tasks: (1) the discovery of health-related websites and web pages, (2) the identification and extraction of health-related statements from these pages, and (3) the attribution of health-related statements to authoritative sources (e.g., for fact-checking). while the first and third steps have been and continue to be the subject of ongoing research, the second step has received much less attention. ## related work the impact of online health information on consumers has attracted the interest of the health sociology research community. for example, user surveys examine how consumers perceive online health information @xcite , e-health services @xcite , or the quality of online health information @xcite . information quality appears to be the most studied characteristic from a health sociology perspective. numerous studies systematically analyzed the quality of websites related to specific topics such as orthodontics @xcite or performanceenhancing drugs @xcite , but more general studies with limitations to specific parts of the web are also common. examples include studies of dietary advice @xcite or the misinterpretation @xcite and exaggeration @xcite of clinical trial results in online news. recent studies also targeted web search snippets @xcite and social media (suarez-lledo and alvarez-galvez, 2021), particularly health misinformation on twitter @xcite . to identify health-related web content, most previous work has focused on whole-page classification, e.g., using medical vocabularies to classify news articles @xcite or using convolutional neural networks for reddit posts @xcite . but there has been little work on classifying shorter passages of text as health-related (e.g., phrases), although this would allow for more fine-grained analyses at the statement level rather than at the whole-page level. keyword extraction and automatic ontology creation with the goal of extracting prototypical words for a given domain are perhaps the most closely related tasks. for example, the c-value/nc-value method uses term frequencies to extract multi-word domain terms from a corpus @xcite . its reliance on syntactic sentence structure, however, renders it inapplicable at the phrase level. contrastive termhood scores, which relate term frequencies from a domain corpus to frequencies from one or more out-of-domain corpora, can be applied more straightforwardly. these include tf •idfinspired measures @xcite , measures estimating how exclusive a term is for a domain @xcite , and combinations or extensions thereof @xcite . we transfer contrastive termhood scoring to measuring health relatedness of phrases (but also sentences) and compare it with the medical entity linkers ctakes @xcite , metamap @xcite , quick-umls @xcite , and scis-pacy @xcite , as well as bertbased classifiers @xcite . ## measuring health relatedness assessing whether a phrase is health-related can be difficult without context; in particular for homonymous (same surface form, different meaning) or polysemous (same surface form, different sense) words. for instance, 'cancer' may refer to a healthrelated malignant tumor, but also to the zodiac sign, which is unlikely to appear in a health-related context. as such, the task of assessing a phrase's health relatedness can be viewed as an extension of wordsense disambiguation. instead of the sense of a particular word, the domain of the sense of a phrase needs to be determined. since there are no large-scale labeled datasets for health relatedness assessment, we rely on contrastive termhood scores that use distant supervision to measure the degree of a word or concept being specific to a certain domain @xcite . instead of training on explicitly labeled words or phrases, contrastive termhood scores are trained on texts that are heuristically labeled. specifically, a word's or phrase's domain specificity depends on its "prominence" in domainspecific or out-of-domain corpora. in this section, we discuss three existing contrastive termhood scores and then explain how we adapt and apply them to health relatedness assessment. ## evaluation in this section, we evaluate our generalized termhood method on datasets of cause-effect statements and compare it to state-of-the-art medical entity linkers and bert-based approaches. ## webis medical causenet 2022 by applying our generalized termhood method for health relatedness assessment to the complete causenet, we create the new webis medical causenet 2022 resource consisting of healthrelated cause-effect statements found on the web. it is important to note, that the statements in causenet-and thus also in our webis medical causenet 2022-are only claimed cause-effect statements. for many of the contained statements, scientific evidence can surely be found (e.g., '@xmath0.g., 'incorrect placement of @xmath1. still, it could be interesting to analyze websites that contain many claimed healthrelated cause-effect statements with respect to whether medical evidence exists or not. such health-sociological analyses are now enabled by our webis medical causenet 2022 resource at web scale since the urls of the pages from which a statement was extracted are part of causenet. the webis medical causenet 2022 comes in four different versions foot_5 based on the best termhood parameterizations from the evaluation. ## conclusions we have proposed generalized termhood-based methods that effectively and efficiently assess the health relatedness of phrases. on cause-effect statements from the web, our new approaches are almost as effective as the best bert-based approaches while being much faster. approaches using state-of-the-art medical entity linkers are slower and less effective. when configuring our new termhood-based methods, it is crucial to select a background health corpus that matches the target language distribution while the other parameters (n-gram length, averaging) are less important. using our methods, we have extracted the webis medical causenet 2022 resource of healthrelated cause-effect statements from the webscale causenet. based on webis medical causenet 2022, health-sociological analyses of online cause-effect relations are now possible at an unprecedented scale compared to previous smallscale analyses of health-related online information. finally, our termhood-based assessment could also be useful in retrieval scenarios. for instance, given the termhood scores' efficiency, they could directly be used at search engine side to quickly assess the health relatedness of some query not seen before (for other queries, the clicked documents usually suffice to assess the health relatedness) and to possibly adopt the retrieval accordingly (e.g., preferring medical resources).