“Devils Are in the Details”: Annotating Specificity of Clinical Advice from Medical Literature (2022)

Abstract: Prior studies have raised concerns over specificity issues in clinical advice. Lacking specificity — explicitly discussed detailed information — may affect the quality and implementation of clinical advice in medical practice. In this study, we developed and validated a fine-grained annotation schema to describe different aspects of specificity in clinical advice extracted from medical research literature. We also presented our initial annotation effort and discussed future directions towards an NLP-based specificity analysis tool for summarizing and verifying the details in clinical advice.

## introduction in medical literature, authors often explain clinical implications after presenting their research findings. for example, "results of this post-hoc analysis suggest that lev may be a suitable option for initial monotherapy for patients @xmath0. clinical advice like this can influence health researchers and practitioners on specific medical practices. hence, it is an important information service to retrieve and analyze clinical advice from medical literature. prior studies have identified some quality issues that may affect the implementation of clinical advice. one concern is the lack of specificity. two studies have compared the implementation outcomes of professionally-designed clinical guidelines with different levels of specificity, and found that concrete and precise descriptions resulted in higher adoption rate @xcite . clinical advice in medical literature also varies in specificity levels. for example, advice sentences appeared in abstracts tend to be less specific compared to those in discussions, where more space is available for explaining the details @xcite . to better retrieve and summarize clinical advice from medical literature, this study aims to develop a taxonomy of specifics in clinical advice, such that they may be retrieved and compared in finer granularity. we developed and validated an annotation schema that can partition a clinical advice sentence to multiple elements: 1) agents; 2) substantial qualifications or elaborations; 3) chain of reasoning; 4) confidence. this annotation schema was developed based on medical research on clinical guidelines and nlp research on modeling specificity as a language construct. we also discussed the future directions for computationally modeling specificity of clinical advice in medical literature. such specificity analysis tool can be used for downstream applications such as detecting quality issues of clinical advice. for example, one study raised severe concern that many recommendations for clinical practice were not supported by findings in the conclusions @xcite . the problem is more severe in abstracts than in discussions. since abstracts are much more accessible than full-text articles, the "spins" in abstracts are also more harmful than those in discussions @xcite ). an nlp-based specificity analysis tool can help compare recommendation details against available evidence, or compare similar recommendations in fine-granularity. ## related work specificity is an important concept in both clinical practice and claim analyses. in medical domain, specificity is defined narrowly, focusing on the detailed information regarding clinical practice and health-related behavior changes. for example, @xcite defined a specific guideline as "creates clinical appropriate criteria for a large number of clinically detailed patient presentations; it does not force consensus" (p.1431). similarly, @xcite argued that specific clinical guidelines give "detailed advice on which performance is appropriate in which situation and in what patient group and determining which factors, or conditions should be taken into account" (p. 367). note that clinical guidelines used in practice are usually developed by professional institutions such as national institute for clinical excellence (nice). they are usually more comprehensive and specific than the clinical advice from individual research papers. compared to the narrow definition in clinical domain, specificity is defined more broadly in the nlp field, referring to how much detailed information is included in a statement. depending on the text domains, researchers have proposed different taxonomies to define specificity. for example, in education domain, the specificity of classroom discussions was defined based on four aspects: "involves one character or scene", "gives substantial qualifications or elaboration", "uses content-specific vocabulary", and "provides a chain of reasoning" @xcite . similarly, arguments in student essays were assigned specificity scores based on occurrence of qualifiers, references to supporting components, hypotheses, and real-world examples @xcite . specificity in other domains was defined quite differently. for example, the specificity of pledges of election manifestos were labelled based on expressions of moral values, intangible goals and outcomes, commitment to the maintenance of functioning policy, means and details to achieve the objectives @xcite . the specificity in social media posts was defined based on their references to specific person, object or event @xcite . although the exact aspects applied to describe specificity differ by domains, they usually cover the anwers to questions about who, what, when, where, why, and how. since the clinical domain and the education domain are most relevant to our task, we defined our annotation schema by combining the definitions from these two domains. ## towards computational modeling of specificity the explosive growth of research output and restricted human capacities in information processing and decision making calls for an nlp-based specificity analysis tool to synthesize and aggregate the scientific evidence and clinical recommendations in research publications. the developed annotation schema may then be used to develop automatic prediction models for clinical advice specifics classification and specifics extraction. based on the occurrence for each specificity dimension, we could frame the task as a sentence-level classification task and to computationally model the specificity level in each advice sentences. utilizing the annotated details under each specificity aspect, the task could also be framed as an information extraction one. information extraction tools could be developed to extract the details of each recommendation. for example, simple rule-based approaches using regular expressions (savova et al., 2010a) may identify the aspects of agents and targets. existing nlp tools for medical concepts (e.g. savova et al., 2010b; zhou et al., 2019; zhang et al., 2021) and clinical relation extractions based on pre-trained language models such as bert (roy and pan, 2021) may further identify other specificity aspects in the develop schema. after extracting the specifics explicitly mentioned in each recommendation, we could compare different versions of semantically similar recommendations across the specifics to detect the inconsistent or exaggerated clinical advice in research literature. ## conclusion in this work we presented a fine-grained annotation schema for describing specificity in clinical advice extracted from medical research literature. an inter-coder agreement check shows the proposed annotation schema reached almost perfect agreement in all dimensions. the annotation schema could be used to develop gold-standard dataset that can be used to develop nlp models for identifying fine-grained specificity aspects in clinical advice, and to support downstream applications such as summarizing clinical advice or fact checking.