Method for Designing Semantic Annotation of Sepsis Signs in Clinical Text (2023)

Abstract: Annotated clinical text corpora are essential for machine learning studies that model and predict care processes and disease progression. However, few studies describe the necessary experimental design of the annotation guideline and annotation phases. This makes replication, reuse, and adoption challenging. Using clinical questions about sepsis, we designed a semantic annotation guideline to capture sepsis signs from clinical text. The clinical questions aid guideline design, application, and evaluation. Our method incrementally evaluates each change in the guideline by testing the resulting annotated corpus using clinical questions. Additionally, our method uses inter-annotator agreement to judge the annotator compliance and quality of the guideline. We show that the method, combined with controlled design increments, is simple and allows the development and measurable improvement of a purpose-built semantic annotation guideline. We believe that our approach is useful for incremental design of semantic annotation guidelines in general.

## introduction annotated clinical text corpora provide natural language processing (nlp) and machine learning (ml) studies the data necessary to find patterns, classify, and predict patient risk and disease progression. compared to models that only utilize structured data from the electronic health record (ehr), many studies and reviews have shown that model performance can increase by incorporating unstructured clinical text @xcite @xcite @xcite . pre-existing annotated clinical corpora include the medical information mart for intensive care (mimic-iii) @xcite , the clinical e-science framework (clef) @xcite , and the informatics for integrating biology and the bedside (i2b2) challenges and national nlp clinical challenges (n2c2) @xcite . however, studies utilizing preexisting annotated corpora must limit their research questions to the specific purpose(s) for which the corpus was annotated. otherwise, the annotations required to answer a research question might be missing or too general. thus, many studies opt to develop their own annotated clinical corpus tailored to capture and extract the necessary information for their research @xcite @xcite . methods with lower requirements for supervision, such as information extraction, commonly use keyword search, rule-based algorithms, and ml to detect clinical cases. however, those methods might not consider the context of the clinical case @xcite . for example, different documented signs within a specific situation can describe a medical condition that is not named. hence, medical expertise is necessary for making annotation judgments and capturing clinical knowledge within the text @xcite . retrieving domain-specific patient knowledge to ascertain or answer clinical questions includes extracting data, information, and knowledge. data are attributes (e.g., names or dates), information gives meaning to data (e.g., location, cause, and time), and knowledge interprets information based on one's role and responsibility (e.g., clinical document's purpose and effect) @xcite . making a quality annotated corpus is an iterative process that includes designing an annotation guideline, annotating text with the guideline, and refining the guideline based on inter-annotator agreement (iaa) @xcite @xcite @xcite . although studies describe how annotated clinical corpora were made, few studies are explicit about the design process. we believe that the acquisition and transformation of clinical questions about the patient cohort into corresponding corpus requirements for retrieving information from the actual text of the annotated corpus should drive the annotation process. ## related work this section provides an overview of studies that describe the design process leading to an annotation guideline and annotated clinical corpus. studies that share their annotation challenges or offer improvements are also included. the clef corpus was semantically annotated to help develop and evaluate the clef information extraction system @xcite . free-text documents in the corpus are histopathology reports, imaging reports, and clinical narratives (i.e., discharge summaries, reports, case notes, audits, letters, or narratives to the general practitioner, consultant, referrer, or patient). initially, templates for the documents using ontology-based entities and relationships were manually filled-in. however, the templates did not directly align with text, and ontology complexity made it time-consuming to fill templates. thus, @xcite iteratively developed an annotation guideline based on a simplified version of the original ontology and template definitions. following established standard nlp annotation methodology @xcite , 2 clinicians annotated 31 documents over 5 sessions, and a third annotator resolved disagreements @xcite . however, due to workload and time constraints, resigning annotators could have impacted the corpus quality and size. thus, @xcite proposed solutions such as pre-annotated documents and a reduced annotation scope. the i2b2 challenges have annotated corpora for various purposes. for example, in the i2b2 nlp challenge of extracting patient medication from discharge summaries, 79 annotators from 20 teams annotated 251 discharge summaries in a community annotation experiment @xcite . the annotation guideline was developed iteratively in 2 phases before the community annotation. for several iterations in phase 1, university students annotated discharge summaries that were measured for iaa and asked questions to aid revisions. this produced a guideline and 17 annotated discharge summaries for phase 2. finally, during phase 2, teams annotated discharge summaries using the guideline and addressed inconsistencies within the 17 annotated discharge summaries to produce a refined guideline. the i2b2 temporal relations corpus contains 310 discharge summaries annotated by 8 annotators @xcite . the annotation guideline was based on the timeml event and temporal expression specification language @xcite and the temporal histories of your medical event (thyme) project annotation guidelines. the corpus development process included: a guideline development pilot study, data selection, preannotation, annotator training session, 2 annotators annotating pre-annotated documents, an adjudicator who resolved disagreements, and evaluation. the 2014 i2b2/uthealth de-identification corpus annotation guideline focuses on removing protected health information (phi) in longitudinal medical records for automatic de-identification system development @xcite . introduced phi subcategories enable downstream analyses to adjust the scope or focus on specific categories. additionally, they compared parallel and serial annotation processes on pre-annotated and unannotated corpora and found that the process does not affect annotation quality @xcite . @xcite utilized a variation of the typical annotation process for 3 different studies. each study's corpus focused on a specific clinical report, such as radiology, chest x-ray, or intensive care unit reports. the process included: defining a study based on clinical needs, selecting data, gaining ethical approval, writing annotation guidelines, creating annotation tools, annotating, building a system with the corpus, and testing if the system meets clinical needs. physicians were guideline designers and annotators, whereas nlp researchers provided technical support and built nlp systems with the corpora. suggestions for improvement included more nlp researcher involvement, consideration for guideline granularity versus annotation time, marking rationale or evidence for a label, and estimating time commitment. @xcite developed their annotation guideline by building off a previous guideline. the rest of the methods were similar: defining annotation tasks, selecting data from stratified random sampling, and annotating with 2 annotators. during the annotation process, 2 annotators annotated the same documents, iaa was measured, and con-sensus sessions were held to resolve disagreements and update the guideline. using the same annotation process, they built gold standard corpora from clinical trial announcements, us food and drug administration (fda) drug labels, and ehr clinical notes. this included clinical notes such as discharge summaries, referrals, reports, and notes for consultations, procedures, plans, or progress. interested in capturing infections caused by central venous catheters, a nurse specializing in infection annotated 2 745 of 22 174 inspected notes @xcite . before inspection, duplicate notes were removed. the guideline was a table containing events for annotation. defined by computer scientists, nurses, and an nlp domain expert, the annotation labels formed a hierarchy starting with generalized events at the top level and more specific events below. they also provided information about data access restrictions to promote patient confidentiality and clinical record extraction. record extraction included physician and nurse notes for admissions, care, plans, evaluations, transfers, and discharge summaries. in this study, we focus on a method of incremental annotation guideline design by intertwining acquisition with testing of corpus requirements and corresponding annotation phases. this ensures that the guideline produces an annotated corpus that fulfills corpus requirements derived from clinical questions, even if the clinical questions are not answerable by the actual data. to the best of our knowledge, there lacks a study that describes this approach in detail. ## objective this study aims to describe our method for designing a semantically annotated corpus for signs of sepsis by starting from clinical questions that formulate the corpus requirements. hence, the main contributions are: (1) providing a detailed description of the guideline design process before annotation, (2) illustrating the systematic and iterative annotation process taken, and (3) discussing insights from the design and annotation process. ## original adverse event dataset we had access to 18 555 norwegian adverse event (ae) reports extracted from a hospital's electronic incident reporting system @xcite . extracted ae reports described procedural deviations, misunderstandings, resource needs, and risky patient behavior. each report has structured data (i.e., identifier, registration date, reporting hospital unit, if the event is patient-related or security-related, event type, and event severity) and an unstructured free-text note. ## semantic annotation design process this section presents the semantic annotation design process leading up to the annotation process and guideline development. a summary can be found in figure 1 . ## annotation process and annotation guideline development following the schedule, synthetic notes were annotated by 4 annotator groups over 5 sessions using a systematic, iterative annotation process for guideline refinement. in each session, 2 annotators from each group annotated notes based on an annotation guideline using the brat rapid annotation tool (brat) @xcite . afterward, annotations were evaluated for iaa and manually inspected to assess if annotations could fulfill corpus requirements and answer the clinical questions. text was tokenized and annotation labels were assigned to tokens before measuring the iaa f 1 -score. disagreements and ambiguities were discussed within the research group, and comments from annotators were incorporated. next, a computer scientist revised the guideline based on discussions. finally, the process was repeated with a new set of notes and the revised guideline. figure 3 shows an example sentence annotated by 2 different annotators. ## general results from sessions 1-5 over 5 sessions, 8 annotators annotated 100 unique synthetic ae notes to produce 770 annotated synthetic ae notes. from session 1, it was clear that subcategory and attribute names should not be used in more than one category, and synonyms should be avoided. for example, simultaneously having "name" as both a sensitivity subcategory and an attribute for the person category raised questions. furthermore, annotators left relationships, attributes, and notes unannotated because they felt those notes were irrelevant to answering the clinical questions. the need for annotating relationships, attributes, and all notes for ml was addressed in session 2. red font emphasized guideline revisions, and the guideline began with an "overview of updated instructions" section to aid annotators in identifying revisions. in sessions 2 and 3, the main revisions were correcting and including missing subcategories to address annotator concerns. session 4 provided a structured terminology for the guideline. a terminology was developed from the guideline to give structure and provide users quick insight into the annotated corpus for downstream analysis @xcite . this restructured the annotation guideline for session 5 by removing ambiguities and allowed ae note querying to answer the clinical questions. for example, the new observation category encompasses the sign category's signs and symptoms and the procedure subcategory "device malfunction signs." the computer scientist who revised the guidelines misinterpreted clinical knowledge and made incorrect assumptions in the previous sessions, so the terminology and restructured guideline were validated by nurses to ensure medical concepts were used correctly before session 5. the session 1-4 annotation guidelines were made available online foot_0 for @xcite , and the session 5 annotation guideline was added online for this study. iaa for different sessions are in figure 4 . ## conclusion our method captures knowledge about sepsis signs in clinical text. we control changes in the annotation guideline by using hierarchical categories and continuous evaluation. through applying a systematic, iterative annotation process, we evaluated the changes using the clinical questions and iaa. the clinical questions evaluate corpus quality, and iaa evaluates annotator compliance and guideline complexity. as the guideline is designed to answer different clinical questions, it is possible to adjust the granularity level as needed to answer different clinical questions. by detailing our design process and annotation process, we hope our method can aid other researchers who cannot utilize pre-annotated corpora in developing an annotated corpus for their research.