The C hilean Waiting List Corpus: a new resource for clinical Named Entity Recognition in S panish (2020)

Abstract: In this work we describe the Waiting List Corpus consisting of de-identified referrals for several specialty consultations from the waiting list in Chilean public hospitals. A subset of 900 referrals was manually annotated with 9,029 entities, 385 attributes, and 284 pairs of relations with clinical relevance. A trained medical doctor annotated these referrals, and then together with other three researchers, consolidated each of the annotations. The annotated corpus has nested entities, with 32.2% of entities embedded in other entities. We use this annotated corpus to obtain preliminary results for Named Entity Recognition (NER). The best results were achieved by using a biLSTM-CRF architecture using word embeddings trained over Spanish Wikipedia together with clinical embeddings computed by the group. NER models applied to this corpus can leverage statistics of diseases and pending procedures within this waiting list. This work constitutes the first annotated corpus using clinical narratives from Chile, and one of the few for the Spanish language. The annotated corpus, the clinical word embeddings, and the annotation guidelines are freely released to the research community.

## introduction the analysis of clinical text has particular challenges due to the extensive use of non-standardized abbreviations, the variability of the clinical language across medical specialties and health professionals, and its restricted availability for privacy reasons, to mention some @xcite . given that most text resources are available for the english language @xcite , focusing on clinical text in spanish represents an opportunity to gather efforts on its development. a common task in natural language processing (nlp) is named entity recognition (ner), which aims to automatically identify essential pieces of information (entities) in a text written in natural language. in the general domain, ner was first defined to identify personal names, organizations, and locations @xcite , to then be extended to a variety of entities depending on the particular application. nowadays, the best results for the original 2003 ner task @xcite are self-attention networks @xcite , differentiable neural architecture search methods @xcite , and lstm-crf enriched with elmo, bert, and flair contextual embeddings @xcite . in the context of clinical nlp, ner is commonly used for the identification of diseases, body parts, or medications @xcite . the automatic extraction of this information allows, for example, the detection of risk factors on discharge records @xcite , personal information @xcite , frequencies and doses of drugs @xcite , or the leverage of epidemiological information on the existence of diseases @xcite . human-annotated clinical corpora are costly, but they are necessary for at least three reasons: 1) the annotation procedure focuses and clarifies the requirements of a computational algorithm, 2) it provides data for resolving nlp tasks, and 3) it provides a benchmark against which to evaluate the results obtained by computational models @xcite . in practice, the manual annotation process implies that the annotator, with expertise in a subject previously discussed and defined as appropriate, reviews the corpus. following guidelines and an annotation scheme, he or she selects a text segment in the document and assigns it to an entity type and, if appropriate, adds an attribute or relation to connect the segment to another entity. ## the waiting list corpus during 2018, we requested the non-ges wl from the 29 health services in the country through transparency law (ministerio secretaría general de la @xcite . these requests were answered positively by 23 of the health services and sent wl datasets for years between 2008 and 2018. as a result, the group has 5,176,858 referrals, originated at the 40 medical and 11 dental specialties defined in the chilean regulation (ministerio de salud de chile, 2011b). the specialties with more referrals are ophthalmology (14.49%), traumatology (9.44%), and otorhinolaryngology (7.53%). the distribution between medical and dental referrals is 83% versus 11%, and 6% of the referrals have missing values in the specialty attribute. considering only the reasons for referral (written in free-text), we have 994,946 different diagnoses. a random subset of these diagnoses was selected for annotation, with the criterion of selecting those with more than 100 characters. using this condition, we reduce the corpus to 107,235 unique candidates. moreover, we removed diagnoses with text imperfections (such as a clear cut at the end of the referral or a text encoding error) or without extra text information (an exact copy of an icd-10 diagnosis). after filtering, one of the managers inspected each of the remaining diagnoses to ensure that they fully met the conditions. even though the referrals come de-identified from the source, this person also checked for any personal information. ## annotation scheme four annotators (three medical students and one medical doctor) were selected for the initial stage of the annotation process, who were permanently supported by three project managers. the choice of annotators and their background is a significant factor. @xcite describe how clinically trained annotators are better than linguists and computer scientists at annotating clinical text with semantic relationships. it is common to collect annotations from workers with advanced medical training, either as general practitioners, researchers with training on general medicine, or final-year medical students @xcite . we worked here with three third-year medical students, whose annotations contributed to the improvement of the annotation guidelines. the annotation process involved three stages as shown in figure 1 . in the first stage, a test version of the annotation guidelines was written, with an in-depth study of other available guidelines for similar entities, such as those published by @xcite and @xcite . these guidelines were evaluated during the annotation of 25 referrals, followed by the curation of a reference. in the second stage, the three medical students annotated 50 identical referrals in weekly annotation rounds for three weeks. in an iterative improvement process, the medical students were retrained after each round of annotation. at this point, the guidelines were further modified to clarify the task and improve consistency. at the end of this stage, the first accepted version of the guidelines was established and released. in stage three, a medical doctor joined the group (namely senior annotator) and was asked to annotate the same 150 referrals done by the students independently. each referral was compared with the previous annotations, with the aim that the analysis and discussion process to find consensus on annotations helped strengthen the senior annotator's training. recruiting medical doctors to invest time on the annotation task is a challenge. therefore, the option of training non-expert annotators such as students, is often considered. however, the annotation of some complex entities, by definition or extension, may lead to low agreement among nonexpert annotators impacting the overall agreement as well @xcite . we addressed this situation by implementing a pre-annotation stage of straightforward entities, such as abbreviations and body parts, done by medical students. thus, the senior annotator could focus on entities, attributes, and relations that required higher clinical expertise. for the consolidation process, we decided to have each annotation revised by a team of four researchers, including the senior annotator, a dentist, the postdoc that created the annotation guidelines, and the principal investigator. this means that once a batch of 150 referrals was fully annotated, the three managers and the senior annotator analyzed and discussed the annotations one by one until an agreement was reached. when consolidated, the referrals became part of the ground truth. in the beginning, the consolidation of 150 referrals took around 6 hours, but by round 4, the time was reduced to approximately 3 hours. it is important to note that we did not use automatic pre-annotation methods: each of the referrals was manually annotated from scratch. we used this time-consuming approach to compensate for the absence of a second senior annotator. ## conclusions and future work there is a lack of language resources for the spanish language in the clinical domain, and the work presented here constitutes the first annotated corpus using chilean narratives. we believe that projects like ours help filling the gap with respect to clinical nlp done in english. this paper shares 900 annotated referrals, the annotation guidelines, clinical word embeddings, and the code used to generate the results presented here. in terms of developing ner models trained on this corpus, future work includes improving the recall for disease and medication, due to the importance of identifying these entities, and dealing with nested ner, for which there is a variety one of our goals working on this corpus and training ner models is to recognize diseases within this waiting list automatically. in particular, telemedicine has been posed as one of the solutions to decrease the waiting times in the chilean public healthcare sector (ministerio de salud de chile, 2011a). to correctly estimate the effect, one needs to summarize the suspected diagnoses and check which of them are eligible for telemedicine consultations. furthermore, diseases that need to be examined rapidly could be prioritized using an automatic detection of diseases. part of the group's expertise is in the genetic components of diseases. for that reason, we want to explore the possible risk factors (genetic or environmental), which could be obtained from the mentions of the patients' family history and habits. in this regard, we pay special attention to identifying relations between family members and diseases, with maternal and paternal components labeled. this corpus is not particularly rich in those entities. however, we are starting to collaborate with a cancer center, and we plan to translate the know-how from this annotated corpus to future projects in that direction.