Efficient Elicitation of Fictitious Nursing Notes from Volunteer Healthcare Professionals (2025)

Abstract: Reliable automatic solutions to extract structured information from free-text nursing notes could bring important efficiency gains in healthcare, but their development is hampered by the sensitivity and limited availability of example data. We describe a method for eliciting fictitious nursing documentation and associated structured documentation from volunteers and a resulting dataset of 397 Danish notes collected and annotated through a custom web application from 98 participating nurses. After some manual refinement, we obtained a high-quality dataset containing nurse notes with relevant entities identified. We describe the implementation and limitations of our approach as well as initial experiments in a named entity tagging setup.

## introduction with the emergence of electronic health records (ehr), the way nurses document their work has changed drastically. printed schemas and handwritten notes were supplanted by computer-based systems like the danish sundhedsplatformen (sp), aiming to reduce data redundancy and errors @xcite . to simplify automatic processing and data reuse, ehr systems emphasize structured documentation. this choice has been described as "technological somnambulism" @xcite and tends to be at odds with the preferences of the clinical professionals, who value usability and flexibility @xcite and experience structured documentation as time-consuming and inefficient @xcite , frequently leading to inadequate documentation @xcite . automatic generation of structured documentation from free-text nurse notes would offer an at-tractive solution to this dilemma. however, the development of such systems across countries and languages is frustrated by the lack of training data due to the stringent privacy constraints surrounding all forms of medical notes @xcite . while some relevant datasets are available @xcite , they are specific to the context in which they were produced and may be of limited use in another location characterised by a different language, different social context or different healthcare procedures. in this paper, we describe and evaluate a method to elicit fictitious nurse notes from volunteering healthcare professionals based on visual stimuli. the collected notes closely mirror real free-text nursing documentation without suffering from the privacy restrictions of authentic notes. emphasising a low time commitment for the volunteers, our method enabled us to collect a high-quality dataset of 397 notes from 98 participating nurses. we describe our procedures for eliciting and curating the dataset and annotating it for information extraction as well as initial experiments on automatic extraction of structured data. our dataset is in danish, but the procedure would be easily generalisable to other languages. ## data collection framework we collected fictitious examples of nursing notes, together with structured annotations of their content, with two goals in mind: 1. the notes collected should mimic authentic nursing notes as much as possible. 2. the entry threshold for participants should be minimal to make recruitment easier. we used visual stimuli to minimize the influence of the stimuli on the participants' word choice, and imposed a time limit on the text entry to simulate real-life time pressure. figure 1 shows the structure of our web application, whose core parts are the stimulus presentation, note capture and structured annotation. dif- stimuli introduction write note annotate highscore/end figure 1: data collection process. after annotating the participant gets the option to repeat or stop. ferent sets of test participants were used to evaluate the design and offer feedback on the web application during the design process. some of the test participants were observed doing the process, other were interviewed afterwards. as we considered a denser and more focused dataset more useful than a sparse dataset covering many areas, some of the nurse-relevant problem areas were omitted in the our data collection to increase the number of items per category. introduction. the introduction page consists of a 4-step guide, including three small video clips demonstrating the process of seeing a stimulus, writing a note and annotating it. initially the introduction included detailed instructions to the participants. however, during testing, most of the test participants did not read the text and quickly pressed "next" to move on to the next step, which led to confusion about the process. to mitigate this, the text was cut significantly and the introduction page was redesigned with three gif animations demonstrating the process. the facebook post advertising this study also described that the purpose was to create fictitious free-text nursing documentation. ## collected data the study was advertised four times in a facebook group with 30,000 nurses, and three medical wards were visited once each to recruit participants. a total of 98 nurses participated in the study, producing 407 notes and 594 annotations. we expect that this number could be increased by offering economic incentives for participation. every note and annotations was manually reviewed for quality control. ## entity tagging exact matches only needed the start and end positions of the instance to make a complete tag, which was done automatically using regular expressions. tags for the partial matches were done manually as the value in the original annotation did not match the instance in the note exactly. some annotations were straightforward, while others required additional work. for example, participants could choose the color "yellow" for urine. however, since the relation to urine was conveyed in the subcategory, this relation was lost. to address this, additional entity types were created. for example the entity type "out" (as something leaving the body), was created for words like "urine" and "stool". the resulting tagset was designed to ensure that, if all entities were accurately identified and appropriately combined, the original structured annotation could be reconstructed. after settling on a tagset the process of tagging all notes began. one person tagged the dataset, using approximately 20 hours. every note was looked at four times. beyond the notes that already had an annotations, every non-annotated note were tagged as well. a total of 23 entity types were used ( ## experiments extracting entities from the dataset could prove to be difficult. some verbs, like "walk", belongs to different categories based on the tense of the word and the surrounding words. the word "big" ("store" in danish) is used both as a description of an amount "the patient consumed two big portions of food" or as a modifier "the patient have big problems eating" (directly translated from danish). additionally, some entity types appear much less frequently than others, resulting in an unbalanced dataset where entities occur between 13 and 201 times. lastly words like "nasogastric tube" (nasalsonde in danish) and "foley cathether" (kad in danish) are not common words and very specific to the medical domain, which might affect the results in a negative way. ## conclusion this study aimed to bridge the gap between structured and free-text documentation in healthcare using nlp techniques. the initial step involved constructing a dataset, which was necessary due to the absence of pre-existing suitable datasets in this domain. following dataset construction, the study focused on extracting relevant information from nursing documentation within this newly created dataset. the creation of a synthetic dataset of annotated nurse notes was accomplished through a web application. this application presented various stimuli to participants, prompting them to write corresponding notes. subsequently, participants annotated their notes using categories reflective of those used in actual ehrs. overall, the quality of the notes was high, although not all annotations were usable. a manual process was employed to eliminate incorrect annotations and convert the annotations into pairs of (entity type, entity). additional support entities were manually added, ensuring that every word relevant to nurse documentation was properly tagged. the process of extracting meaningful information from nurse documentation was approached as a ner task. performance evaluation revealed that the the danish, norwegian and multilingual models had similar performances, with the best being the danish which achieved a macro f1 score of 0.744, surpassing the crf model, which scored 0.704. this performance difference highlights the necessity and efficiency of more advanced models like bert in handling complex ner tasks. however, it is important to note that the entity type/entity instance pairs extracted through this ner process do not directly correspond to the structured format which is used in ehrs. this gap underscores a potential area for future research, where the focus could be on transforming these pairs into ehr-compatible triples. such a transformation is crucial for the practical application of this research in real-world ehr systems, potentially facilitating smoother integration of automated nlp-based documentation tools into healthcare workflows. nevertheless, this study demonstrates that it is possible to generate synthetic nurse notes and extracting information relevant to nurse documentation from them.