P ri M ock57: A Dataset Of Primary Care Mock Consultations (2022)

Abstract: Recent advances in Automatic Speech Recognition (ASR) have made it possible to reliably produce automatic transcripts of clinician-patient conversations. However, access to clinical datasets is heavily restricted due to patient privacy, thus slowing down normal research practices. We detail the development of a public access, high quality dataset comprising of 57 mocked primary care consultations, including audio recordings, their manual utterance-level transcriptions, and the associated consultation notes. Our work illustrates how the dataset can be used as a benchmark for conversational medical ASR as well as consultation note generation from transcripts.

## introduction the use of automatic speech recognition (asr) is widespread in the clinical domain but it is generally used to alleviate the administrative burden of clinical notes through dictation @xcite . however, the adoption of telemedicine, especially in primary care, generates vast quantities of clinical interaction recordings. additionally, asr models have become much more robust to applications in the clinical domain. in turn, this is beneficial for downstream natural language processing (nlp) tasks, such as information extraction from clinical conversations @xcite and automatic generation of consultation notes @xcite @xcite . despite this being an active area of research it still lacks a commonly recognised asr benchmark due to the sensitive nature of clinical conversations. furthermore, as the datasets are not shared, research teams always need to invest time and resources into making their own private dataset. these limitations slow down progress in the field. we release foot_0 a high quality public dataset of primary care consultation audio recordings, including manual transcriptions and associated consultation notes, which is the basis of our contributions: as for open-access datasets, he et al. (2020) compile and release two clinical dialogue datasets in chinese and english, covering a wide range of clinical specialties. @xcite do the same for covid-19 related clinical dialogue. these datasets are gathered from online clinical question answering sources; while they are relevant for clinical chatbot research, they are not representative of clinical interactions and do not include audio. @xcite provide a dataset of audio recordings, automated transcripts and consultation notes for 70 mock psychiatric consultations -but no human transcripts. automatic consultation note generation and other long-form text summarisation tasks have rapidly developed due to recent advances in natural language generation (nlg) architectures @xcite . several studies @xcite @xcite @xcite @xcite @xcite use proprietary datasets of transcripts and notes to train nlg models endto-end, and a number of them carry out automatic or human evaluations on their proprietary test sets. however, in a similar fashion to the asr studies discussed above, most studies don't publish these resources; hence, it is again prohibitively difficult to compare their proposed methods. @xcite provide the only open access clinical dataset that could be used as a benchmark but it only contains psychiatric consultations, which is less applicable to primary care. ## dataset the requirements for releasing a dataset containing personal health information (phi) are typically costly and involve collecting patient consent and/or de-identification, which is especially challenging with audio recordings. we built a mock consultation dataset as close as possible to the real conditions as a pragmatic alternative. the diagram in consultation type count otitis 2 anaphylactic reaction 3 cardiovascular 11 dermatitis 4 fever 4 urinary tract infection 6 upper respiratory infection 6 asthma 2 gastroenteritis 8 mental health 3 physical injury 2 migraine 6 figure 1 shows an overview of the data collection process. ## asr benchmark we perform a baseline study of asr for clinical conversations by passing the audio recordings of the mock consultations through commonly used open-source and commercial speech-to-text engines: both models are end-to-end and do not use a language model. ## consultation note generation benchmark we evaluate the models on our dataset and report common summarisation metrics scores: rouge-1, -2 & -l @xcite which compute the f-score across ngrams between generated and human notes; and bertscore @xcite , which computes the similarity between bert embeddings of the notes. the results can be seen in a more detailed evaluation of this task can be found in @xcite ; example notes can be found in appendix table a.3. ## conclusion we present a dataset of 57 high quality mocked consultation audio recordings, their manually aligned and diarised transcripts, and consultation notes. by publishing this dataset, we hope to offer a benchmark for future studies in both asr for clinical conversations and consultation note generation for the primary care domain.