Information Extraction from S wedish Medical Prescriptions with Sig-Transformer Encoder (2020)

Abstract: Relying on large pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) for encoding and adding a simple prediction layer has led to impressive performance in many clinical natural language processing (NLP) tasks. In this work, we present a novel extension to the Transformer architecture, by incorporating signature transform with the self-attention model. This architecture is added between embedding and prediction layers. Experiments on a new Swedish prescription data show the proposed architecture to be superior in two of the three information extraction tasks, comparing to baseline models. Finally, we evaluate two different embedding approaches between applying Multilingual BERT and translating the Swedish text to English then encode with a BERT model pretrained on clinical notes.

## introduction medical prescription notes written by clinicians about patients contains valuable information that the structured part of electronic health records (ehrs) does not have. manually extracting and annotating such information from large amount of textual fields in medical records such as prescription notes is time-consuming, and expensive since it has to be provided by domain specialists who are in high demand. information extraction (ie), a specialised area in natural language processing (nlp), refers to the automatic extraction of structured information such as concepts, entities and events from free text @xcite . while rulebase ie systems are still widely used in healthcare @xcite , machine learning-based data-driven approaches have gained much more interest due to their ability to scale and learn to recognise complex patterns. in this paper, we present a set of medical prescriptions written in swedish and develop a neural networkbased system that extracts important information from the textual data. recent advances in pretraining large-scale contextualised language representations, including @xcite and bert @xcite , have proven to be a successful strategy for transfer learning and pushed the performance in many nlp tasks of general purpose. clinical notes differ substantially to general-domain text and biomedical literature, in terms of its linguistic characteristics. several studies have finetuned general-domain language models such as bert on in-domain clinical text (e.g. electronic health records) @xcite @xcite for downstream clinical nlp tasks. as summarised in @xcite many such tasks can be formulated as a classification or regression task, wherein either a simple linear layer or sequential models such as lstm or conditional random field (crf) are added after the bert encoding of the input text as task-specific prediction layer. bert's model architecture is based on the transformer model @xcite , which employs the self-attention mechanism to attend to different positions of the input sequence. as it doesn't contain any recurrence or convolution, the model has to add positional encoding in order to model the order of the sequence (e.g. word order). signature transform, initially introduced in rough path theory as a branch of stochastic analysis, is a non-parametric approach of encoding sequential data while capturing the order information in the data. it has shown successes in the recent years in a range of machine learning tasks involving sequential modelling @xcite @xcite . we hypothesise the signature transform method can be integrated in the transformer model, in which its ability to naturally capture sequential ordering complements the ability of attending to important positions and learning long-range dependencies in self-attention. as a preliminary study, our aim is to develop a machine learning-based system that extracts relevant information from the swedish medical prescription notes foot_0 , namely quantity, quantity tag and indication. the contributions of this work are as follows: one encodes the swedish text directly using multilingual bert (m-bert) @xcite while for the other approach we translate the prescriptions and then apply clinical-bert @xcite ) that is pretrained on clinical text in english. ## background clinical language representation has attracted an increasing interest in the natural language processing (nlp) community (kalyan and sangeetha, 2020). more recently with the advent of bidirectional encoder representations from transformers (bert) @xcite , fine-tuning of general-domain language models has been widely adopted for many clinical nlp tasks foot_1 . among them two clin-icalbert studies @xcite have conducted fine-tuning from either bert or biobert @xcite ) using de-identified clinical notes from mimic-iii @xcite . in particular, @xcite showed the effectiveness of its clinicalbert-based model for predicting hospital readmissions. many nlp applications in the medical domain can be formulated as either token classification, sequence classification or sequence regression, in which a pretrained language model such as clin-icalbert can be used to encode the input token sequence (which returns either the encoding for every input token or just the [cls] token) and then a task-specific prediction model is added on top to generate the final output @xcite . in this section, we describe the transformer encoder and the signature transform, which are the bases of our proposed architecture. ## sig-transformer encoder in this section, we present our proposed extension to the transformer encoder (te), named sig-transformer encoder (ste). as depicted in figure 1 , ste consists of a stack of n identical layers, and each layer has three sub-layers. it replaces the multi-head self-attention sub-layer in te with our additive multi-head sig-attention mechanism, followed by two position-wise fully connected feed-forward sub-layers. same as the transformer model, we also employ a residual connection @xcite at the last sub-layer. followed by layer normalisation @xcite . the two feed-forward sub-layers and the embedding layers, produce output of the same dimension d @xmath0. ## data our proposed sig-transformer encoder can be used for general purpose representation learning, and any classifier or regressor can be added on top of it to perform down-stream tasks. we conduct all of our experiments with a medical prescription data for the class of "beta-blockers (atc code c07)", provided by the karolinska university hospital. it is sampled from a much larger database of 41 million medical prescription records. this dataset contains 3,852 non-duplicated prescriptions written in swedish. some examples of annotated prescriptions are shown in the medical practitioners at the hospital have annotated three labels, in which we use quantity for regression while quantity tag and indica-tion as two classification tasks: one challenge of the data is the remarkable class imbalance in its labels, as shown in ## conclusions and future work in this work, we propose a new extension to the transformer architecture, named sig-transformer encoder (ste), by incorporating signature transform with the self-attention mechanism. as a preliminary study, we aimed to automatically extract information related to quantity, quantity tag and the indication label from a swedish medical prescription dataset. we demonstrated good performance in two of the three tasks in a multi-task learning framework. lastly, we compared two embedding approaches between applying clinical-bert (on translated text) and multilingual bert (m-bert). although one of our proposed models, namely m-bert+ste, reported the best performance for quantity and quantity tag, all models failed to perform on the indication task which we provided possible explanations. for future work, we plan to apply the proposed ste models to the much larger prescription database and investigate ways to improve the labelling for indication. we also plan to benchmark our models on other clinical tasks such as predicting hospital readmission @xcite , as well as the more general nlp tasks evaluated in @xcite . moreover, a further investigation to better understand the contribution of signature transform and the interplay between signature and attentions would be very insightful.