Synthetic Data Generation and Multi-Task Learning for Extracting Temporal Information from Health-Related Narrative Text (2021)

Abstract: Extracting temporal information is critical to process health-related text. Temporal information extraction is a challenging task for language models because it requires processing both texts and numbers. Moreover, the fundamental challenge is how to obtain a large-scale training dataset. To address this, we propose a synthetic data generation algorithm. Also, we propose a novel multi-task temporal information extraction model and investigate whether multi-task learning can contribute to performance improvement by exploiting additional training signals with the existing training data. For experiments, we collected a custom dataset containing unstructured texts with temporal information of sleep-related activities. Experimental results show that utilising synthetic data can improve the performance when the augmentation factor is 3. The results also show that when multi-task learning is used with an appropriate amount of synthetic data, the performance can significantly improve from 82. to 88.6 and from 83.9 to 91.9 regarding micro-and macro-average exact match scores of normalised time prediction, respectively.

## introduction extracting temporal information from text is important linguistic skill to process health-related text. also, there are a lot of potential applications of temporal information extraction in the healthrelated domain, including forecasting treatment effect @xcite , early detecting diseases @xcite , and tracking treatment progress @xcite . with the recent trends of telehealth, an automated system that can extract temporal information from the health-related narrative text can provide benefits to not only healthcare professionals but also recipients enabling active engagement, such as selfmonitoring. in this paper, we consider the use-case of a sleep diary, which is a summary of sleep designed to gather information about daily sleep patterns @xcite . a typical sleep diary consists of a series of close-ended questions to record the time. by writing sleep diaries, people can keep track of sleep, monitor sleep habits, and document sleeping problems which can be shared with their sleep therapists. we focus on extracting temporal information from a free-text sleep diary, to achieve this, a system should extract temporal expressions from the unstructured user-generated text and normalise the extracted temporal expressions into a standard format, as illustrated in figure 1 . temporal information extraction from usergenerated text is a challenging task. first of all, it requires processing not only text but also numbers (e.g., 11pm or 23:00). but recent pre-trained language models @xcite have difficulty in processing numbers @xcite @xcite because these language models are pretrained with language modelling objectives. even though there have been recent studies on training language models to process numerical information @xcite , the remaining challenge is how to obtain a large amount of training data. a second challenge is that there are various ways of describing the same normalised time. for example, the normalised time 23:00 can be expressed as 11, 11 pm, 23:00, eleven o'clock, etc. this issue is, even more, severe when dealing with usergenerated text that is typically noisy: the usergenerated text is prone to spelling errors and grammatical errors and contains a lot of abbreviations @xcite . to address this, a sufficient amount of training dataset containing pairs of various temporal expressions and normalised time values is required. a third challenge is that there are different types of temporal expressions which of each is difficult to extract. for example, temporal expressions include not only standalone times (e.g., 23:00) but also relative times (e.g., 5 minutes after), counts (e.g., 3 times), duration (e.g., for an hour), and frequencies (e.g., once per hour). for relative time expressions, the challenge is how to annotate temporal expressions and model dependencies. for count time expressions, the challenge is to deal with ambiguous terms, such as 'several times' and 'a few times'. the last challenge is how to collect large-scale data while developing a proof-of-concept model to validate the hypothesis. especially for healthrelated data, the data collection requires rigorous process of considering privacy and ethical aspects, which might result in a slow process. moreover, typical machine learning development process includes the multiple cycles of collecting a new dataset and updating a model to improve the performance of model. therefore, the challenge is how to train a machine leanring model when only a low very low amount of training data is available. therefore, the main research question of this paper is how to extract temporal information from user-generated noisy text with the limited number of training data. to this end, we propose a synthetic data generation algorithm to augment the size of training data. we also propose a multitask model and investigate whether the multi-task learning strategy is beneficial to the target task by exploiting additional training signals from the existing training data. the main contributions of this paper include the followings: ## related work there are two lines of approach in temporal information processing. one is rule-based and the other is machine learning-based. generally, rule-based systems achieve high performances in a normalisation task @xcite . however, rule-based systems have difficulties in dealing with ambiguous phrases or relative expressions @xcite . another line of approach is machine learningbased approaches. previous works have focused on detecting temporal links between entities and classify the temporal relations between them @xcite ) rather than predicting the exact time of events. recently, @xcite propose a system that can directly extract start and end-points for events from the text. however, the remaining gap is that it is not entirely end-to-end: @xcite used the text with ground truth event spans and normalized temporal expressions as inputs. moreover, even though machine learning models show promising results, the fundamental challenge is how to obtain data. not only data acquisition can be difficult but also data labelling can be time-consuming and expensive. ## sleep diary analysis this section describes the dataset collected for experiments. the following subsections explain the details of use-case definition, data collection protocol, data labelling scheme, and an initial data analysis result. ## multi-task temporal information extraction model 4.1 task formulation we formulate a temporal information extraction and normalisation task similar to a question and answering task. therefore, each data point is transformed @xmath0, text, @xmath1, the text is sleep diary text, and the answer is a normalised time with a type value and 4 digits. for example, a system is expected to predict a list of answers [none, 2, 2, 3, 0] given @xmath2, i went to bed at half past 10...>. formally, an entity q @xmath3, where q is the set of the sleep-related event entities decribed in then the task is to predict an answer a @xmath5, a t1 ij , a t2 ij , a t3 ij , a t4 ij ] given a sequence of tokens [[cls] q j [sep] p i , [sep]], where [cls] and [sep] are special tokens for classification and separation, respectively. a type ij is the ground truth label for the type value of the normalised time and a t1 ij , a t2 ij , a t3 ij , and a t4 ij is the the ground truth labels for the each digit of the normalised time. ## discussion effects of using synthetic data the first row of ## limitations and future work even though we show the effectiveness of the proposed method by validating on the collected dataset, some points can be further studied. first of all, the proposed models estimate a normalised time value conditioned on an input which is a pair of sleep diary text and sleep-related event entity label. however, since most sleep diary texts do not contain all the event entities, an additional module is required to detect which entities are mentioned in the given text and how many times each entity is mentioned in the given text. similar to the previous work by @xcite , the answer span detection head of the proposed multi-task model can be used as a detection module. secondly, even though the proposed models can handle relative expressions by using specific type values (i.e., +, -), a linking algorithm is currently missing. a potential solution is to add a head that can predict a starting point, similarly to the previous work by @xcite . the third limitation is that the proposed models process only temporal information. to completely analyse sleep diary, extracting contextual and qualitative information is required @xcite . in the future study, we will train a model to extract both temporal and other information from text data. to achieve this, we will collect more data that are longer and contain rich information about the context of the night and sleep. ## conclusions in this paper, we propose a model that can extract temporal information from health-related narrative text. we conducted experiments to investigate how to utilise synthetic data and multi-task learning to improve the performance of normalised time prediction. experimental results show that utilising synthetic data for training can contribute to performance improvement the most when the augmentation factor is set to 3. the results also show that when multi-task learning is used with synthetic data appropriately, the performance can be significantly improved. in the future study, we will extend the current work to extract not only temporal information but also contextual and qualitative information from text.