New Domain, Major Effort? How Much Data is Necessary to Adapt a Temporal Tagger to the Voice Assistant Domain (2021)

Abstract: Reliable tagging of Temporal Expressions (TEs, e.g., Book a table at L’Osteria for Sunday evening) is a central requirement for Voice Assistants (VAs). However, there is a dearth of resources and systems for the VA domain, since publicly-available temporal taggers are trained only on substantially different domains, such as news and clinical text. Since the cost of annotating large datasets is prohibitive, we investigate the trade-off between in-domain data and performance in DA-Time, a hybrid temporal tagger for the English VA domain which combines a neural architecture for robust TE recognition, with a parser-based TE normalizer. We find that transfer learning goes a long way even with as little as 25 in-domain sentences: DA-Time performs at the state of the art on the news domain, and substantially outperforms it on the VA domain.

## introduction many natural language processing (nlp) applications rely on a temporal tagger to successfully identify and normalize temporal expressions (tes: e.g. seven in the @xmath0:00). examples include question answering, summarization, and information extraction @xcite . temporal tagging serves to anchor events on the temporal axis and contributes to event ordering sequences @xcite . this is particularly useful for voice assistants (vas), that is software agents such as apple's siri or amazon's alexa, which are able to interpret spoken human queries (commands) and help their users perform simple tasks, including scheduling tasks such as setting reminders or creating and editing calendar events. for example, given the query delete my monday's meeting, a va might have to retrieve information from a calendar corresponding to the day the user is referring to as monday. in order to succeed in such tasks, vas require a reliable temporal tagger, which can identify tes and classify them into categories (te recognition, for example, date vs. time) and then convert them into machine-readable canonical values (te normalization, e.g. seven in the @xmath1:00). the major shortcoming of current temporal taggers is arguably their domain dependence, as it is well known that nlp tools degrade on outof-domain data. the publicly available temporal taggers @xcite @xcite have been developed and evaluated on domain-specific datasets annotated according to the timeml standard @xcite , notably the news @xcite , social media @xcite , narrative @xcite , or clinical domain @xcite . in contrast, to our best knowledge, there is no existing temporal tagger optimized for the va domain, which differs considerably from other domains: it is dominated by concise stand-alone commands, typically referring to single future events (e.g., add yoga to my calendar tomorrow at 6), often outside disambiguating discourse. as a result, coreference and event ordering play a smaller role than in other domains. also, va queries, compared to the news domain, contain more references to the time of an event (at 6) and to regular event repetitions (wake me up every day at 7), as well as more underspecified or vague time expressions (remind me to call mom later this evening) @xcite . a possible solution to overcome the problem of the scarcity of tagged training data for the va domain is to adopt a transfer learning approach @xcite . however, this leaves open the question of what the training curve looks like: how add my appointment at varin salon @xmath2:30" @xmath3:30 @xmath4:30" @xmath5:30 @xmath6. figure 1 : timeml example from @xcite . much data is necessary until performance "flattens out"? we investigate the performance of a temporal tagger pre-trained on news and fine-tuned on the va domain and find that a surprisingly small amount of data (less than 100 in-domain sentences) is sufficient to achieve reasonable performance on the low-resource target domain, substantially outperforming existing systems on the va domain. paper structure. we first contrast annotated data in the news and va domain (sec. 2). after an overview of related work (sec. 3), we introduce da-time, a hybrid temporal tagger for the va domain, which uses a neural model for te recognition and a parsing-based model for te normalization (sec. 4). after describing the experimental setup (sec. 5), we present a detailed evaluation for varying amount of target domain annotations (sec. 6). ## related work the first tempeval challenge @xcite focused on the automatic extraction of temporal relations given a timeml-annotated dataset. tempeval-2 @xcite introduced the task of temporal tagging of tes for the english news domain, consisting in their recognition and normalization, and as a prerequisite for temporal information extraction, which also includes the extraction of events and of their temporal relations. tempeval-3 (uzzaman et al., 2013) extended the task to multilingual settings providing timex3 annotation in english and spanish. more recent tempeval challenges @xcite @xcite also branched out to the clinical domain. as to temporal tagging in different domains (e.g., news, narrative, colloquial, autonomic), @xcite addressed potential challenges, observing that existing temporal taggers work sufficiently well only in the domain they were developed for. this is probably why, to the best of our knowledge, work on temporal tagging has so far only been considered in within-domain settings. tempeval-3 can serve as a showcase of approaches to temporal tagging. the nine participants tackled the task either with rule-based, data-driven, or hybrid methods @xcite . heidel-time @xcite , a rule-based system, obtained the top rank. the system used regular expression-based rules to identify and normalize time expressions in multilingual settings @xcite . later, they extended their rules to cover different domains (e.g., narrative, colloquial) @xcite . when tes were underspecified (e.g. january 6th), domain-sensitive strategies (such as searching for contextual cues or identifying a reference time) were adopted to normalize them (e.g. to normalize january 6th as the previous january 6th or the forthcoming one). as rule-based systems are typically crafted to work for their reference domain, heideltime is not able to identify and normalize expressions that are more typical of concise commands to a va, such as book a slot for the 5th, where the month is not mentioned. uw-time @xcite is a hybrid semantic parsing-based tagger using combinatory categorial grammar @xcite . compared to heideltime, uw-time successfully combines hand-engineered and trained rules, showing the benefit of context-handling over rule-based approach. uw-time can use features such as the tense of a verb to determine if the tes refer to either the past or the future, or can determine if a four-digit number in a text refer to a year or not depending on the context. uw-time was evaluated on the news and narrative domain and set the current state-of-the-art of temporal tagging on the tempeval-3 evaluation set, working exceptionally well but with a high degree of domain specificity. ## da-time we now present a hybrid system for temporal tagging, which we use to investigate domain adaptation of temporal tagging: da-time (for domain-adapted time tagger). da-time is a pipeline of a neural te recognizer and a rule-based normalizer foot_2 . ## conclusion identifying time expressions (tes) is a crucial part of the interaction between a voice assistant (va) and a user, but only small annotated te corpora exist in the te domain. in this paper, we have presented da-time, a hybrid model combining a neural te recognizer with a rule-based te normalizer, and assessed how much data is necessary to finetune da-time on the va domain after pre-training on the much better resourced news domain. we find that our da-time model, which performs competitively with the state of the art on news, can be fine-tuned very effectively on the va domain. while, unsurprisingly, the best performance is achieved with the full target domain training set, already 10% of that dataset -some 25 sentences -is sufficient to achieve major improve-ments over the news-trained model. particularly relevant is the improvement on the value f1 metric, i.e., the quality of the normalized tes. to our best knowledge, this is also the first approach to consider the granularity of temporal unit following the timeml annotation and iso 8601 standard, and to leverage it to recognize tes in parallel with timex3 types in a parallel setting. timex3 type and unit are both crucial inputs for our hybrid normalizer. our normalizer encodes some domain-specific assumptions (e.g., about underspecified tes). these are particularly important in handling long tes. while our normalizer is domain-specific, leveraging on temporal units can ease domain adaptation to new domains. we believe that the small amount of necessary data for fine-tuning is promising for the generalization of temporal taggers for other specific domains. in the future, further improvement may be brought by leveraging anchored time information to identify relations among tes. taking into consideration of other timeml tags (event, signal) can improve some of the current limitations of the model (for example by identifying event-time relationships or prepositional modifiers). more generally speaking, training temporal taggers in a more end-to-end fashion is a promising direction that appears particularly feasible in the voice assistant domain. considering da-time as a baseline model could lead to further neural-based research in the va domain or for other application domains where identification of temporal information is important.