DS - TOD : Efficient Domain Specialization for Task-Oriented Dialog (2022)

Abstract: Recent work has shown that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining in downstream task-oriented dialog (TOD). These approaches, however, exploit general dialogic corpora (e.g., Reddit) and thus presumably fail to reliably embed domain-specific knowledge useful for concrete downstream TOD domains. In this work, we investigate the effects of domain specialization of pretrained language models (PLMs) for TOD. Within our DS-TOD framework, we first automatically extract salient domain-specific terms, and then use them to construct DomainCC and DomainReddit – resources that we leverage for domain-specific pretraining, based on (i) masked language modeling (MLM) and (ii) response selection (RS) objectives, respectively. We further propose a resource-efficient and modular domain specialization by means of domain adapters – additional parameter-light layers in which we encode the domain knowledge. Our experiments with prominent TOD tasks – dialog state tracking (DST) and response retrieval (RR) – encompassing five domains from the MultiWOZ benchmark demonstrate the effectiveness of DS-TOD. Moreover, we show that the light-weight adapter-based specialization (1) performs comparably to full fine-tuning in single domain setups and (2) is particularly suitable for multi-domain specialization, where besides advantageous computational footprint, it can offer better TOD performance.

## introduction task-oriented dialog (tod), where conversational agents help users complete concrete tasks (e.g., book flights or order food), has arguably been one of the most prominent nlp applications in recent years, both in academia @xcite @xcite and industry (e.g., @xcite . like for most other nlp tasks, fine-tuning of pretrained language models (plms) like bert @xcite and @xcite pushed the state-of-the-art in tod tasks @xcite , with lm pretraining at the same time alleviating the need for large labeled datasets @xcite . more recent tod work recognized the idiosyncrasy of dialog -i.e., that dialogs represent interleaved exchanges of utterances between two (or more) participants -and proposed pretraining objectives specifically tailored for dialogic corpora @xcite @xcite . for instance, @xcite pretrain their tod-bert model on the concatenation of nine human-to-human multi-turn dialog datasets. similarly, @xcite pretrain a general-purpose dialog encoder on a large corpus from reddit by means of response selection objectives. encoding dialogic linguistic knowledge in this way led to significant performance improvements in downstream tod tasks. while these approaches impart useful dialogic linguistic knowledge they fail to exploit the fact that individual task-oriented dialogs typically belong to one narrow domain (e.g., food ordering) or few closely related domains (e.g., booking a train and hotel; @xcite . given the multitude of different downstream tod domains (e.g., ordering food is quite different from booking a flight) it is, intuitively, unlikely that general dialogic pretraining reliably encodes domain-specific knowledge for all of them. in this work, we propose domain specialization for task oriented dialog (ds-tod), a novel domain specialization framework for task-oriented dialog. ds-tod, depicted in figure 1 , has three steps: (1) we extract domain-specific terms (e.g., terms related to ordering taxi or terms related to buying a train ticket) from the training portions of a task-specific tod corpus; (2) we next use the extracted terms to obtain domain-specific data from large unlabeled corpora (e.g., reddit); (3) finally, we conduct intermediate training of a plm (e.g., bert) on the domain-specific data in order to inject the domain-specific knowledge into the encoder. as a result, we obtain a domain-specialized plm, which can then be fine-tuned for downstream tod tasks, e.g., dialog state tracking. contributions. we advance the state-of-the-art in tod with the following contributions: (i) departing from general-purpose dialogic pretraining (e.g., @xcite , we leverage a simple terminology extraction method to construct domaincc and domainreddit corpora which we then use for domain-specific lm and dialogic pretraining, respectively. (ii) we examine different objectives for injecting domain-specific knowledge into plms: we empirically compare masked language modeling (mlm) applied on the "flat" domain dataset domaincc against two different response selection (rs) objectives @xcite applied on the dialogic domainreddit corpus. we demonstrate the effectiveness of our specialization on two tod tasks -dialog state tracking (dst) and response retrieval (rr) -for five domains from the multiwoz dataset @xcite . (iii) we propose modular domain specialization for tod via adapter modules @xcite . additional experiments reveal the advantages of adapter-based specialization in multi-domain tod: combining domainspecific adapters via stacking @xcite or fusion @xcite (a) performs en par with or outperforms expensive multi-domain pretraining, while (b) having a much smaller computational footprint. 1 ## data collection we create large-scale domain-specific corpora in two steps: given a collection of in-domain dialogs we first extract salient domain terms ( §2.1); we then use these domain terms to filter content from ccnet @xcite as a large general corpus and reddit as a source of dialogic data ( §2.2). 1 assume n mutually close domains and a bi-domain downstream setup (any two domains). with an adapter-based approach, we pretrain one adapter for each domain (complexity: n ) and then combine the adapters of the two domains intertwined in the concrete downstream setup. in contrast, multi-domain specialization would require one bi-domain pretraining for each two-domain combination (complexity: n foot_0 ). ## domain specialization methods the next step in ds-tod is the injection of domainspecific knowledge through intermediate model training on domaincc and domainreddit. to this end, we train a plm (1) via masked language modeling on domaincc and (2) using two different response selection objectives on do-mainreddit. finally, for all objectives, we com-domain ngrams taxi taxi, contact number, book a taxi, booked, time schedule, pickup, leaving, booked type, booking completed, departing, destination, cab, completed booked, honda, ford, audi, lexus, toyota, departure, skoda, lexus contact, toyota contact, ford contact, volvo, train station, departure site, tesla, audi contact, honda contact, skoda contact, picking, departing, volkswagen attraction museum, college, entrance, attraction, information, centre town, center town, entertainment, swimming pool, gallery, sports, nightclub, pounds, park, postcode, architecture, centre area, center area, cinema, church, trinity college, entrance free, jello gallery, post code, town centre, town center, downing college train train station, travel time, leaving, pounds, train ticket, departing, payable, train leaving, cambridge, london, reference id, arrive, destination, kings cross, total fee, departure, arriving, book a train, booked, stansted, stansted airport, peterborough, traveling, trip, airport, booking successful, norwich hotel hotel, nights, parking, free parking, wifi, star hotel, price range, free wifi, guesthouse, guest house, internet, guest, hotel room, star rating, expensive room, priced, rating, book room, moderately priced, moderate price, stay for, reservation, breakfast available, book people, fully booked, booking, reference restaurant restaurant, food, price range, expensive, cheap, priced, chinese food, italian food, moderately priced, south town, book table, city, north town, serving, city centre, city center, european food, reservation, food type, phone address, centre town, center town, expensive restaurant, moderate price, cuisine, restaurant center, restaurant centre, south town, expensive price, east town, cheap restaurant, indian food, asian food, british food, book people ## experiments we demonstrate the effectiveness of our domainspecialization framework by comparing it to nonspecialized baseline models and thoroughly compare different specialization methods from §3. evaluation task and measures. we evaluate our domain-specialized models and baselines on two prominent downstream tod tasks: dialog state tracking (dst) and response retrieval (rr). dst is treated as a multi-class classification task based on a predefined ontology, where given the dialog history, the goal is to predict the output state, i.e., (domain, slot, value) tuples. for our implementation, we follow @xcite , and represent the dialog history as a sequence of utterances. the model then needs to predict slot values for each (domain, slot) pair at each dialog turn. we report the joint goal accuracy, in which the predicted dialog states are compared to the ground truth slot values at each dialog turn. the ground truth contains slot values for all the (domain, slot) candidate pairs. a prediction is considered correct if and only if all predicted slot values exactly match its ground truth values. rr is a ranking problem, relevant for retrieval-based tod systems @xcite . following @xcite and @xcite , we adopt recall at top rank given 100 randomly sampled candidates (r 100 @1) as the evaluation metric for rr. data. in the pretraining procedure, we use the domain-specific portions of our novel domaincc and domainreddit resources ( §2). for the mlm training, we randomly sample 200k domainspecific contexts from domaincc and dynamically mask 15% of the subword tokens. for rs-class and rs-contrast, we randomly sample 200k instances from domainreddit. we evaluate the efficacy of the methods on dst and rr using mul-tiwoz 2.1 @xcite . since we aim to understand the effect of the domain specialization, we construct domain-specific training, development, and testing portions from the original data set by assigning them all dialogs that belong to a domain (i.e., both single-and multi-domain dialogs) from respective overall (train, dev, test) portions. ## results and discussion overall performance. we report downstream dst and rr results in in both dst and rr, tod-bert massively outperforms bert due to its conversational knowledge. domain specialization brings gains for both plms across the board. the only exception is full mlm-fine-tuning of tod-bert (i.e., tod-bert-mlm vs. tod-bert; -4% for rr and -0.8% for dst): we believe that this is an example of negative interference -while tod-bert is learning domain knowledge, it is -because of mlm-based domain training -forgetting the conversational knowledge obtained in dialogic pretraining @xcite . this hypothesis is further supported by the fact that adapter-based mlm specialization of tod-bert -which prevents negative interference by design -brings slight performance gains (i.e., tod-bert-mlm-adapter vs. tod-bert; +0.8% for dst and +1.0% for rr) and is consistent with the concurrent findings of @xcite . overall, domain specialization with rs seems to be more robust than that via mlm-ing, with the two variants (rs-class and rs-contrast) exhibiting similar average performance across evaluation settings. this points to the importance of injecting both the knowledge of dialogic structure as well as domain knowledge for performance gains in tod tasks in the domain of interest. interestingly, the gains from domain specialization are significantly more pronounced for taxi than for other domains. we relate this to the proportion of the single-domain dialogs for a given domain in multiwoz, which is by far the largest (24%, see an encouraging finding is that, on average, adapter-based specialization yields similar gains as specialization via full fine-tuning: given that adapter fine-tuning is substantially more efficient, this holds the promise of more sustainable tod. ## reproducibility we provide the complete lists of keywords used for domain-specific corpus filtering in ## conclusion we introduced ds-tod -a novel framework for domain specialization of plms for task-oriented dialog. given a collection of in-domain dialogs, we extract domain terms and use them to filter indomain dialogic corpora. our experimental study, on five domains of the multiwoz dataset, shows that domain specialization, especially by means of response selection objectives on the dialogic indomain corpora, leads to consistent gains in tod tasks: dialogue state tracking and response retrieval. we hope that our domain-specific resources catalyze research on domain specialization for tod, especially for multi-domain setups. our future efforts will focus on the joint domain-and languagespecialization for task-oriented dialog.