Injecting Domain Knowledge in Language Models for Task-oriented Dialogue Systems (2022)

Abstract: Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data. Previous studies augmented PLMs with symbolic knowledge for different downstream NLP tasks. However, knowledge bases (KBs) utilized in these studies are usually large-scale and static, in contrast to small, domain-specific, and modifiable knowledge bases that are prominent in real-world task-oriented dialogue (TOD) systems. In this paper, we showcase the advantages of injecting domain-specific knowledge prior to fine-tuning on TOD tasks. To this end, we utilize light-weight adapters that can be easily integrated with PLMs and serve as a repository for facts learned from different KBs. To measure the efficacy of proposed knowledge injection methods, we introduce Knowledge Probing using Response Selection (KPRS) – a probe designed specifically for TOD models. Experiments on KPRS and the response generation task show improvements of knowledge injection with adapters over strong baselines.

## introduction pre-trained language models (plms), such as bert @xcite , bart @xcite , @xcite , and xlnet @xcite , have advanced the state-of-the-art of various natural language processing (nlp) technologies and demonstrated an exceptional ability to store and utilize linguistic, factual, and commonsense knowledge. consequently, plms form the backbone of many recent nlp applications and have been successfully employed as modular components in the context of task-oriented dialogue (tod), responsible for sub-tasks including dialogue state tracking and response generation @xcite . since they are exposed to large quantities of general data during training, plms store a wide variety of diverse and general knowledge in their parameters @xcite such as capitals of nations, biographical details of famous individuals, and other facts of varying granularity. commercially deployed tod systems, however, typically require access to more restricted, domain-specific categories of knowledge in order to produce informative and factually accurate responses to user queries. 2 such information may include addresses of particular local attractions, detailed restaurant menus, train routes, or ticket prices, and is unlikely to be found in the plm's training data. due to its specialized nature, this knowledge is often stored in external knowledge bases (kbs) that are accessed at run-time by tod systems via external queries. this process introduces additional complexity into the dialogue model design and requires implementing kb queries and code wrappers as part of system setup, causing a substantial overhead especially for non-experts. querying external kbs can also be disadvantageous when the kb is small, or is not changing in real time (as is the case with catalogs, restaurants' menus, etc). we identify the decoupling of domain-specific knowledge from the dialogue model as a shortcoming to be remedied and instead propose to inject this knowledge directly into the model's parameters. this eliminates the need for querying external kbs, streamlining the creation and deployment of tod systems. injecting domain-specific information into tod systems that can guide and inform model behavior and may be subsequently updated and modified by the user is not a trivial task. ideally, this should be accomplished in a manner that is efficient, architecture-agnostic, and compatible with off-the-shelf plms. in order to satisfy these requirements, we adopt light-weight adapter networks as repositories of domain-specific knowledge (kb-adapters for short). such adapters can be trained to memorize kb facts 3 and integrated into pretrained plms through the fusion of hidden representations, as illustrated in figure 1 . our work is in line with past studies that demonstrated the utility of adapters as stores of factual and linguistic knowledge outside of tod @xcite . importantly, injecting knowledge into tod models through adapters is computationally less demanding than injecting domain-specific facts by finetuning entire dialogue models on synthetic data, as explored in @xcite , which facilitates efficient updating of the injected knowledge. to quantify the success of the knowledge injection procedure, we develop the knowledge probing using response selection (kprs) task and benchmark (see §3). kprs leverages contrastive dialogue response pairs to probe the extent of memorization of domain-specific facts by the evaluated dialogue model, whereby one response is consistent with the corresponding kb, while the other is not. to our knowledge, both kprs and the use of adapters for domain-specific knowledge injection in tod represent novel contributions of our work. we conduct experiments that evaluate plms equipped with domain-specific kb-adapters on the kprs benchmark as well as the more conventional 3 we use the term fact to refer to individual kb entries. our contributions can be summarized as follows: ## kb-adapters for domain-specific knowledge injection we conceptualize kb adapters as repositories of domain-specific information that guide the plms' predictions to be consistent with kb contents. the proposed knowledge injection process is divided into two stages: (1) memorization: adapters are trained to memorize domain-specific kb facts; (2) utilization: plms are trained to leverage adapters when reasoning about entities and their attributes. during the memorization stage, adapters are connected to the frozen plm and tasked with reconstructing corrupted kb facts, thereby memorizing associations between entity and attribute mentions. during the utilization stage, the plm (now unfrozen) is given access to frozen adapters and learns to leverage their memorized knowledge to make more accurate predictions on downstream tasks such as rg. as a result, plms can generalize to unseen inputs by virtue of their domain-general pretraining while receiving domain-specific guidance in their predictions by the knowledge encoded in adapter representations. when training kb-adapters, we allocate a single adapter for each individual domain kb (e.g. hotel or restaurant). this results in shorter training times per adapter and (if needed) facilitates efficient re-training of adapters to reflect changes in the associated kbs. 4 this allows for a straightforward extension of tod systems equipped with kb-adapters to new domains, as this only requires training a single, new domain-specific adapter that can be used in concert with existing ones. nevertheless, we also consider a setting where we train a ## knowledge-probing using response selection (kprs) benchmark in this study, we investigate the ability of language models to verify and retrieve domain-specific facts within the tod setting. to this end, we propose the "knowledge-probing using response selection" (kprs) task and the associated benchmark. kprs allows us to examine whether domain-specific knowledge, such as entities and their attributes, that is stored within the parameters of the evaluated model can be successfully accessed and guide the model's predictions. being knowledgeable about domain-specific entities in this manner can benefit dialogue models when reasoning about and replying to user queries. we show this to be the case for the response generation task in §5.3. kprs is a contrastive evaluation benchmark that measures whether the probed model has memorized and can accurately retrieve domain-specific knowledge contained within a specified kb. it is derived from multiwoz 2.2 dialogues @xcite (development and test portions only) and covers four domains: restaurant, hotel, attraction, and train. given a dialogue context, the task presented to the evaluated model is to score responses that are either compatible or incompatible with the information contained in the kb. importantly, kprs should not be regarded as a stand-alone evaluation task, but rather as a probing mechanism that can offer informative insights into a model's ability to access domain-specific facts stored within its parameters, similar to other knowledge probes, e.g. @xcite . specifically, a fact-aware model should be able to distinguish between an appropriate ("reference") dialogue response that is compatible with the knowledge base information from an inappropriate ("distractor") response that contradicts the domainspecific knowledge. by design, the two responses are minimally different -identical except for attribute values associated with entities described in the kb, such as restaurant names or departure times of trains. hence, to identify the correct dialogue response, a model must be able to distinguish values that are compatible with domain-specific information from those that are not. ## results & analysis we examine the models' ability to memorize and retrieve facts learned from the knowledge base in §5.1 and the impact of knowledge injection on downstream tasks in §5.2 and §5.3. models were evaluated in the single-domain setting where only one single adapter corresponding to the specified domain was active at evaluation time with test samples belonging exclusively to the adapter domain (a multi-domain setting is discussed in §5.5) ## limitations one of the main limitations of the presented approach is its reliance on manually constructed fact templates. we experimented with fine-tuning kg-adapters directly @xmath0, rel, @xmath1, but found that the use of templates improves the ability of models to apply the memorized knowledge in downstream applications. in light of this, possible future extensions of our work may include creation of domain-agnostic strategies for knowl-edge injection that do not necessitate manual design of templates for each new domain. another limitation comes from the fact that the proposed approach is suitable only for static and pseudo-dynamic kbs , i.e. that can change periodically, such as a seasonal menu or a database of cars manufactured by a company. however, it is not suited for real-time databases (e.g. databases that store the availability of rooms in a hotel) since for every kb change the corresponding adapter needs to be retrained in order to be updated. additionally, while injecting knowledge into the language model has been shown to be effective for making it available during fine-tuning on downstream tasks, the knowledge stored in the adapters' parameters might not be accurate enough for certain real world applications due to the imperfect fact memorization we observed in our experiments. finally, the introduced kprs task only evaluates the extent to which a model can access factual information stored in its parameters. it does not not assess the model's ability to understand and use this knowledge for complex reasoning tasks, e.g. counting the number of cars in a specific price range, or listing the items on a menu that do not contain a certain ingredient. this could be an exciting direction for future research. ## discussion and conclusion in this study, we proposed a method for tightly integrating external knowledge with the internal representations of plms by storing domain-specific information within light-weight adapter networks that guide model predictions. such adapters can memorize kb contents with high accuracy, which decreases slightly for larger kbs. an important contribution of our work is the kprs probe designed to measure the ability of tod models to reason about kb entities and their attributes. as part of our experiments, we showed that kb-adapters clearly benefit the identification and generation of tod responses that are consistent with dialogue history and relevant kb entries, and showcased the advantages of using adapters for knowledge injection as opposed to sequential fine-tuning. our investigation demonstrates that dialogue models can access domain-specific knowledge without having to query external kbs. this is an important finding as it can pave the way towards reducing the query engineering overhead in chatbot design, thus lowering the entry barrier for developing and deploying real-world tod systems.