Lexical Knowledge Internalization for Neural Dialog Generation (2022)

Abstract: We propose knowledge internalization (KI), which aims to complement the lexical knowledge into neural dialog models. Instead of further conditioning the knowledge-grounded dialog (KGD) models on externally retrieved knowledge, we seek to integrate knowledge about each input token internally into the model’s parameters. To tackle the challenge due to the large scale of lexical knowledge, we adopt the contrastive learning approach and create an effective token-level lexical knowledge retriever that requires only weak supervision mined from Wikipedia. We demonstrate the effectiveness and general applicability of our approach on various datasets and diversified model structures.

## introduction vacuous responses @xcite , such as, i don't know, are commonly observed in end-to-end neural dialog models @xcite . this is mostly because these models ignore the knowledge that resides in people's minds during a conversation. to bridge the gap, many existing works @xcite have attempted to condition the dialog model on external knowledge, either a sentence or a paragraph, retrieved based on the utterance and/or previous context. this curates datasets with utterance-response-knowledge triples (see fig 1(a) ). these knowledge-grounded dialog (kgd) models, despite demonstrated effectiveness, suffer from two major problems. first, equipping models with sentence-level knowledge alone will limit responses' informativeness and diversity. as shown in fig 1 (a), with the knowledge retrieved giving the utterance, a kgd model can relate j.k rowling to khalsa aid. however, retrieval based solely on sentence embeddings will result in ignorance of lexical knowledge associated with individual tokens. in this example, the knowledge about j.k rowling, covid-19, donates, and india, is ignored during the retrieval, due to the semantic gaps between those lexical knowledge sentences (see fig 1(b) ) and the utterance. this makes it rather difficult (if not impossible) for the model to generate responses carrying relevant information as shown in fig 1 @xcite . second, retrieving knowledge for open-domain dialogs during inference incurs heavier computation, often involving similarity search over tens of millions of passages @xcite . existing systems @xcite alleviate this problem relying on pre-selecting a small candidate set based on tf-idf @xcite , in sacrifice of the diversity and the accuracy of the retriever. directly conditioning the dialog model on the retrieved text, these models are easily effected by the quality of the constructed candidate set and are thus prone to errors @xcite @xcite . in this work, we propose to complement the lexical knowledge into neural dialog models by knowledge internalization (ki), a training approach based on contrastive learning @xcite . the central idea of ki is to integrate more fine-grained lexical knowledge about each input token internally into model parameters (e.g., word embeddings), rather than further conditioning the model on externally retrieved knowledge (e.g., directly copy and/or modify tokens from external knowledge when decoding). our research contributions include: ## related work to address the vacuous responses problem in neural dialog models, researchers propose to ground dialogs on real world knowledge and construct new corpora that contain utterance-response-knowledge triples. specifically, responses are grounded to external knowledge derived from different knowledge sources @xcite @xcite @xcite @xcite @xcite . among different sources, textual knowledge @xcite @xcite receives the most attention as it is easy to obtain and scale. however, the construction of knowledge-grounded datasets is costly and time-consuming. to build a more practical system without assuming a given knowledge, recent studies enhance kgd models with an extra knowledge selection component @xcite @xcite . most existing kgd models can be viewed as models with externalized knowledge, where knowledge is explicitly used as part of the model input. the principle behind these models is to copy words and/or modify sentences from external knowledge when generating responses @xcite @xcite . our ki, on the other hand, does not explicitly present knowledge to dialog models for reading and/or copying. instead, we inject and store external knowledge into models' parameters and encourage models to elicit the encoded knowledge during generation. the idea of knowledge internalization has also been explored in language modeling. factual knowledge @xcite @xcite , visual knowledge @xcite and syntactic knowledge @xcite have been injected into language models (lms) and shown great promise in improving the performance of downstream tasks. ki differs from those knowledge-enhanced lms in two aspects: (i) ki can be trained end-to-end with dialog models, while applying lms on dialog generation often requires multiple rounds of pre-train and fine-tune. (ii) ki is lightweight that barely introduces extra parameters to the dialog model while applying lms usually introduces hundreds of millions of extra parameters. ## knowledge internalization for neural dialog models in this section, we illustrate how to train a dialog model with knowledge internalization. to infuse more fine-grained lexical knowledge to a neural dialog model, we assume a dialog corpus where each token is aligned with relevant knowledge (we will discuss the construction of such a corpus in §4). in particular, for an input sentence x in the corpus, we assume each token x @xmath0. ## retrieval of token-level lexical knowledge in this section, we present how to train an effective retriever to mine knowledge for each token in the dialog corpora. given a dialog utterance @xmath1, x 2 , . . . , x n }, the trained retriever will retrieve a relevant knowledge k i for each token x i in x. the constructed token-knowledge alignments can then be used to train a knowledge-internalized neural dialog model as in § 3. ## experimental setups in this section, we present the datasets and metrics used for evaluation. ## experiments we evaluate the performance of ki by comparing it with three sets of baselines: all model structures and training setups are given in appendix b. ## method analysis in this section, we perform an in-depth analysis to understand the effectiveness of ki. ## conclusion we propose knowledge internalization (ki), which aims to incorporate the lexical knowledge into neural dialog models. models with ki can generate informative and diverse responses without explicitly conditioning on external knowledge. to provide the fine-grained knowledge needed in ki, we also build an effective token-level lexical knowledge retriever that contextually align tokens in a sentence to their related knowledge. we show the effectiveness and general applicability of ki by evaluating ki on various datasets and diversified model structures.