A UGMENTING
T RANSFORMERS
WITH
KNN
B
ASED
C OMPOSITE M EMORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Various machine learning tasks can beneﬁt from access to external information of different modalities such as text and images
Recent work has focused on learning architectures with large memories capable of storing this knowledge
We propose augmenting generative Transformer neural networks with KNN -based Information
Fetching KIF modules Each KIF module learns a read operation to access ﬁxed external knowledge
We apply these modules to generative dialogue modeling , a challenging task where information must be ﬂexibly retrieved and incorporated to maintain the topic and ﬂow of conversation
We demonstrate the effectiveness of our approach by identifying relevant knowledge from Wikipedia, images and human-written dialogue utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.
I NTRODUCTION
Machine learning solutions to various tasks, such as game-playing or dialogue, are often dependent on external information.
This information can take multi-modal forms, including structured knowledge bases, free text, and images, and also comes in overwhelmingly large quantities.
A pressing challenge is to create models that can identify which speciﬁc elements of multiple information sources are relevant, and incorporate them into standard architectures on each task.
Previous work has explored incorporating large external memories into neural network layers (Weston et al., 2014; Sukhbaatar et al., 2015; 2019; Lample et al., 2019).
Many existing approaches focus on using attention over the memory slots, which is computationally intensive and becomes less effective as the the size of the memory grows.
In this work, we propose representing multiple sources of external information as ﬁxed encodings and using K Nearest Neighbors search to fetch relevant information.
KNN search is computationally efﬁcient and scalable, and libraries like faiss (Johnson et al., 2019) allow KNN to be easily used on GPUs and integrated into neural networks.
As the external memories are kept ﬁxed, they do not require any training to learn the memories along with the model.
We can thus scale more easily to larger memories by learning only the KNN-based read operation to identify relevant information from the memory.
Our core contribution proposes an efﬁcient, KNN-based Information Fetching ( KIF ) module that can access relevant external knowledge, combine knowledge from different sources, and integrate this information into standard sequence to sequence architectures.
We apply these ﬂexible modules to two dialogue datasets, challenging tasks where generative models can leverage external information to write coherent, on-topic responses.
We show that relevant information can be identiﬁed from hundreds of thousands of candidates in a multi-modal, multi-knowledge-source setting to improve the performance of generative dialogue models.
On both datasets, we achieve state of the art results compared to generative models and match the quality of retrieval models.
R ELATED W ORK
Incorporating External Knowledge into Neural Networks.
Augmenting neural networks with memory, or longer term components that can be accessed with read and write operations, has been explored in various proposed architectures.
For example, Memory Networks (Weston et al., 2014;
Sukhbaatar et al., 2015; 2019) introduce attention mechanisms over large external memories.
Neural cache models (Grave et al., 2016) simplify these to access previous memories with a dot product.
Previous work has studied how to read and write into these memory architectures (Rae et al., 2016;
Graves et al., 2014; Joulin & Mikolov, 2015).
Another line of research has focused on computational scalability for larger external memories.
For example, Chandar et al.
(2016) propose a hierarchical memory network rather than a ﬂat one and Rae et al.
(2016) learn sparse operations to read and write.
Lample et al.
(2019) focus on learning memories of up to one million slots and how to efﬁciently access the slots using product keys.
Beyond explicit memory representations, it may be possible to store information implicitly during training time by memorizing common patterns present in text (Petroni et al., 2019).
We focus on learning to fetch relevant information from multiple explicit external multi-modal knowledge sources and integrate them into one network.
Work has also focused on computationally efﬁcient softmax operations (Mnih & Hinton, 2009;
Grave et al., 2017; Chen et al., 2015).
Many approximate softmaxes use KNN-like operations to form clusters, and the softmax operation is constrained by the slow calculation of the exponential.
Our use KNN beneﬁts from efﬁcient and scalable libraries such as faiss and nmslib .
Generative Dialogue.
We develop a general architecture for incorporating external information and apply it to the case of generative dialogue models.
Previous work in dialogue has leveraged knowledge as necessary information to accomplish the task.
For example, airline and restaurant booking tasks often use API calls to access information about reservation times and availability (Bordes et al., 2016).
In contrast, our work focuses on how to incorporate unstructured knowledge, such as free text found on the web.
Previous work has employed architectures that attend over the available knowledge and identify relevant pieces of information, which scales poorly with large quantities of information (Dinan et al., 2018;
Qin et al., 2019; Lian et al., 2019).
In this work, we replace the use of attention over external information with the output of our KNN module.
On the modeling side, work has explored both generative (Serban et al., 2016a;b) and retrieval based models (Zhang et al., 2018), which identify the best utterance from the training set to return as the dialogue response.
This often leverages self-attention or cross-attention mechanisms (Humeau et al., 2019).
Further work has explored hybrid models, for example using the output of a retrieval model as input for a generative model (Dinan et al., 2018; Weston et al., 2018).
We extend these approaches by augmenting generative models with retrieval-like operations based on KNN search, allowing dialogue models to ﬂexibly incorporate various sources of external knowledge.
KNN- BASED I NFORMATION F ETCHING M ODULES
Broadly, the KNN-based Information Fetching (KIF) module assumes a model M can access inputs
X = x 1 , x 2 , . . .
, x n to produce outputs O =
o 1 , o 2 , . . .
, o n .
In a setting without additional supporting information, the model will process inputs to make output predictions: M ( x i )
=
ˆ o i .
However, in many tasks, additional information is present, represented as E = { e 1 , e 2 , . .
e m } .
To incorporate E into M , we encode each element of X and E to a ﬁxed-size vector representation.
This can be accomplished in a variety of ways, for example with an encoder neural network.
Then, to make predictions, the model encodes x i and uses K Nearest Neighbors to ﬁnd the closest related information in E .
The representations of the identiﬁed nearest neighbors are combined in a weighted sum, where each of the k retrieved neighbors is weighted by its similarity to x i .
These operations are differentiable, so they can be incorporated into neural networks in a straightforward way.
All elements of the knowledge source E are pre-computed and kept ﬁxed — we do not backpropagate to affect the embeddings of the pre-encoded knowledge.
However, this lack of backpropagation can introduce a mismatch between the encoding of E and the model that is training, as the training model has constantly changing representations because the weights are being learned.
The model must learn a function to align its representations to the external memory.
To circumvent this misalignment, we learn a mapping operator f E ( x ) that maps elements of X into the information representation space E .
Concretely, f E ( x ) is a multi-layer perceptron with ReLU nonlinearities.
From the input elements of X , f E ( x ) learns a representation of an output close to the corresponding projection of X into E .
This can be interpreted as learning a read operation on a ﬁxed external memory.
If there was no change to the encoding of the model compared to the pre-computed knowledge, then the ideal mapping operator would be the identity function.
However, as the model changes signiﬁcantly during the training process, the nonlinear mapping capability of f E ( x ) is essential to be able to identify the correct knowledge E from the input X .
Thus, a model augmented with KIF will incorporate external knowledge in the following manner.
First, we ﬁnd the k nearest elements to the projection of x i in E based on KNN search using inner product, and then the relevant elements are encoded by M .
[Equation]
These elements are weighted by their nearest neighbor scores and summed.
This is then concatenated to the representation of x i and used by M to form the prediction:
[Equation]
This is easily extended to using multiple modules simultaneously.
For instance, two sources of information, E 1 and E 2 , can be combined by identifying the top candidates of each information source.
The weighted sum of the KIF output on each information source is concatenated with x i .
Finally, different sources of information may not be required for every prediction and some information sources can be more important than others.
To allow the model to make more ﬁne-grained decisions about what information to use from what source, and how much of it, we add a gating mechanism using a sigmoid function around each weighted sum of KNN representations.
KIF1 i and KIF2 i denote the KIF module from Equation 1 applied to E 1 and E 2 respectively.
[Equation]
A PPLYING KIF TO D IALOGUE T ASKS
We describe how to apply our method to the task of generative dialogue, a challenging setting where models must autoregressively generate engaging and on-topic responses.
We investigate dialogue for two main reasons: ﬁrst, dialogue agents must be able to consult relevant information to maintain the topic of the conversation.
Second, retrieval-based agents have strong performance compared to generative ones, due to their ability to copy dialogue utterances from the training set.
Using KIF, we can incorporate the beneﬁts of retrieval architectures into generative, knowledge-based models.
KIF for Generative Dialogue In a dialogue setting, x i represents the text of the conversation i .
A conversation consists of multiple back-and-forth utterances (or turns).
For example, a conversation could consist of 4 turns: x i =
[ x i, 1 , x i, 2 , x i, 3 , x i, 4 ] where x i, 4 is the direct utterance the model should respond to, and the earlier utterances are the conversation context .
Standard generative dialog models use a Transformer neural network as M and want to produce an output o i that is an appropriate response to the conversation.
However, in many cases, the conversation history alone does not include all of the information required to produce an appropriate response.
To incorporate knowledge, models often concatenate a knowledge source E such as Wikipedia to x i , such that M ([ x i , e 1 , e 2 , . .
, e n ])
=
ˆ o i , and use attention modules to identify the most relevant knowledge.
However, this approach is computationally intensive when handling large quantities of information.
Further, attention mechanisms have been found to operate poorly over long sequences, as the mechanism is blurry and struggles to make ﬁne-grained decisions (Fan et al., 2018).
The same is true for hierarchical approaches, which lack scalability.
We augment Transformer sequence-to-sequence (seq2seq) networks with KIF.
We experiment on two dialogue tasks, Wizard of Wikipedia (Dinan et al., 2018) and Engaging Imagechat (Shuster et al., 2018).
We use faiss (Johnson et al., 2019) to perform KNN search efﬁciently and at scale.
Wizard of Wikipedia
The goal of the Wizard of Wikipedia dataset is to train knowledgeable agents that can chat in any domain.
The dataset contains 1,365 various topics discussed in 18, dialogues in the training set, totalling 166,787 training utterances.
The topic is included as the ﬁrst utterance of the conversation.
The dataset includes relevant Wikipedia sentences for each turn of the chat, identiﬁed by an information retrieval system.
Our model for Wizard of Wikipedia has access to two sources of external information, E 1 and E 2 :
• E 1 is Wikipedia Knowledge provided by the dataset as evidence to support knowledgeable chitchat.
The scale of this KNN search is to ﬁlter through an average of 34 sentences.
The KIF module uses dialogue features to fetch relevant knowledge to condition upon to generate the subsequent utterance.
• E 2 is Training Utterances .
To incorporate the beneﬁts of retrieval-based dialogue models to the generative setting, we use KIF to identify relevant utterances from the training set and take their responses as input.
If many conversations about dogs have already occurred, models should be able to take advantage of these human-written examples to improve their generations.
There are around 170 K dialogue utterances as inputs to KNN search.
This can be interpreted as incorporating the beneﬁts of retrieval models by identifying an utterance with similar structure as the text the model would like to generate.
Access to these two sources of knowledge can be seen as as learning a template and a topic separately.
Sample templates can be identiﬁed from the training utterances, and topic-speciﬁc information learned by accessing the Wikipedia knowledge.
To better identify relevant training utterances from the large quantity available, we break down x i into conversation sub-features for a more ﬁne-grained match in the KNN search step.
We concatenate the encoding of the most recent dialogue utterance (e.g. x i, last ) with the encoding of the dialogue context and the turn number.
These are known to be salient conversation features.
The most recent dialogue utterance is the direct turn the model is responding to, and the dialogue context may provide additional clues.
The turn number is important, as earlier turns are often more generic (e.g. hi, how are you doing today ) and later turns are more speciﬁc.
Engaging ImageChat
The goal of Engaging ImageChat is to create agents capable of chitchatting about images, selected from the YFFC100M dataset (Thomee et al., 2015).
The dataset contains 186,782 dialogues in the training set, each about a unique image, totalling 355,862 utterances.
Agents are assigned one of 215 personalities (e.g. sweet ) to increase engagingness.
We use a MultiModal neural network designed to handle both image input and text input.
Following Shuster et al.
(2018), the images are encoded using a ResNeXt (Xie et al., 2017).
To extract the ﬁnal image representation, we project the 2048 -dimensional output of the image encoder to 512 -dimensions using a deep multi-layer perceptron with ReLU activation units.
The conversation history, which includes the personality, is encoded with a Transformer encoder network.
The image and conversation are combined using the Multimodal Sum Combiner proposed in Shuster et al.
Our model for Engaging Imagechat has access to two sources of external information, E 1 and E 2 :
• E 1 is Chat on Similar Images .
While there are over 180 K different images used in this dataset, many of the images are similar.
For example, conversations associated with two pictures of dogs could be relevant to each other.
The model is able to fetch from around 160 K different images and returns 6 turns of related chat for each image. Fetching from E [Table]
[Table]
consists of identifying Related Image Chats, or conversations on similar topics (as similar images are likely to have similar conversations).
• E 2 is Training Utterances .
Similar to the motivation for the previous dataset, we allow the model to identify training utterances that could be useful for responding in the current conversation.
The scale of this fetching task is large: 350 K dialogue utterances.
This could be interpreted as identifying utterances with similar structure to what the model would like to generate, and is complementary to the topic-based Related Image Chats.
To identify relevant information from the training utterances, we use the same dialogue features in the KNN search step, with one modiﬁcation: we add the personality provided by the dataset.
As utterances from speakers with the same personality are likely to be more related, this feature improves the quality of the fetched information.
E XPERIMENTAL S ETUP 5. I MPLEMENTATION D ETAILS
We use parl.ai (Miller et al., 2017) to implement our models.
We use byte-pair encoding (Sennrich et al., 2015) to represent the text to better handle the rare word problem (Dinan et al., 2018;
Fan et al., 2017). Our generative Transformer models have 8 encoder layers and 8 decoder layers, with FFN size 2048, embedding dimension 512, and 4 attention heads.
We optimize using Adam (Kingma & Ba, 2014) and the inverse square root learning schedule (Vaswani et al., 2017) with 10k warmup updates.
The initial learning rate is 0.0001 and we optimize for model perplexity.
We use a dropout of 0.5 and set gradient clipping to 0.1.
We set k = 5 for all cases.
We pre-train the Transformer seq2seq model used for both datasets on 250M comments from Reddit.
The comments are parsed to maintain conversational threads, so the encoder network has been exposed to conversational context at training time.
The ResNeXt encoder is pretrained on 3.5 billion images (Mahajan et al., 2018).
For both datasets, we model a vocabulary size of 54944 based on the BPE-based vocabulary from the Reddit pretraining.
We tuned the learning rate and batchsize hyperparameters together.
The model size is not tuned, as it was pre-trained with this size and thus kept ﬁxed.
5. E VALUATION
Generation We generate with beam search, setting the beam size to 4 with 3-gram blocking.
Automatic Metrics Following Dinan et al.
(2018), we compute F1 , a metric of unigram overlap, between the generated utterance and the human-written utterance.
For generative models, utterances are generated using beam search.
For retrieval models, the next utterance is predicted by ranking the entire set of training utterances, and the highest scoring utterance is chosen.
In Wizard of Wikipedia, there are two test sets: one set of seen topics, or topics that have been seen at training time with new test-time dialogues.
The second set is unseen , or topics that have not been encountered at all during training time.
We evaluate on both of these subsets.
Human Evaluation We follow the setup and use the analysis questions proposed in the
Acute-Eval dialogue evaluation system (Li et al., 2019).
For reproducibility, we adopt this existing evaluation setting that has been applied to several dialogue datasets.
We collect 100 human-bot dialogues on a crowdsourcing platform for both datasets.
Then, we show pairs of dialogues side by side, and ask the following questions:
• Who would you prefer to talk to for a long conversation?
• If you had to say one of the speakers is interesting and one is boring, who would you say is more interesting?
• Which speaker sounds more human?
• Which speaker has more coherent responses in the conversation?
• If you had to say that one speaker is more knowledgeable and one is more ignorant, who is more knowledgeable?
(Wizard of Wikipedia only)
We measure the percentage of time one model was chosen over the other, taking the agreement between three evaluators.
To reduce variance, the dialogues that are paired in the evaluation were collected on the same topic for Wizard of Wikipedia and collected on the same image for Engaging
ImageChat.
Each topic and image used is unique and taken from the test set randomly.
5. B ASELINES
We compare Transformers augmented with KIF to the state of the art retrieval models published on each dataset, as well as two additional generative baselines that have access to knowledge:
• Transformer Memory Networks .
To contrast the ability of KIF to existing work, we compare our
models to published Transformer Memory Networks ( Dinan et al., 2018).
These models encode each piece of external information independently with a Transformer Encoder, and these are stored as memory slots.
To access information in the memory slots, a model performs dot-product attention between the memory slots and the dialogue context.
In Dinan et al.
the knowledge selection from Wikipedia was supervised with either a two-stage model where the ﬁrst model was trained to predict the right knowledge, or an end-to-end model with an auxiliary loss for knowledge prediction accuracy.
• Retrieve and Reﬁne .
We implement a hybrid model (Weston et al., 2018) that incorporates top retrieval candidates as additional input to Generative Transformer MemNets.
All of the generative baselines are initialized with the same pre-training on Reddit that we use for our models for fair comparison on modeling quality.
For existing published models, we re-train the open-sourced generative models with the same pre-training.
R ESULTS
We describe the results of incorporating KIF modules into Transformer networks.
We display an example conversation between a human and our model in Figure 4, and show the top scoring Wikipedia knowledge and Training Utterance fetched by KIF modules.
6. KIF IS E FFECTIVE FOR I NCORPORATING K NOWLEDGE
Automatic Evaluation.
Comparing KIF augmented Transformer networks to published baselines and Retrieve and Reﬁne, we ﬁnd improved results (see Table 1 and Table 2).
For Wizard of
Wikipedia, the improvement in F1 score is almost 10 points.
A major contributing factor is the construction of the dataset — as each dialogue turn is grounded in a speciﬁc knowledge sentence from Wikipedia, improving the ability to identify the relevant fact strongly improves performance.
Contrasting the results from the seen and unseen test sets, the improvement on unseen is worse — it is harder to fetch training utterances for unseen topics.
Imagechat has no explicit dependency on knowledge.
We see a 2 point improvement, indicating that KIF can be generally useful.
Human Evaluation.
Results are shown in Figure 2.
On both datasets, we ﬁnd there is large improvement over existing generative models (green).
Evaluators agree that KIF-augmented Transformers are generally more coherent and human-sounding.
Comparison to existing retrieval models (blue) is more nuanced.
Along the lines of existing work (Zhang et al., 2018; Dinan et al., 2018), we ﬁnd that retrieval-based models score well in human evaluations that ask how human or interesting a dialogue sounds, as they return human-written utterances from the training set.
A surprising result is that KIF-augmented Transformers are voted more human sounding than retrieval models on Wizard of Wikipedia.
This is because the dataset’s human utterances are long and factual due to the tendency of crowdworkers to copy Wikipedia.
Sometimes humans chatting with the retrieval bot would respond uh... that’s an interesting fact?
Otherwise, our model scores similarly to retrieval models, with most of the evaluations not having statistically signiﬁcant differences.
On Engaging ImageChat, while our model has signiﬁcantly improved over the generative baseline, it does not beat retrieval based methods in sounding more human or being more interesting.
The retrieval baseline directly copies perfectly human-written utterances from the training set, so it is a difﬁcult baseline to beat with a generative model.
6. S CALING KIF TO C HALLENGING R ETRIEVAL S ETTINGS
KIF modules can be used in more realistic and challenging settings for knowledge retrieval that test the scalability of the module.
In Figure 3(a), we compare the Generative Transformer MemNet Baseline with KIF-Augmented Transformers in three settings.
The ﬁrst is the standard Wikipedia sentences provided by the dataset (average 34 sentences).
Then, we extend to providing the full Wikipedia article (average 215 sentences) and ﬁnally to providing multiple Wikipedia articles (average 1035 sentences), identiﬁed using the conversation’s topic.
This increasing size of available knowledge could be realistic for settings where it is unclear what information is most relevant, if ﬁltering steps to preprocess the data remove potentially relevant information, or if information synthesis from multiple knowledge sources is necessary to produce a high quality generation.
As the Wikipedia knowledge becomes more difﬁcult to identify, performance decreases, but still outperforms the baseline that uses the dataset-provided set of 34 sentences.
Comparing the scaling capability of KIF to the standard Generative Transformer MemNet Baseline highlights the advantage of using KNN.
The attention-based mechanism used in Dinan et al.
(2018) struggles to identify salient information when given increasingly larger quantities of knowledge, unlike the KNN based information fetch.
We hypothesize the attention mechanism is challenged by the softmax over a larger quantity of inputs, as it can be difﬁcult to make sharp distinctions.
6. A BLATIONS
Multiple Knowledge Sources.
For both Wizard of Wikipedia and Engaging ImageChat, multiple knowledge sources are used — training set utterances to capture the capability of a retrieval-based model as well as knowledge from Wikipedia or related chats based on image features. The perfor- [Table] [Table]
using KIF.
Salient conversation features improve performance on both datasets.
In the 3 Sources case, source of external information is unrelated.
Performance as k varies. mance using fetching the training utterances alone with KIF is shown in Table 3. There is a decrease in performance if only the training utterances can be accessed as an external knowledge source. We further analyze the importance of accessing training utterances compared to other possible modeling techniques to improve Transformer models.
Table 3 contrasts KIF on Training Utterances with two methods of improving generative models: post-editing and using retrieved human utterances as input.
Both models are Transformer seq2seq models initialized with our Reddit pre-training where the target is the human written utterance.
In post-editing, the input is the predicted utterance from the best generative model.
In retrieved human utterances, the input is the utterance returned by the best retrieval model.
In both comparisons, we see our model is able to leverage the training set utterances returned by KIF to improve performance.
Multiple Modalities for Fetching.
On Engaging ImageChat, we identify conversations on similar images and utterances in similar dialogue contexts.
We examine the importance of being able to fetch based on the image features to access chats from related images, as the KIF module can be used for multiple modalities.
Table 3 shows results without image-based KIF, which loses 0.5 F compared to Table 2.
The large size of the training set allows separate KIF modules to be helpful.
KIF using images could be interpreted as identifying topic-based information, and KIF using conversations as identifying templates for the generated response.
Dialogue Features for KNN Performance.
The quality of the KNN search is critical to the performance of KIF modules.
As the external knowledge is kept ﬁxed, KIF must be able to align the dialogue context with the knowledge to identify relevant pieces of information.
In Table 4, we show that matching on more features can improve the quality of the retrieved information.
Using only the encoding of the immediate previous utterance can improve results on Wizard of Wikipedia by 7 F1 points, but this is further improved by also leveraging the encoding of context (+1.8 F1) and additionally using the dialogue turn number (+1 F1). These features are available in the datasets, and we leverage them to improve the relatedness of retrieved knowledge. [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] Effect of Gating. We analyze the effect of the gating mechanism used in KIF by evaluating the capability of the gate to identify and focus on salient information.
On Wizard of Wikipedia, we concatenate a third source of information: dialogue turns from a completely different corpus called
PersonaChat (Zhang et al., 2018).
This dataset looks quite different — short utterances without factual knowledge — and should be easy for the model to identify as distinct from Wizard of Wikipedia.
As shown in Figure 3(b), if KIF on PersonaChat is included without gating, it has a harmful effect as the model includes irrelevant information.
When equipped with gating, the model learns to use the gate to ignore some inputs, and can recover almost the full performance of the model without this irrelvant information source.
Size of k. Figure 3(c) shows the performance on Wizard of Wikipedia when varying the amount of knowledge from both sources.
Generally, being able to access multiple relevant pieces of information is helpful, but too much information can be harmful.
This is likely because the weighted sum operation becomes more blurry if too many sentences are summed.
C ONCLUSION
We present a KNN-based Information Fetching module that learns to identify relevant information from external knowledge sources by learning a mapping-based read operation.
KIF modules beneﬁt from the scalability and efﬁciency of K Nearest Neighbors search, enabling computation with large external memories.
We show in the context of two dialogue datasets that relevant knowledge can be identiﬁed and incorporated to create more engaging, high quality dialogue. A. A BLATIONS Concatenation of Fetched Knowledge.
The knowledge of multiple KIF modules is combined by concatenating to the input representation (e.g. the dialogue context).
We examine two other alternatives to incorporating knowledge from multiple sources, speciﬁcally addition and inner product.
Results shown in Table 5 display that concatenation outperforms both of these alternatives, and that inner product performs better than addition.
Adding Features Used in KIF to Baselines.
To improve KNN search of relevant information, the
KIF modules use additional features such as the dialogue turn to identify the most useful elements from the ﬁxed memory.
We investigate the importance of those features on the dialogue modeling itself by augmenting the baselines with these fetching features.
Results are shown in Table 6.
For Wizard of Wikipedia, we add an explicit representation of the turn number and the concatenate the most recent turn again (as the KIF modules use these features to fetch knowledge).
The addition of these features to the baseline model does not signiﬁcantly improve performance.
For ImageChat, we experiment with representing the personality within the dialogue history as the
KIF module uses the personality feature.
In our Transformer baseline model following Shuster et al.
(2018), the personality is represented as a separate feature.
The removal of the personality feature is harmful, but including the personality in the dialogue history as the KIF modules do is not as good compared to the separate personality representation learned by Shuster et al.
A. A DDITIONAL H UMAN E VALUATION
Additional human evaluation results are shown in Figure 6.
On the left, the same human evaluation on the Seen test set is repeated on the Unseen test set, showing similar trends.
On the right, we display an analysis of the variance of our human evaluation.
The study was repeated on three different days.
There is greater variance on the More Human and More Interesting questions, as perhaps different evaluators have different understanding of these aspects.
Further, comparison with the Retrieval baseline has less variance compared to Generative models.
It is possible that the Retrieval model is a bit easier to evaluate given the written text is always copied from a human written utterance and is usually devoid of mistakes.