P oly - encoders : architectures and pre - training strategies for fast and accurate multi - sentence scoring
A bstract
The use of deep pre-trained transformers has led to remarkable progress in a number of applications (Devlin et al., 2019).
For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately.
The former often performs better, but is too slow for practical use.
In this work, we develop a new transformer architecture, the Poly-encoder , that learns global rather than token level self-attention features.
We perform a detailed comparison of all three approaches, including what pre-training and ﬁne-tuning strategies work best.
We show our models achieve state-of-the-art results on four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.
I ntroduction
Recently, substantial improvements to state-of-the-art benchmarks on a variety of language understanding tasks have been achieved through the use of deep pre-trained language models followed by ﬁne-tuning (Devlin et al., 2019).
In this work we explore improvements to this approach for the class of tasks that require multi-sentence scoring: given an input context, score a set of candidate labels, a setup common in retrieval and dialogue tasks, amongst others.
Performance in such tasks has to be measured via two axes: prediction quality and prediction speed, as scoring many candidates can be prohibitively slow.
The current state-of-the-art focuses on using BERT models for pre-training (Devlin et al., 2019), which employ large text corpora on general subjects: Wikipedia and the Toronto Books Corpus (Zhu et al., 2015).
Two classes of ﬁne-tuned architecture are typically built on top: Bi-encoders and
Cross-encoders.
Cross-encoders (Wolf et al., 2019; Vig & Ramea, 2019), which perform full (cross) self-attention over a given input and label candidate, tend to attain much higher accuracies than their counterparts, Bi-encoders (Mazar´e et al., 2018; Dinan et al., 2019), which perform self-attention over the input and candidate label separately and combine them at the end for a ﬁnal representation.
As the representations are separate, Bi-encoders are able to cache the encoded candidates, and reuse these representations for each input resulting in fast prediction times.
Cross-encoders must recompute the encoding for each input and label; as a result, they are prohibitively slow at test time.
In this work, we provide novel contributions that improve both the quality and speed axes over the current state-of-the-art.
We introduce the Poly-encoder, an architecture with an additional learnt attention mechanism that represents more global features from which to perform self-attention, resulting in performance gains over Bi-encoders and large speed gains over Cross-Encoders.
To pre-train our architectures, we show that choosing abundant data more similar to our downstream task also brings signiﬁcant gains over BERT pre-training.
This is true across all di ﬀ erent architecture choices and downstream tasks we try.
We conduct experiments comparing the new approaches, in addition to analysis of what works best for various setups of existing methods, on four existing datasets in the domains of dialogue and information retrieval (IR), with pre-training strategies based on Reddit (Mazar´e et al., 2018) compared to Wikipedia / Toronto Books (i.e., BERT).
We obtain a new state-of-the-art on all four datasets with our best architectures and pre-training strategies, as well as providing practical implementations for real-time use.
Our code and models will be released open-source.
R elated W ork
The task of scoring candidate labels given an input context is a classical problem in machine learning.
While multi-class classiﬁcation is a special case, the more general task involves candidates as structured objects rather than discrete classes; in this work we consider the inputs and the candidate labels to be sequences of text.
There is a broad class of models that map the input and a candidate label separately into a common feature space wherein typically a dot product, cosine or (parameterized) non-linearity is used to measure their similarity.
We refer to these models as Bi-encoders .
Such methods include vector space models (Salton et al., 1975), LSI (Deerwester et al., 1990), supervised embeddings (Bai et al., 2009; Wu et al., 2018) and classical siamese networks (Bromley et al., 1994).
For the next utterance prediction tasks we consider in this work, several Bi-encoder neural approaches have been considered, in particular Memory Networks (Zhang et al., 2018a) and Transformer Memory networks (Dinan et al., 2019) as well as LSTMs (Lowe et al., 2015) and CNNs (Kadlec et al., 2015) which encode input and candidate label separately.
A major advantage of Bi-encoder methods is their ability to cache the representations of a large, ﬁxed candidate set.
Since the candidate encodings are independent of the input, Bi-encoders are very e ﬃ cient during evaluation.
Researchers have also studied a more rich class of models we refer to as Cross-encoders , which make no assumptions on the similarity scoring function between input and candidate label.
Instead, the concatenation of the input and a candidate serve as a new input to a nonlinear function that scores their match based on any dependencies it wants.
This has been explored with Sequential Matching
Network CNN-based architectures (Wu et al., 2017), Deep Matching Networks (Yang et al., 2018),
Gated Self-Attention (Zhang et al., 2018b), and most recently transformers (Wolf et al., 2019; Vig &
Ramea, 2019; Urbanek et al., 2019).
For the latter, concatenating the two sequences of text results in applying self-attention at every layer.
This yields rich interactions between the input context and the candidate, as every word in the candidate label can attend to every word in the input context, and vice-versa.
Urbanek et al.
(2019) employed pre-trained BERT models, and ﬁne-tuned both
Bi- and Cross-encoders, explicitly comparing them on dialogue and action tasks, and ﬁnding that
Cross-encoders perform better.
However, the performance gains come at a steep computational cost.
Cross-encoder representations are much slower to compute, rendering some applications infeasible.
T asks
We consider the tasks of sentence selection in dialogue and article search in IR.
The former is a task extensively studied and recently featured in two competitions: the Neurips ConvAI2 competition (Dinan et al., 2020), and the DSTC7 challenge, Track 1 (Yoshino et al., 2019; Jonathan K. Kummerfeld & Lasecki, 2018; Chulaka Gunasekara & Lasecki, 2019).
We compare on those two tasks and in addition, we also test on the popular Ubuntu V2 corpus (Lowe et al., 2015).
For IR, we use the Wikipedia Article Search task of Wu et al.
The ConvAI2 task is based on the Persona-Chat dataset (Zhang et al., 2018a) which involves dialogues between pairs of speakers.
Each speaker is given a persona, which is a few sentences that describe a character they will imitate, e.g. “I love romantic movies”, and is instructed to get to know the other.
Models should then condition their chosen response on the dialogue history and the lines of persona.
As an automatic metric in the competition, for each response, the model has to pick the correct annotated utterance from a set of 20 choices, where the remaining 19 were other randomly chosen utterances from the evaluation set.
Note that in a ﬁnal system however, one would retrieve from the entire training set of over 100k utterances, but this is avoided for speed reasons in common evaluation setups.
The best performing competitor out of 23 entrants in this task achieved 80.7% accuracy on the test set utilizing a pre-trained Transformer ﬁne-tuned for this task (Wolf et al., 2019).
The DSTC7 challenge (Track 1) consists of conversations extracted from Ubuntu chat logs, where one partner receives technical support for various Ubuntu-related problems from the other.
The best performing competitor (with 20 entrants in Track 1) in this task achieved 64.5% [-R@
-]R@1 (Chen & Wang, 2019).
Ubuntu V2 is a similar but larger popular corpus, created before the competition (Lowe et al., 2015); we report results for this dataset as well, as there are many existing results on it.
Finally, we evaluate on Wikipedia Article Search (Wu et al., 2018).
Using the 2016-12-21 dump of English Wikipedia ( ∼ 5M articles), the task is given a sentence from an article as a search query, ﬁnd the article it came from.
Evaluation ranks the true article (minus the sentence) against 10, other articles using retrieval metrics.
This mimics a web search like scenario where one would like to search for the most relevant articles (web documents).
The best reported method is the learningto-rank embedding model, StarSpace, which outperforms fastText, SVMs, and other baselines.
We summarize all four datasets and their statistics in Table 1.
[Table]
M ethods
In this section we describe the various models and methods that we explored.
4.1 T ransformers and P re - training S trategies
Transformers Our Bi-, Cross-, and Poly-encoders, described in sections 4.2, 4.3 and 4.4 respectively, are based on large pre-trained transformer models with the same architecture and dimension as BERT-base (Devlin et al., 2019), which has 12 layers, 12 attention heads, and a hidden size of 768.
As well as considering the BERT pre-trained weights, we also explore our own pre-training schemes.
Speciﬁcally, we pre-train two more transformers from scratch using the exact same architecture as BERT-base.
One uses a similar training setup as in BERT-base, training on 150 million of examples of [INPUT, LABEL] extracted from Wikipedia and the Toronto Books Corpus, while the other is trained on 174 million examples of [INPUT, LABEL] extracted from the online platform
Reddit (Mazar´e et al., 2018), which is a dataset more adapted to dialogue.
The former is performed to verify that reproducing a BERT-like setting gives us the same results as reported previously, while the latter tests whether pre-training on data more similar to the downstream tasks of interest helps.
For training both new setups we used XLM (Lample & Conneau, 2019).
Input Representation Our pre-training input is the concatenation of input and label [INPUT,LABEL], where both are surrounded with the special token [S], following Lample & Conneau (2019).
When pre-training on Reddit, the input is the context, and the label is the next utterance.
When pre-training on Wikipedia and Toronto Books, as in Devlin et al.
(2019), the input is one sentence and the label the next sentence in the text.
Each input token is represented as the sum of three embeddings: the token embedding, the position (in the sequence) embedding and the segment embedding.
Segments for input tokens are 0, and for label tokens are 1.
Pre-training Procedure Our pre-training strategy involves training with a masked language model (MLM) task identical to the one in Devlin et al.
In the pre-training on Wikipedia and
Toronto Books we add a next-sentence prediction task identical to BERT training.
In the pre-training on Reddit, we add a next-utterance prediction task, which is slightly di ﬀ erent from the previous one as an utterance can be composed of several sentences.
During training 50% of the time the candidate is the actual next sentence / utterance and 50% of the time it is a sentence / utterance randomly taken from the dataset.
We alternate between batches of the MLM task and the next-sentence / nextutterance prediction task.
Like in Lample & Conneau (2019) we use the Adam optimizer with learning rate of 2e-4, β 1 = 0 .
9, β 2 = 0 .
98, no L2 weight decay, linear learning rate warmup, and inverse square root decay of the learning rate.
We use a dropout probability of 0.1 on all layers, and a batch of 32000 tokens composed of concatenations [INPUT, LABEL] with similar lengths.
We train the model on 32 GPUs for 14 days.
Fine-tuning After pre-training, one can then ﬁne-tune for the multi-sentence selection task of choice, in our case one of the four tasks from Section 3.
We consider three architectures with which we ﬁne-tune the transformer: the Bi-encoder, Cross-encoder and newly proposed Poly-encoder.
4.2 B i - encoder
In a Bi-encoder, both the input context and the candidate label are encoded into vectors:
[Equation]
where T 1 and T 2 are two transformers that have been pre-trained following the procedure described in 4.1; they initially start with the same weights, but are allowed to update separately during ﬁnetuning.
T ( x ) = h 1 , .., h N is the output of a transformer T and red ( · ) is a function that reduces that sequence of vectors into one vector.
As the input and the label are encoded separately, segment tokens are 0 for both.
To resemble what is done during our pre-training, both the input and label are surrounded by the special token [S] and therefore h 1 corresponds to [S].
We considered three ways of reducing the output into one representation via red ( · ): choose the ﬁrst output of the transformer (corresponding to the special token [S]), compute the average over all outputs or the average over the ﬁrst m ≤ N outputs.
We compare them in Table 7 in the Appendix.
We use the ﬁrst output of the transformer in our experiments as it gives slightly better results.
Scoring The score of a candidate cand i is given by the dot-product s ( ctxt , cand i ) =
y ctxt · y cand i .
The network is trained to minimize a cross-entropy loss in which the logits are y ctxt · y cand 1 , ..., y ctxt · y cand n , where cand 1 is the correct label and the others are chosen from the training set.
Similar to Mazar´e et al.
(2018), during training we consider the other labels in the batch as negatives.
This allows for much faster training, as we can reuse the embeddings computed for each candidate, and also use a larger batch size; e.g., in our experiments on ConvAI2, we were able to use batches of 512 elements.
Inference speed In the setting of retrieval over known candidates, a Bi-encoder allows for the precomputation of the embeddings of all possible candidates of the system.
After the context embedding y ctxt is computed, the only operation remaining is a dot product between y ctxt and every candidate embedding, which can scale to millions of candidates on a modern GPU, and potentially billions using nearest-neighbor libraries such as FAISS (Johnson et al., 2019).
4.3 C ross - encoder
The Cross-encoder allows for rich interactions between the input context and candidate label, as they are jointly encoded to obtain a ﬁnal representation.
Similar to the procedure in pre-training, the context and candidate are surrounded by the special token [S] and concatenated into a single vector, which is encoded using one transformer.
We consider the ﬁrst output of the transformer as the context-candidate embedding:
[Equation]
where first is the function that takes the ﬁrst vector of the sequence of vectors produced by the transformer.
By using a single transformer, the Cross-encoder is able to perform self-attention between the context and candidate, resulting in a richer extraction mechanism than the Bi-encoder.
As the candidate label can attend to the input context during the layers of the transformer, the Crossencoder can produce a candidate-sensitive input representation, which the Bi-encoder cannot.
For example, this allows it to select useful input features per candidate.
Scoring To score one candidate, a linear layer W is applied to the embedding y ctxt , cand to reduce it from a vector to a scalar:
[Equation]
Similarly to what is done for the Bi-encoder, the network is trained to minimize a cross entropy loss where the logits are s ( ctxt , cand 1 ) , ..., s ( ctxt , cand n ), where cand 1 is the correct candidate and the others are negatives taken from the training set.
Unlike in the Bi-encoder, we cannot recycle the other labels of the batch as negatives, so we use external negatives provided in the training set.
The
Cross-encoder uses much more memory than the Bi-encoder, resulting in a much smaller batch size.
Inference speed Unfortunately, the Cross-encoder does not allow for precomputation of the candidate embeddings.
At inference time, every candidate must be concatenated with the input context and must go through a forward pass of the entire model.
Thus, this method cannot scale to a large amount of candidates.
We discuss this bottleneck further in Section 5.4.
4.4 P oly - encoder
The Poly-encoder architecture aims to get the best of both worlds from the Bi- and Cross-encoder.
A given candidate label is represented by one vector as in the Bi-encoder, which allows for caching candidates for fast inference time, while the input context is jointly encoded with the candidate, as in the Cross-encoder, allowing the extraction of more information.
The Poly-encoder uses two separate transformers for the context and label like a Bi-encoder, and the candidate is encoded into a single vector y cand i .
As such, the Poly-encoder method can be implemented using a precomputed cache of encoded responses.
However, the input context, which is typically much longer than a candidate, is represented with m vectors ( y 1 ctxt ..
y mctxt ) instead of just one as in the Bi-encoder, where m will inﬂuence the inference speed.
To obtain these m global features that represent the input, we learn m context codes ( c 1 , ..., c m ), where c i extracts representation y ictxt by attending over all the outputs of the previous layer.
That is, we obtain y ictxt using:
[Equation]
where
[Equation]
The m context codes are randomly initialized, and learnt during ﬁnetuning.
Finally, given our m global context features, we attend over them using y cand i as the query: y ctxt = (cid:88)
i w i
y ictxt where ( w 1 , .., w m ) =
softmax( y cand i · y 1 ctxt , .., y cand i · y mctxt )
The ﬁnal score for that candidate label is then y ctxt · y cand i as in a Bi-encoder.
As m < N , where N is the number of tokens, and the context-candidate attention is only performed at the top layer, this is far faster than the Cross-encoder’s full self-attention.
E xperiments
We perform a variety of experiments to test our model architectures and training strategies over four tasks.
For metrics, we measure Recall@ k where each test example has C possible candidates to select from, abbreviated to R@ k / C , as well as mean reciprocal rank (MRR).
5.1 B i - encoders and C ross - encoders
We ﬁrst investigate ﬁne-tuning the Bi- and Cross-encoder architectures initialized with the weights provided by Devlin et al.
(2019), studying the choice of other hyperparameters (we explore our own pre-training schemes in section 5.3).
In the case of the Bi-encoder, we can use a large number of negatives by considering the other batch elements as negative training samples, avoiding recomputation of their embeddings.
On 8 Nvidia Volta v100 GPUs and using half-precision operations (i.e. ﬂoat operations), we can reach batches of 512 elements on ConvAI2.
Table 2 shows that in this setting, we obtain higher performance with a larger batch size, i.e. more negatives, where 511 negatives yields the best results.
For the other tasks, we keep the batch size at 256, as the longer sequences in those datasets uses more memory.
The Cross-encoder is more computationally intensive, as the embeddings for the (context, candidate) pair must be recomputed each time.
We thus limit its batch size to 16 and provide negatives random samples from the training set.
For DSTC7 and Ubuntu V2, we choose 15 such negatives; For ConvAI2, the dataset provides 19 negatives.
[Table]
The above results are reported with Bi-encoder aggregation based on the ﬁrst output.
Choosing the average over all outputs instead is very similar but slightly worse (83.1, averaged over 5 runs).
We also tried to add further non-linearities instead of the inner product of the two representations, but could not obtain improved results over the simpler architecture (results not shown).
We tried two optimizers: Adam (Kingma & Ba, 2015) with weight decay of 0.01 (as recommended by (Devlin et al., 2019)) and Adamax (Kingma & Ba, 2015) without weight decay; based on validation set performance, we choose to ﬁne-tune with Adam when using the BERT weights.
The learning rate is initialized to 5e-5 with a warmup of 100 iterations for Bi- and Poly-encoders, and 1000 iterations for the Cross-encoder.
The learning rate decays by a factor of 0.4 upon plateau of the loss evaluated on the valid set every half epoch.
In Table 3 we show validation performance when ﬁne-tuning various layers of the weights provided by (Devlin et al., 2019), using Adam with decay optimizer.
Fine-tuning the entire network is important, with the exception of the word embeddings.
With the setups described above, we ﬁne-tune the Bi- and Cross-encoders on the datasets, and report the results in Table 4.
On the ﬁrst three tasks, our Bi-encoders and Cross-encoders outperform the best existing approaches in the literature when we ﬁne-tune from BERT weights.
E.g., the Biencoder reaches 81.7% R@1 on ConvAI2 and 66.8% R@1 on DSTC7, while the Cross-encoder achieves higher scores of 84.8% R@1 on ConvAI2 and 67.4% R@1 on DSTC7.
Overall, Crossencoders outperform all previous approaches on the three dialogue tasks, including our Bi-encoders (as expected).
We do not report ﬁne-tuning of BERT for Wikipedia IR as we cannot guarantee the [Table] test set is not part of
[Table]
5.2 P oly - encoders
We train the Poly-encoder using the same batch sizes and optimizer choices as in the Bi-encoder experiments.
Results are reported in Table 4 for various values of m context vectors.
The Poly-encoder outperforms the Bi-encoder on all the tasks, with more codes generally yielding larger improvements.
Our recommendation is thus to use as large a code size as compute time allows (see Sec. 5.4).
On DSTC7, the Poly-encoder architecture with BERT pretraining reaches 68.9% R with 360 intermediate context codes; this actually outperforms the Cross-encoder result (67.4%) and is noticeably better than our Bi-encoder result (66.8%).
Similar conclusions are found on Ubuntu
V2 and ConvAI2, although in the latter Cross-encoders give slightly better results.
We note that since reporting our results, the authors of Li et al.
(2019) have conducted a human evaluation study on ConvAI2, in which our Poly-encoder architecture outperformed all other models compared against, both generative and retrieval based, including the winners of the competition. [Table]
5.3 D omain - specific P re - training
We ﬁne-tune our Reddit-pre-trained transformer on all four tasks; we additionally ﬁne-tune a transformer that was pre-trained on the same datasets as BERT, speciﬁcally Toronto Books + Wikipedia.
When using our pre-trained weights, we use the Adamax optimizer and optimize all the layers of the transformer including the embeddings.
As we do not use weight decay, the weights of the ﬁnal layer are much larger than those in the ﬁnal layer of BERT; to avoid saturation of the attention layer in the
Poly-encoder, we re-scaled the last linear layer so that the standard deviation of its output matched that of BERT, which we found necessary to achieve good results.
We report results of ﬁne-tuning with our pre-trained weights in Table 4.
We show that pre-training on Reddit gives further state-ofthe-art performance over our previous results with BERT, a ﬁnding that we see for all three dialogue tasks, and all three architectures.
The results obtained with ﬁne-tuning on our own transformers pre-trained on Toronto Books +
Wikipedia are very similar to those obtained with the original BERT weights, indicating that the choice of dataset used to pre-train the models impacts the ﬁnal results, not some other detail in our training.
Indeed, as the two settings pre-train with datasets of similar size, we can conclude that choosing a pre-training task (e.g. dialogue data) that is similar to the downstream tasks of interest (e.g. dialogue) is a likely explanation for these performance gains, in line with previous results showing multi-tasking with similar tasks is more useful than with dissimilar ones (Caruana, 1997).
5.4 I nference S peed
An important motivation for the Poly-encoder architecture is to achieve better results than the Biencoder while also performing at a reasonable speed.
Though the Cross-encoder generally yields strong results, it is prohibitively slow.
We perform speed experiments to determine the trade-o ﬀ of improved performance from the Poly-encoder.
Speciﬁcally, we predict the next utterance fordialogue examples in the ConvAI2 validation set, where the model scores C candidates (in this case, chosen from the training set).
We perform these experiments on both CPU-only and GPU setups.
CPU computations were run on an 80 core Intel Xeon processor CPU E5-2698.
GPU computations were run on a single Nvidia Quadro GP100 using cuda 10.0 and cudnn 7.4.
We show the average time per example for each architecture in Table 5.
The di ﬀ erence in timing between the Bi-encoder and the Poly-encoder architectures is rather minimal when there are only 1000 candidates for the model to consider.
The di ﬀ erence is more pronounced when considering 100k candidates, a more realistic setup, as we see a 5-6x slowdown for the Poly-encoder variants.
Nevertheless, both models are still tractable.
The Cross-encoder, however, is 2 orders of magnitude slower than the Bi-encoder and Poly-encoder, rendering it intractable for real-time inference, e.g. when interacting with a dialogue agent, or retrieving from a large set of documents.
Thus, Polyencoders, given their desirable performance and speed trade-o ﬀ , are the preferred method.
We additionally report training times in the Appendix, Table 6.
Poly-encoders also have the beneﬁt of being 3-4x faster to train than Cross-encoders (and are similar in training time to Bi-encoders).
C onclusion
In this paper we present new architectures and pre-training strategies for deep bidirectional transformers in candidate selection tasks.
We introduced the Poly-encoder method, which provides a mechanism for attending over the context using the label candidate, while maintaining the ability to precompute each candidate’s representation, which allows for fast real-time inference in a production setup, giving an improved trade o ﬀ between accuracy and speed.
We provided an experimental analysis of those trade-o ﬀ s for Bi-, Poly- and Cross-encoders, showing that Poly-encoders are more accurate than Bi-encoders, while being far faster than Cross-encoders, which are impractical for real-time use.
In terms of training these architectures, we showed that pre-training strategies more closely related to the downstream task bring strong improvements.
In particular, pre-training from scratch on Reddit allows us to outperform the results we obtain with BERT, a result that holds for all three model architectures and all three dialogue datasets we tried.
However, the methods introduced in this work are not speciﬁc to dialogue, and can be used for any task where one is scoring a set of candidates, which we showed for an information retrieval task as well.
A T raining T ime
We report the training time on 8 GPU Volta 100 for the 3 datasets considered and for 4 types of models in Table 6.
[Table]
B R eduction layer in B i - encoder
We provide in Table 7 the results obtained for di ﬀ erent types of reductions on top of the Bi-encoder.
Speciﬁcally we compare the Recall@1 / 20 on the ConvAI2 validation set when taking the ﬁrst output of BERT, the average of the ﬁrst 16 outputs, the average of the ﬁrst 64 outputs and all of them except the ﬁrst one ([S]).
[Table]
C A lternative C hoices for C ontext V ectors
We considered a few other ways to derive the context vectors ( y 1 ctxt , ..., y mctxt ) of the Poly-encoder from the output ( h 1 ctxt , ..., h Nctxt ) of the underlying transformer:
• Learn m codes ( c 1 , ..., c m ), where c i extracts representation y ictxt by attending over all the outputs ( h 1 ctxt , ..., h Nctxt ).
This method is denoted “Poly-encoder (Learnt-codes)” or “Polyencoder (Learnt-m)”, and is the method described in section 4.
• Consider the ﬁrst m outputs ( h 1 ctxt , ..., h mctxt ).
This method is denoted “Poly-encoder (First m outputs)” or “Poly-encoder (First-m)”.
Note that when N < m , only m vectors are considered.
• Consider the last m outputs.
• Consider the last m outputs concatenated with the ﬁrst one, h 1 ctxt which plays a particular role in BERT as it corresponds to the special token [S].
The performance of those four methods is evaluated on the validation set of Convai2 and DSTC and reported on Table 8.
The ﬁrst two methods are shown in Figure 2.
We additionally provide the inference time for a given number of candidates coming from the Convai2 dataset on Table 9.
[Table]
1 Attention Code
[Table]
4 Attention Codes
[Table]
16 Attention
[Table] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]