We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is  trained to minimize perplexity , an automatic metric that we compare against human judgement of multi-turn conversation quality. To capture this judgement, we propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of good conversation. Interestingly, our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72\% on multi-turn evaluation) suggests that a human-level SSA of 86\% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79\% SSA, 23\% higher than the next highest scoring chatbot that we evaluated.
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72\% on multi-turn evaluation) suggests that a human-level SSA of 86\% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79\% SSA, 23\% higher in absolute SSA than the existing chatbots we evaluated.
A
simply
R
, an automatic metric that we compare against human judgement of multi-turn conversation quality. To capture this judgement, we
of the next token. We also
R
good conversation. Interestingly, our
a human-like multi-turn conversation. Our
R
than the next highest scoring chatbot that
in absolute SSA than the existing chatbots