B o B : BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data (2021)

Abstract: Maintaining a consistent persona is essential for dialogue agents. Although tremendous advancements have been brought, the limited-scale of annotated personalized dialogue datasets is still a barrier towards training robust and consistent persona-based dialogue models. This work shows how this challenge can be addressed by disentangling persona-based dialogue generation into two sub-tasks with a novel BERT-over-BERT (BoB) model. Specifically, the model consists of a BERT-based encoder and two BERT-based decoders, where one decoder is for response generation, and another is for consistency understanding. In particular, to learn the ability of consistency understanding from large-scale non-dialogue inference data, we train the second decoder in an unlikelihood manner. Under different limited data settings, both automatic and human evaluations demonstrate that the proposed model outperforms strong baselines in response quality and persona consistency.

## introduction various approaches have been explored to introduce explicit personas in dialogue models @xcite @xcite . the persona can be defined as a composite of elements of identity, such as profiles and background personal facts. in persona-based dialogues, the generated responses are conditioned not only on the dialogue context but also on some predefined personas, so the presenting personality could be more consistent. existing persona-based dialogue models heavily utilize a set of persona-related dialogue data @xcite , such as the personachat @xcite . this kind of crowd-sourced dataset covers rich persona features, namely "persona-dense". nevertheless, the scale of such crowd-sourced datasets is limited by the expensive costs: two annotators are asked to act the part of a given provided persona and chat naturally to get to know each other during the conversation. on the other hand, conversations in daily life are not always persona-related. according to twitter content analysis, less than 10% messages on twitter reveal personal anecdote or activities at home or work and even less for personally identifiable information @xcite . as a result, the large-scale data collected from social media would only contain a limited amount of persona-related dialogues, which is "persona-sparse". the limited-scale of crowd-sourced data and the persona-sparsity in large-scale data present one common challenge: a model trained on limited personalized data cannot sufficiently understand persona consistency. as shown in figure 1 , a 12-layer gpt2 @xcite finetuned on the personachat dataset still shows a lack of consistency. after rethinking the essence of persona-based dialogue generation, we can find that it requires the dialogue agent to own the capabilities to 1) understand the persona-response consistency and 2) generate a persona-related response given the dialogue context. obviously, an ideal dataset that satisfies both features are difficult to annotate. however, once we disentangle persona-based dialogue generation into two sub-tasks: consistency understanding and dialogue generation, it is easy to find abundant data resources for them. for consistency understanding, we may leverage large-scale nondialogue inference data, such as snli @xcite and mnli @xcite as the training data. as for dialogue generation, we already have various large-scale persona-sparse datasets. inspired by the aforementioned motivation, in this work, we explore to learn a consistent personabased dialogue model from limited personalized dialogues, with the assistance of large-scale nondialogue inference data. specifically, the proposed model consists of an encoder e, an auto-regressive decoder d 1 for response generation, and a bidirectional decoder d 2 for consistency understanding. given personas p and dialogue query q, the e and d 1 jointly work in an encoder-decoder manner to capture a typical query to response mapping f g (s|q, p ), and generate a coarse response representation r 1 . then r 1 and personas p are fed into the bidirectional decoder d 2 to map r 1 to final response representations r 2 : f u (r 2 |s, p ). since the consistency understanding part f u (r|s, p ) is independent of the dialogue query q, it can be learned on non-dialogue inference datasets. here an unlikelihood training objective @xcite is applied to make contradicted cases in the inference data less likely so that d 2 could acquire the ability of consistency understanding. we initialize all modules from bert @xcite and name the proposed model bertover-bert (bob). to verify the effectiveness of our model, we experiment on two limited data scenarios: 1) a persona-dense scenario @xcite with low-resource settings @xcite , and 2) a persona-sparse scenario @xcite . both automatic and human evaluations indicate that our model generalizes well under different settings and outperforms strong baselines on most metrics, especially on persona consistency. contributions in this work are three-fold: meanwhile, various pre-training methods have also been applied in this field. @xcite al. (2019) and golovanov et al. (2019) show that fine-tuning pre-trained gpt on the persona-dense dataset can improve the quality of generated responses. @xcite propose an attention-routing mechanism in a gpt-based model to control the flow of persona information. @xcite explore how to leverage bert model for dialogue generation. different large-scale pretrained chatbots @xcite ) also show their effectiveness on persona-based dialogues. disentangled representation the concept of "disentangling" can be defined as transformations that only change some properties of the underlying model while leaving all other properties invariant @xcite . the variational autoencoder (kingma and welling, 2013) could be regarded as a disentangled representation learning framework, and various methods are built within it @xcite . unlikelihood training likelihood tries to maximize the probability of target sequence, while unlikelihood corrects known biases by minimizing the probability of negative candidates @xcite . closely related to our work, @xcite first explored unlikelihood training in addressing dialogue logical contradictions. they get contradicted dialogues from personachat according to dnli @xcite , a personachatoriented dialogue inference dataset. then unlikelihood training is applied to reduce the probability of contradicted responses. different from @xcite , with carefully designed decoders, our model could learn from large-scale non-dialogue inference datasets, making it generalizable to different scenarios, such as persona-dense and personasparse datasets, as will be seen in our experiments. ## conclusions in this work, we propose a novel bert-based dialogue model to learn from limited personalized data by disentangling response generation and consistency understanding. unlikelihood training with non-dialogue inference data is introduced to en-hance the model's understanding ability. experiments on two publicly available datasets demonstrate that our model can be trained with limited personalized dialogue data while still obtain significant improvements over strong methods.