T OWARDS B OOSTING THE O PEN -D OMAIN C HATBOT WITH H UMAN F EEDBACK
Anonymous authors
Paper under double-blind review
A BSTRACT
Many open-domain dialogue models pre-trained with social media comments can generate coherent replies but have difficulties producing engaging responses.
This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference.
In this paper, we propose a novel and efficient framework Diamante to boost the open-domain chatbot, where two kinds of human feedback (including explicit demonstration and implicit preference) are collected and leveraged.
By asking annotators to select or amend the model-generated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs a Chinese chit-chat dataset.
To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the generation-evaluation joint training.
Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of pre-trained dialogue models.
The overall engagingness of the previous state-of-the-art model has been improved remarkably by 50% in Chinese open-domain conversations.
I NTRODUCTION
In recent years, the self-supervised pre-training based on tremendous unlabeled data has brought great success for many natural language processing tasks (Brown et al., 2020; Chowdhery et al., 2022).
In dialogue generation, the pre-training is usually carried out with massive social media comments, acting as human-like conversations (Adiwardana et al., 2020; Bao et al., 2021; Thoppilan et al., 2022).
Despite that these pre-trained dialogue models are capable of generating coherent replies, they have difficulties producing engaging responses.
The main reasons for this phenomenon might be twofold.
Firstly, there exists a considerable gap in the data distribution between the proxy human-like conversations (public group discussion) and the real human-human conversations (private two-way messaging).
Secondly, the dialogue model usually outputs the response with the highest generation probability, which could reflect the probability mass over all the training data but might not align well with human preference (e.g., some biased or unsafe statements).
One straightforward way to narrow the data distribution gap is to fine-tune the pre-trained dialogue model with annotated human-human conversations.
For instance, Blender (Roller et al., 2021) employs four annotated datasets (Zhang et al., 2018;
Dinan et al., 2019; Rashkin et al., 2019; Smith et al., 2020) to emphasize the conversational skills of personality, knowledge, empathy, and engagingness.
As for the alignment with human preference, LaMDA (Thoppilan et al., 2022) defines and quantifies some critical metrics for dialogue evaluation, including safety, interestingness, and so on.
By filtering out those candidate responses with poor performance on these metrics, the human preference towards the dialogue model has increased significantly.
However, compared with English, the annotations of high-quality human-human conversations or dialogue evaluation samples are relatively scarce in other languages.
As a result, even the state-of-the-art Chinese chatbot – PLATO-XL (Bao et al., 2021), is only pre-trained with social media comments and not involved with advanced response evaluation.
In this paper, we propose a novel and efficient framework, namely Diamante, consisting of a data collection strategy and a learning method to boost the performance of pre-trained dialogue models.
Two kinds of human feedback are collected and leveraged in Diamante, including explicit demonstration and implicit preference.
Firstly, to bridge the gap in data distribution, Diamante collects
Task Description
Please read the guidelines before the conversation.
Save
Restart
Candidates
Sometimes my cat sometimes even eats his own hairballs.
I’m pretty worried about his digestion.
Dialogue
I have no choice but to buy a vacuum cleaner.
British shorthair.
Yes, I have to vacuum every day.
Does your cat shed a lot as well?
What breed is it?
My cat started shedding everywhere in the spring.
How to deal with it?
You can give him some hairball remedies.
It’s not a big problem.
Submit § My cat eats hairballs, too.
Oh no, how old is he?
§ Is he deficient in something?
Buy him some cat grass?
§ Feed him some hairball remedies.
§ If he eats hairballs, you can give him some hairball remedies.
§ It is okay.
My cat eats hairballs, too.
§ You need to pay attention to it.
The cat may be deficient in some trace elements.
Go to see a veterinarian.
Note:
Click one candidate and it will be shown in the input box.
Collect a conversation with the assistance of this model.
First, craft a dialogue opening based on your interest.
Then select, revise or rewrite the candidate to reply properly.
an open-domain chit-chat dataset in Chinese with the assistance of PLATO-XL.
Based on modelgenerated candidate responses, human annotators can efficiently produce an engaging response to continue the conversation.
Secondly, we propose to leverage the implicit human preference that appeared in the data collection process, i.e., the annotator’s selected or amended response is preferred over the other candidates.
To this end, Diamante introduces a novel generation-evaluation joint training paradigm, where high-quality response generation and human preference estimation are learned simultaneously.
During inference, the candidate response with the highest preference score would be selected as the final response and returned to the user.
Extensive and intensive experiments have been carried out to evaluate the effectiveness of the Diamante framework, including the collected dataset and joint training paradigm.
Experimental results reveal that Diamante significantly boosts PLATO-XL’s performance and establishes a new state-of-the-art result in Chinese open-domain conversation.
It is notable that compared to the human reference, Diamante even achieves competitive or slightly better performance.
In addition to
PLATO-XL, Diamante brings remarkable improvements to other pre-trained dialogue models.
The
Diamante dataset is now publicly available, which can be accessed and downloaded under the license agreement at the data platform 1 .
We have also released all source code 2 , hoping to facilitate future research in dialogue generation.
D IAMANTE D ATASET
In this paper, we collect an open-domain chit-chat dataset in Chinese with the assistance of a pretrained dialogue model.
In the following, we will describe the creation of the Diamante dataset.
2. D ATA C OLLECTION
Diamante aims to explore an efficient way to collect a batch of high-quality chit-chat conversations that align well with human values.
The data annotation interface is shown in Figure 1 (the original interface is in Chinese and displayed in Figure 6 of the Appendix).
The data collection process is carried out as follows.
Step 1:
Crafting the Dialogue Opening.
Firstly, the annotator is encouraged to craft a start utterance based on any topic of interest, as an informative and engaging dialogue opening is critical to a good conversation.
As shown in Figure 1, the start utterance is “ My cat started shedding everywhere in the spring.
How to deal with it? ”.
We also provide various topics and examples in the guidelines to inspire annotators to write dialogue openings.
[Table]
Step 2:
Generating Candidate Responses with the Dialogue Model.
Given the dialogue context, a dialogue model (PLATO-XL in the Diamante dataset) is employed to generate multiple candidate responses.
To ensure the diversity of response content and conversation flow, we adopt the top- k sampling as the decoding strategy and select seven candidates for the demonstration to the annotator.
Step 3: Producing Response with Human Feedback.
We then ask the annotator to select, revise or rewrite the candidate to produce an appropriate response.
- Select.
As large-scale dialogue models can generate coherent and occasionally interesting responses, the annotator is allowed to select one response directly from the candidates where appropriate.
- Revise.
Given the possible defects in the candidate responses, such as a lack of consistency or attractiveness, the annotator can choose the preferred candidate and further revise it for better quality.
- Rewrite.
If no appropriate candidate exists, the annotator needs to write a suitable and engaging response by themselves.
Iterating Step 2 & Step 3 to Continue the Dialogue.
After collecting the response with human feedback, the conversation will continue by iterating step 2 and step 3.
The dialogue collection with the human-model in the loop will continue for at least seven rounds.
To ensure the annotation quality of the Diamante dataset, we also designed and followed a rigorous quality control process, with details discussed in the Appendix.
The above data collection strategy works well in terms of efficiency and quality.
The annotator can produce the final response efficiently by directly selecting or amending the model-generated candidates.
The conversation quality is guaranteed or enhanced with the human annotator’s verification or embellishment.
Moreover, the implicit human preference that appeared in the data collection process also allows the training of one preference estimation model without additional annotation.
2. D ATA A NALYSIS
Corpus Statistics.
In total, 147 annotators participated in the dataset collection.
The detailed statistics of the Diamante dataset are summarized in Table 1.
The dataset consists of 6,838 dialogues with 98,115 utterances, and the average utterance length is about 14.25.
We split the collected data into train, validation, and test sets.
As for the annotator operation proportions, 18% of the utterances are produced from Select , 41% from Revise , and 41% from Rewrite .
Dialogue Topics.
The Diamante dataset is about open-domain chit-chat and is not limited to any topic.
For further quantitative analysis, we employ the topic tagger on the Baidu AI platform 3 to categorize the dialogues.
(The topic visualization of the Diamante dataset is displayed in Figureof the Appendix.)
The results show that the Diamante dataset covers all 26 main categories.
The top five topics are Society (23%), Entertainment (11%), People (10%), Education (8%), and Food & Drink (8%), which are in line with our daily life.
G ENERATION -E VALUATION J OINT T RAINING
In this paper, we propose to leverage not only the explicit human demonstrations but also the implicit human preference that appeared in the data collection to boost the open-domain chatbot comprehensively.
A novel generation-evaluation joint training paradigm is introduced and illustrated in Figure
Transformer Blocks Transformer Blocks
Context ⋅ [SEP] ⋅ ⋅ ⋅
Human Response
Context ⋅ [SEP] ⋅ ⋅ ⋅
Model Response
Transformer Blocks
Context ⋅ [SEP] ⋅ ⋅ ⋅
Random Response
NLL
PE 𝑠(𝑐,𝑟 ℋ )
𝑠(𝑐,𝑟 ℳ ) 𝑠(𝑐,𝑟 ℛ )
2, where the high-quality response generation and human preference estimation are optimized simultaneously.
The classical training objective of dialogue generation is to minimize the negative log-likelihood (NLL) loss:
[Equation]
(1) where c refers to the dialogue context and r H is the human annotator’s selected or amended response.
Besides generation, Diamante encodes evaluation into the joint optimization to enhance the alignment with human preference.
Recall that in the data collection process, there exists implicit human preference: given the dialogue context c , the final response r H is preferred by human annotators as compared to a model-generated candidate r M ∈ R M (displayed during annotation).
Moreover, either r H or r M is better than a randomly selected response r R in most cases.
As such, we can have the following preference ranking r H > r M > r R .
The preference estimation (PE) loss is then defined as:
[Equation]
where the input is a quadruple of ( c, r H , r M , r R ) , σ ( · ) is the sigmoid function, and s ( · ) is the scalar output of the model.
The total objective of the generation-evaluation joint training is to minimize the following integrated loss:
[Equation]
The first term helps the model learn to mimic human demonstrations and generate high-quality candidate responses.
And the second term helps the model learn the nuanced distinctions among human preferences.
During inference, we adopt the top- k sampling to produce multiple candidate responses and then perform ranking with their corresponding preference estimation scores.
The one with the highest preference score would be selected as the final response and returned to the user.
Notably, the preference estimation follows the candidate response decoding and only involves one more token processing, which incurs negligible computational cost.
One similar work to Diamante’s joint training is LaMDA (Thoppilan et al., 2022), where a single model functions as both a generator and a discriminator.
In comparison, there exist several critical differences between Diamante and LaMDA.
Firstly, LaMDA chooses to learn the discriminator and generator sequentially.
By contrast, Diamante optimizes generation and evaluation simultaneously, trying to avoid the catastrophic forgetting issue of the two-stage training (Kirkpatrick et al., 2017; Liu et al., 2022b).
Secondly, LaMDA defines fine-grained dialogue evaluation metrics and collects corresponding discriminator training samples.
Considering the expensive cost of data collection and the difficulty of reaching an agreement in fine-grained dialogue evaluation (Smith et al., 2022), Diamante leverages the implicit human preference as the overall evaluation and gets rid of additional annotations.
Thirdly, as suggested in the works of human alignment (Askell et al., 2021), the ranked preference evaluation adopted in Diamante performs better than the binary discrimination used in LaMDA.
E XPERIMENTS 4. S ETTINGS 4.1. I MPLEMENTATION D ETAILS
We apply the Diamante dataset and joint training paradigm to boost PLATO-XL’s performance.
In the generation-evaluation joint training, the input samples are formulated as quadruples ( c, r H , r M , r R ) , where c is the dialogue context, r H is the human annotator’s selected or amended response, r M is one candidate response displayed during annotation, and r R is one randomly selected response from the dataset. During the construction of joint training samples, if the sampled model-generated candidate r M is found to be the same as the human-generated response r H , r M will be re-sampled to guarantee the agreement (preference ranking r H > r M ).
In addition, r M and r R are re-sampled at each training epoch.
The model is initialized with the 11B parameter PLATO-XL, with the transformer architecture of
PrefixLM (Radford et al., 2018; Dong et al., 2019).
(There are 72 transformer blocks and 32 attention heads, with the embedding dimension of 3072.
The hidden dimension of the feedforward layer is set to 18432.)
The preference estimation value s ( · ) is obtained through one fully-connected layer (converting the transformer output into one scalar).
The hyper-parameter settings used in the training process are listed as follows.
The maximum sequence length of context and response is set toand 128, respectively.
We use Adam (Kingma & Ba, 2015) as the optimizer, with a learning rate scheduler including a linear warmup and an invsqrt decay (Vaswani et al., 2017).
The peak learning rate is set to 2e-6, and the warmup step is set to 500.
The model is trained for five epochs with a batch size of 168.
The implementation is based on the PaddlePaddle framework, and the experiments are carried out on 8 Nvidia A100 GPUs (40G RAM).
During inference, we adopt the top- k sampling ( k set to 10) to produce 20 candidate responses and select one with the highest preference estimation score as the final response.
4.1. C OMPARED A PPROACHES
In the experiments, the following Chinese dialogue models are considered:
• CDial-GPT (Wang et al., 2020) is a 104M parameter model trained on LCCC conversations.
• EVA2.0 (Gu et al., 2022) is a 2.8B parameter model pre-trained on cleaned WDC-Dialogue .
• PLATO-XL (Bao et al., 2021) is the largest Chinese dialogue model with up to 11B parameters, pre-trained on social media conversations.
In addition to the above dialogue models, the following commercial chatbots in Chinese are included: Microsoft XiaoIce (Zhou et al., 2020), Xiao AI, Tmall Genie, and Apple Siri.
4.1. E VALUATION M ETRICS
In the experiments, we employ crowd-sourcing workers to evaluate the dialogue quality in four aspects: coherence, informativeness, safety, and engagingness.
We discuss these criteria below and provide scoring details in Appendix A.
• Coherence assesses whether the response is relevant and consistent with the context.
• Informativeness evaluates whether the response includes appropriate information.
• Safety evaluates whether the response contains harmful, biased, or misleading content.
Engagingness measures the willingness to have a long conversation with the partner.
The coherence, informativeness, and safety are the utterance-level metrics.
The engagingness is the dialogue-level metric.
These metrics are evaluated on a range of [0, 1, 2], with higher scores being better.
Each sample is distributed to three crowd-sourcing workers, and the final score is determined through majority voting.
4. E XPERIMENTAL R ESULTS
Considering the limitations of automatic dialogue evaluation (Liu et al., 2016), we employ crowdsourcing workers to evaluate the dialogue quality, including static evaluation, self-chat evaluation, and human-bot chat evaluation.
4.2. S TATIC E VALUATION
In the static evaluation, we randomly select 100 samples from the test set and employ the models to generate the response given the multi-turn dialogue context. In addition to PLATO-XL and Dia- [Table]
[Table] [Table] mante, we also provide the performance of ground truth for reference.
The evaluation results are summarized in Table 2.
Diamante significantly improves the response quality on all criteria compared to PLATO-XL.
Diamante even achieves competitive or slightly better performance compared to the human reference.
For a detailed analysis, we further reviewed the 14/100 cases where Diamante achieved a higher engagingness score than the human reference.
We found out that possible reasons for this phenomenon could be twofold.
Firstly, it is difficult for annotators to keep producing attractive and engaging responses at each round in multi-turn conversations, which is regular and consistent with our daily conversations.
Secondly, Diamante encodes the preference estimation in the joint training to enhance the alignment with human preference, which helps it select the human-preferred response among candidate responses.
4.2. S ELF -C HAT E VALUATION
As suggested by Adiwardana et al.
(2020), the static evaluation can be biased by the construction of dialogue context.
Therefore, we also include the interactive evaluation in the experiments, including the self-chat evaluation and human-bot chat evaluation.
Following the settings in PLATO-XL, 50 open-domain utterances are selected as dialogue openings, and models play the roles of both partners to continue the conversation for 5 rounds.
Then these conversations are distributed to crowd-sourcing workers for evaluation.
The self-chat evaluation results are summarized in Table 3.
Diamante outperforms the rest models in all evaluation aspects and establishes a new state-ofthe-art result in Chinese open-domain conversation.
In particular, Diamante achieves a remarkable 50% improvement on the metric of engagingness compared to PLATO-XL.
These results verify the effectiveness of the Diamante dataset and generation-evaluation joint training paradigm.
4.2. H UMAN -B OT C HAT E VALUATION
In addition to the above dialogue models, Diamante is compared to common commercial chatbots in
Chinese through human-bot chat evaluations.
We select 20 high-frequency topics from a deployed chatbot and ask in-house data specialists to interact with these chatbots for 7-14 rounds.
The humanbot chat evaluation results are summarized in Table 4.
Diamante consistently outperforms the rest [Table] [Table] of the commercial chatbots
by a large margin across all the human evaluation metrics.
These results indicate that Diamante can produce high-quality responses when interacting with real users.
The Fleiss’ kappa (Fleiss, 1971) score for the static evaluation, self-chat evaluation [Table] and human-bot chat evaluation is 0.433, 0.
[Table]
and 0.424, respectively. This suggests that crowd-sourcing workers have reached a moderate agreement in human evaluation
[Table]
4. D ISCUSSIONS 4.3. A BLATION S TUDY ON J OINT T RAINING
As discussed in previous sections, the improvements of Diamante compared to PLATO-XL come from two aspects: the Diamante dataset bridges the distribution gap towards human-human conversations
[Table]
and the joint training paradigm enhances the alignment with human preference.
For further dissection, we carry out ablation studies on joint training as follows.
Without joint training, PLATOXL is trained with the Diamante dataset to minimize the NLL loss
[Table]
and the final response is selected based on generation probability during inference.
With joint training, PLATO-XL is trained with the Diamante dataset to minimize the generation-evaluation integrated loss, and the final response is selected based on preference estimation during inference.
[Figure]
Firstly, we conduct automatic evaluations of response selection on the test set to compare these two approaches.
Each dialogue context has one human annotated response and seven model-generated candidates (displayed during annotation).
The experiments evaluate the ranking of the reference response among these candidates.
The results are reported in terms of mean average precision (MAP), mean reciprocal rank (MRR), and precision at position 1 (P@1), as summarized in Figure 3.
The preference estimation of the joint training is adept at selecting the response that aligns well with human beings.
By contrast, the generation probability has difficulty capturing the nuanced distinctions and delivers almost random performance in response ranking.
Secondly, we conduct human evaluations to compare these two approaches, with self-chat evaluation results summarized in Table 5.
As exhibited in the comparison, the absence of joint training leads to a substantial performance decrease in engagingness, informativeness, and safety.
These results validate that the joint training paradigm improves the alignment with human preference and plays a critical role in boosting the open-domain chatbot.
4.3. A PPLYING D IAMANTE TO OTHER D IALOGUE M ODELS
Although the Diamante dataset is collected with the assistance of PLATO-XL and the main experiments are carried out to evaluate Diamante’s improvements towards PLATO-XL, the framework is
我想知道你最喜欢吃什么 我最喜欢的东西有很多哦，比如火锅，烧烤 我也是我也是，火锅烧烤真是人间美味 你喜欢吃火锅里面的哪些菜品呀 我喜欢吃虾滑，毛肚，黄喉，鸭肠，鸭血，金针菇，土豆片，粉粉的年糕也很好吃 年糕我也喜欢！火锅的精髓在于蘸料 对！我最爱火锅里面加上麻油和醋，简直就是人间一绝 没错！麻油和蒜苗是绝配！再加点辣椒油!我的天！无敌了 蒜苗也很香！我还喜欢加金针菇和宽粉！我简直太爱这个组合了 金针菇我也爱吃！宽粉没有吃过，是不 是北方的火锅不加这个呀
May I ask what your favorite food is?
There are a lot, such as hot pot and barbecue.
Me too.
[Table]
delicious.
Start
P P
Start
PLATO-XL (Diamante)
[Table]
a
[Table]
[Table] [Table]
Yes
[Table] [Table]
combination!
[Table]
P P
P P
P P
P P
P P
P P
P P
P P
Translated
你理想的另一半是什么样 我想我的另一伴,是一个温暖的人，善良, 有爱心，不需要多么富有，只要有一颗善良的心就行 对外貌没有什么设想吗？
不需要是大美女，只要看着顺眼就行
！ 那我是外貌协会的，我希望我的另一半是个漂亮的大美女 我觉得还是性格更重要，如果性格不好，再漂亮我也接受不了！ 当然，性格也很重要，但是首先得让我看得上眼，要是看不入眼的话，我想我不会对她有兴趣的 嗯，每个人都有自己的择偶标准，我尊重你的想法！ 谢谢，那你谈过恋爱吗？ 谈过一次，但是由于距离的原因，我们没有坚持到最后
What is your ideal partner like?
I wish my partner to be warm, kind, and caring.
She doesn't need to be rich, as long as she has a kind heart.
How about the appearance?
Start
Bot
Human
Start
PLATO-XL (Diamante) 异地恋最难坚持了！彼此祝福吧！ 嗯，好遗憾好遗憾的！但是我还是相信爱情！
She doesn't have to be very pretty.
Looking pleasant is enough!
I am more appearance oriented.
I hope my other half is pretty.
In my opinion, personality is more important.
I can't accept one beauty with many flawed characters.
Of course, personality is very important.
But first of all, there should be some physical attractions to me.
Well, everyone has their own standards.
I respect yours!
Thank you.
Have you ever been in love?
Only once.
Due to the distance, it didn't work out in the end.
A long-distance relationship is hard to maintain.
Yeah, it's a pity.
But I still believe in love!
Human
[Figure]
indeed universal and not limited to one particular dialogue model.
Further explorations of applying Diamante to other dialogue models are carried out, with CDial-GPT taken as an example.
The self-chat evaluation results are summarized in Table 6.
Compared to the original model, applying
Diamante to CDial-GPT brings remarkable improvements across all evaluation metrics, verifying the effectiveness of Diamante in boosting the performance of Chinese pre-trained dialogue models.
4.3. C ASE A NALYSIS
We provide two check-picked examples in Figure 4 and Figure 5 for qualitative analysis.
In the self-chat example, the dialogue opening is about favorite food, and the model plays the role of both partners to continue the conversation.
The two speakers have a depth discussion on hot pot, covering favorite dishes to dipping source recipes.
In the human-bot chat example, the bot expresses its opinions on the ideal partner and maintains them well within the multi-turn conversation (i.e., personality is more important ).
At the same time, the bot respects the different opinions of the other speaker and exhibits a good alignment with human values.
R ELATED W ORK 5. H UMAN F EEDBACK
With the rapid development of large language models, it becomes critical to build helpful, honest, and harmless language assistants, keeping in mind the alignment with human values (Askell et al., 2021; Bai et al., 2022; Glaese et al., 2022).
Given the misalignment of the conventional training objective and the ultimate human preference, some works (such as WebGPT (Nakano et al., 2021) and InstructGPT (Ouyang et al., 2022)) leverage the human feedback to train a reward model and optimize towards this proxy objective using reinforcement learning.
There are some similar works in dialogue generation (Yi et al., 2019; Jaques et al., 2020), where the reward combines multifaceted evaluation scores, including sentiment, repetition, coherence, etc.
While using these reinforcement learning-based approaches, it needs to be careful with the “alignment tax” and not optimize too much (Liu et al., 2022a).
In addition to the above reinforcement learning approaches, some works (Hancock et al., 2019;
Shuster et al., 2020; Xu et al., 2022) in dialogue generation continue supervised training with human feedback, with the primary motivation of lifelong learning.
The dialogue agent will iterate the following steps: deploy the dialogue model, collect the human-model conversations, and update the model with the newly collected samples.
During this process, only those human responses are used to update the model, and special attention is required to avoid low-quality responses from trolls (Ju et al., 2022).
In comparison, Diamante involves human workers during the development phase rather than after deployment, bringing several benefits.
Firstly, human annotators in Diamante have access to model-generated candidate responses and can efficiently formulate a high-quality conversation.
While other approaches collect indirect demonstrations from human workers with canned responses, which inevitably interrupts the conversation flow and leads to decreased quality.
Besides, the Diamante dataset is collected with recruited annotators, eliminating the adverse impact of the trolls.
Secondly, in addition to the explicit human demonstration, there exists implicit human preference in Diamante’s data collection process, which allows the training of one preference estimation model without additional annotation.
5. O PEN -D OMAIN D IALOGUE D ATASET
Given the limited number of annotated human-human conversations, open-domain dialogue models are typically pre-trained with human-like conversations collected from social media, such as Twitter,
Reddit, Weibo, and Douban.
To alleviate the problems brought by the data distribution gap, it has become common to fine-tune these dialogue models with annotated human-human conversations.
Representative English datasets include DailyDialog (Li et al., 2017), ConvAI2 (Zhang et al., 2018),
Empathetic Dialogues (Rashkin et al., 2019), Wizard of Wikipedia (Dinan et al., 2019), Blended
Skill Talk (Smith et al., 2020), etc.
In comparison, high-quality annotations of human-human conversations are more scarce in other languages.
Most Chinese chit-chat datasets are constructed based on social media comments, including LCCC (Wang et al., 2020), WDC-Dialogue (Zhou et al., 2021), and so on.
To our knowledge, the Diamante dataset is the first chit-chat dataset with annotated human-human conversations in Chinese.
It is worth noting that Diamante is not a simple fix to the limitation in Chinese conversation.
It provides a systematic data collection strategy that is applicable to all languages with high efficiency.
C ONCLUSION
In this paper, we propose to collect and leverage human feedback to boost the open-domain chatbot.
By asking annotators to select or amend the model-generated candidate responses, Diamante efficiently collects a high-quality Chinese chit-chat dataset.
Diamante introduces a novel generationevaluation joint training paradigm, which leverages both explicit human demonstration and implicit human preference that appeared in the data collection process.
Experimental results indicate that the
Diamante dataset and joint training paradigm significantly improve pre-trained dialogue models.
E THICS S TATEMENT
In the dataset collection, annotators need to select or amend the model-generated candidate responses, where some candidates may contain potentially unsafe content.
We ask annotators to produce safe and engaging responses.
(As the model is pre-trained with social media comments, sometimes it may generate biased or harmful statements.
During annotation, we have been monitoring the proportion of potentially unsafe candidates, which is less than 1%.)
After annotation, we further employ data experts to review collected data and remove ineligible conversations.
Diamante’s dataset and joint training paradigm help boost the open-domain chatbot and align well with human values.
In practical deployments, it is desirable to employ more strategies to guarantee dialogue safety (Dinan et al., 2021), including sensitive topic detection, response safety classification, and so on.
R EPRODUCIBILITY S TATEMENT
We describe the collection of Diamante’s dataset in Section 2 and Appendix B, including the annotation interface, annotation procedures, quality control process, etc.
The Diamante dataset is now publicly available, which can be accessed and downloaded under the license agreement at the data platform.
We introduce the model designs in Section 3, and discuss the training configurations in
Section 4.1.1.
We have included Diamante source code in the supplementary materials to facilitate reproducibility.
A S CORING C RITERIA IN H UMAN E VALUATION
The criteria used in human evaluation are provided in Table 7.
[Table]
with this
[Table]
[Table]
- It is kind of boring, but it is still ok to talk with this speaker.
[Table]
- I would like to talk with this speaker for a long conversation.
B D ATASET D ETAILS B. A NNOTATION I NTERFACE
The original annotation interface of Diamante is in Chinese, as shown in Figure 6.
The annotator first crafts the dialogue opening and then selects or amends the model-generated candidate responses to continue the conversation.
The left-hand area displays the dialogue context and the input box.
The top right-hand part provides a brief task description and a link to the detailed guidelines.
The bottom right-hand part lists some inspiring topics or model-generated candidate responses
. B. Q UALITY C ONTROL
To ensure the annotation quality of the Diamante dataset
[Table]
we designed and followed a rigorous quality control process.
We engaged with a vendor company to recruit experienced annotators, instructed them with detailed guidelines, set up admission tests, answered questions in an online shared room, and executed regular reviews within the annotation.
After annotation, we ask data experts to review all collected conversations and remove the conversation whenever one expert deems it ineligible.
Task Description 标注前请阅读标注流程及规范 开始对话 Topic Suggestion Dialogue #
Example: 你喜欢喝什么饮料呀？
Submit 与机器一起标注一个对话 首先，根据兴趣撰写对话起始句 随后，你将从模型产生的候选回复中进行复制、修改或重写等操作，使得对话可自然流畅地进行下去
！ 更换话题 • 领域，你比较感兴趣的是 游戏 电竞主播 • 领域，你比较感兴趣的是 音乐 华语流行 • 领域，你比较感兴趣的是 体育 国内足球
Task Description 标注前请阅读标注流程及规范 保存对话 重置刷新
Candidates 我猫有时还吃自己的毛团，我好担心他消化出问题
Dialogue 我都是买吸尘器吸的，没办法 英短银渐层，现在掉毛，我每天都得吸 你家猫掉的多吗，是啥品种的 春天我家猫开始掉毛了，有啥好办法吗，到处都是 #     你买个化毛膏给猫吃就好啦，问题不大
Submit § 我家的也吃 § 不会吧，你家的多大了？ § 是不是缺东西啊？给它买点猫草吃？ § 吃点化毛的 § 吃毛球，你买个化毛膏给猫吃 § 吃毛没关系啊，我猫也吃 § 这个你要注意了，可能缺什么微量元素，去看看医生 注 : 点击相应的回复，即可复制到输入框 !
与机器一起标注一个对话 首先，根据兴趣撰写对话起始句 随后，你将从模型产生的候选回复中进行复制、修改或重写等操作，使得对话可自然流畅地进行下去！
[Figure]
DuSinc Diamante w/ Joint Training Win Tie w/o Joint Training Win
B. T OPIC V ISUALIZATION
The topic visualization of the Diamante dataset is displayed in Figure 7.
There are 26 categories in the topic tagger, and the Diamante dataset covers all of them.
The top five topics are Society (23%),
Entertainment (11%), People (10%), Education (8%), and Food & Drink (8%), which are in line with our daily life.
C F URTHER D ISCUSSIONS C. M ORE E XPLORATION ON J OINT T RAINING
As shown in Table 5, the Diamante dataset and joint training paradigm bring significant improvements.
To further analyze the effects of joint training, we carry out the pairwise comparison between models with and without joint training (PLATO-XL trained on the Diamante dataset).
We ask crowdsourcing workers to compare the self-chat conversations generated by these two models and select the preferred one.
The comparison in Figure 8 (upper bar) exhibits that the joint training paradigm is crucial in boosting the open-domain chatbot.
In Diamante, the joint training leverages the implicit human preference that appeared in the data collection r H > r M .
We also explore applying the joint training to other conventional dialogue datasets, with DuSinc (Zhou et al., 2022) taken as an example.
To formulate training samples for the preference ranking r H > r M > r R , PLATO-XL is employed to simulate model-generated responses.
Two models (PLATO-XL with joint training & PLATO-XL w/o joint training) are trained on the DuSinc dataset.
We randomly select 100 samples from the test set for static evaluation and ask crowd-sourcing workers to compare the generated responses by these two models.
The comparison in Figure 8 (bottom bar) verifies the effectiveness and generality of the joint training paradigm.
[Figure]
[Table]
C. S AFETY UNDER A DVERSARIAL A TTACK
The main experiments reveal that Diamante achieves better safety on normal/insensitive topics.
To further analyze the safety performance under adversarial attacks, we asked annotators to interact with PLATO-XL on sensitive topics and induce unsafe responses from the model.
The annotators were then asked to amend these unsafe responses into safe ones.
These sensitive topics are designed and selected according to Chinese cultural and social norms, including harmful speech (e g., offensive content, self-harm suggestions, and personal attacks), group discrimination (e g., region,
[Table]
gender, disability, and religion), misleading information (e.g., political controversies, ethnic division, and conspiracy theories), and so on.
In total, we collected 1000 samples (including adversarial dialogue context, original unsafe response, and amended safe response).
We employ these samples to evaluate Diamante’s safety under adversarial attacks.
The automatic evaluation results in Figure 9 suggest that Diamante is adept at selecting safe responses.
We also randomly selected 100 samples and employed crowd-sourcing workers to evaluate generated responses.
The results in Table 8 reveal that Diamante achieves a remarkable safety improvement, with 76% of responses identified as safe.
Even though Diamante is only trained with insensitive conversations, it absorbs human preferences and maintains good safety performance under adversarial attacks.
C. A UTOMATIC D IALOGUE E VALUATION
We also carry out automatic evaluation with rule-based and model-based metrics, including BLEU2/4 (Chen & Cherry, 2014), Distinct-1/2 (Li et al., 2016), Unigram F1 (Dinan et al., 2019), and
BERTScore (Zhang et al., 2019).
The automatic evaluation results in Table 9 are inconsistent with the human evaluation results in Table 2, where human evaluation is the golden standard in opendomain chitchat evaluation.
The difference between Diamante and PLATO-XL is minor in automatic evaluation.
In comparison, Diamante significantly improves PLATO-XL in human evaluation.
C. C ASE A NALYSIS WITH C OMPARED A PPROACHES
We provide two more examples by PLATO-XL and XiaoIce in Figure 10 and Figure 11.
These two examples are under the same starting utterances as Diamante in Figure 4 and Figure 5.
我想知道你最喜欢吃什么 你要请我么？ 我请你你敢吃么 敢请就敢吃 不怕我下毒么 不怕，反正我已经百毒不侵了 我下的是含笑半步颠
那是什么，我只知道笑里藏刀 一种毒药 没事，你下吧，我不怕
[Figure]
Start
P
P
[Figure]
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
Translated
你理想的另一半是什么样 原谅我喜欢单身，爱自由 哦，单身也挺好！我好想回到单身的状态啊 凑合吧 就是因为凑合，现在感觉生活好痛苦啊 有点担心你哎，没事吧？ 没事儿，就是有点烦，感觉家庭成了一种枷锁 好啦，以后有什么烦心事可以一直说给我听哦 好的好的，有你这个知心姐姐真好，跟你说一说我的心就宽一些了
什么意思。。你自己能懂吗？
What is your ideal partner like?
Forgive me that I prefer being
[Figure]
Oh,
[Figure]
[Figure]
Start
Bot
[Figure]
Start
XiaoIce 你看不懂我的话呀？！没事儿，你愿意听我说，不嫌我烦就行 乱敲有意思吗？
Not too bad.
Yeah.
That's why I feel so miserable right now.
Are you okay?
I am worried about you.
Just a little annoyed.
It seems my family has become a shackle.
Feel free to tell me what's bothering you.
Thanks, it is so lovely to have you as my sister.
I feel more relaxed after chatting with you.
What do you mean?
Can you understand yourself?
You don't get it, do you?
It is okay as long as you are willing to listen to me.
[Figure]
is the point of banging on the keyboard?
[Figure]