Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2023)

Abstract: Frozen models trained to mimic static datasets can never improve their performance. Models that can employ internet-retrieval for up-to-date information and obtain feedback from humans during deployment provide the promise of both adapting to new information, and improving their performance. In this work we study how to improve internet-driven conversational skills in such a learning framework. We collect deployment data, which we make publicly available, of human interactions, and collect various types of human feedback – including binary quality measurements, free-form text feedback, and fine-grained reasons for failure. We then study various algorithms for improving from such feedback, including standard supervised learning, rejection sampling, model-guiding and reward-based learning, in order to make recommendations on which type of feed- back and algorithms work best. We find the recently introduced DIRECTOR model (Arora et al., 2022) shows significant improvements over other existing approaches.

## introduction large language models employed as dialogue agents are primarily trained on human-written documents and human-human conversations collected from the web for pre-training @xcite , and humanhuman crowdsourced conversations @xcite for fine-tuning. the models are then used at inference time to conduct conversations with humans, with no further learning taking place @xcite . humanmodel conversations -which are never seen at training time -can have a quite different distribution to the original human-human training data used, and our current techniques can lose performance due to lack of robustness to such deviations @xcite . ## related work there are a number of existing methods for collecting human feedback from human-model conversations. deployed models can be improved in 1 https://parl.ai/projects/fits/ symmetric conversations conducted between models and humans by learning to mimic human conversationalists, as shown in the light dialogue game @xcite . this is not directly applicable if the conversations are asymmetric, for example in the case of one speaker (human) who asks the questions, and the other (bot) who always answers, as there would be no human supervision of the answers. in the non-symmetric case, one can however try to make use of the textual response from humans when conversing with the bot, but alternative learning methods must then be used. @xcite studies models that learn how to ask questions in order to learn from the answers, while @xcite learns from general textual feedback/comments, particularly in the case where the bot has produced a low quality response. another approach is to learn a reward signal (positive or negative reaction) based on user textual responses, as shown in the "self-feeding chatbot" @xcite . finally, rather than using conversational feedback, one can use sophisticated web-based uis to collect data, for example stack ranking potential responses @xcite . outside of the dialogue domain, there are numerous studies attempting to improve language skills from deployment, including never-endinglearning from language data @xcite , learning for the web search task directly @xcite or the dynabench system which covers a number of nlp tasks @xcite . @xcite also learns to use internetaugmentation for generation, like this work, but for question answering, not multi-turn dialogue. ## feedback learning methods in the following, we will describe the methods we will experiment with for learning from the collected human feedback. ## experimental results we provide automatic evaluation results in internet-augmentation helps first, this is an expected result, due to the nature of our tasks, but we find that using internet-augmentation helps in line with other internet-based dialogue tasks @xcite @xcite . we find that bb2 and seeker, which both perform internet search and condition on documents, outperform bb1 that does not. this improvement is quite large, e.g. bb1 has 24.8% good responses, compared to bb2 and seeker having 33.2% and 49.3% respectively. seeker, which has a modular search architecture that aims to use retrieved knowledge more accurately, performs markedly better than bb2, which is in line with previous results on other datasets @xcite . human feedback helps across the board we find different kinds of feedback can improve our base models bb2 3b and seeker 3b; we will analyse specific methods further in the subsequent discussion. these overall improvements can be seen in terms of all the human evaluation metrics measured (good response%, rating, and all three error breakdown types), as well as the automatic evaluation metrics we measured (f1 and ppl). we also generally (although not in every single case) see correlation between automatic and human evaluation metrics, e.g. the best methods are best in both types of metric. modular superior to non-modular feedback in the modular feedback setting humans give feedback about what has gone wrong in the pipeline of the model: whether the internet search query was poor, or the document/knowledge chosen after searching was poorly chosen. taking into account modular feedback outperforms using only supervised feedback of final responses in both automatic metric and human evaluations for both bb2 and seeker models. for bb2 we see close to 2% improvement in good responses for modular feedback compared to supervised feedback (40.3% → 42.0%), with both far superior to bb2 without feedback (33.2%). however, seeker which has a modular design, and hence is much easier to supply modular feedback to (as the supervision can directly train each module) sees a larger improvement of 4.5% (52.2% → 56.7%). free-form feedback is useful (but not as much as gold labels) free-form feedback also gives clear gains over the baseline model for both bb2 and seeker, but falls short of supervised feedback by 3% and 1% respectively for the two model variants. this does not seem surprising as supervised feedback directly gives a clear loss to optimize (simply try to generate the suggestion) whereas feedback is less clear a signal, depending on how it is phrased. however, we do not rule out other free-form feedback algorithms giving better results in the future, see e.g. @xcite for a recent method. combining multiple feedback signals (where di-rector works best) if one has access to multiple feedback signal types, some of the algorithms we have tried are capable of using them all. in particular, we can train director with both binary feedback (to train the classifier head) and module feedback (to train the language modeling head for the different modules). this gives the best results out of all methods for both base models by quite a margin in both automatic and human evaluations. e.g., for improving the bb2 base model this gives 47.0% good responses, compared to the original baseline of 33.2% or even director with only binary feedback of 37.8%. we see this trend is also apparent in other algorithms, as we also measure the performance of supervised feedback + reranking, which also gives gains over either of those methods alone in automatic evaluations, although it still lags behind director. iterative deployment and feedback collection improves results further during the process of evaluating all the models that were trained with v1 data described above, more data was collected from those models, which we refer to as the v2 split (see subsection 3.5). we can thus then train models on the v2 split, yielding potentially improved models. in the ideal case one could conduct an iterative continual learning setup, each time retraining on the data collected from previous rounds, improving further each time. we test this setup by training director (module+binary feedback), our best system from v1, with the v2 data split. the result shown in very large models benefit from feedback from smaller models opt-175b, either in zero-shot or few-shot variants is only pre-trained on dialogue data, and not fine-tuned on our task, and performs reasonably -but not better than smaller models that are fine-tuned. blenderbot 3 @xcite is trained with the modular supervision feedback data collected from the smaller (3b parameter) models, in addition to fine-tuning on other standard dialogue datasets. this model provides the best human evaluation metrics of all the systems we test, with a good response rate of 64.8% and a rating of 4.08. this indicates: (i) how important fine-tuning with relevant data is even to very large models; and (ii) even though our data was collected with feedback from small models fine-tuning using this data still brings large gains to larger models. this is an encouraging result as models are improving in architecture and increasing in scale over time, but data we have collected in the past should still remain useful for these models in the future. we provide cherry picked and lemon picked examples of bb3-175b in appendix b, as well as comparing to opt-175b. while there a number of success cases, even our best models still make factual errors and contradictions in some cases. hence, it appears that continued interaction with further feedback collection in the future will be beneficial for further improvements. ## conclusion in conclusion, we have studied whether a conversational model can learn new skills after the standard pre-training / fine-tuning setup by interacting with humans during its deployment. we study the use of different kinds of user feedback data and different learning algorithms for leveraging them, in order to compare their performance. we find that granular (modular) feedback about types of errors can yield strong performance, which can also work very well in conjunction with binary feedback using the recently introduced director model, yielding our best results. evidence also suggests that iterative retraining and redeployment also brings further gains, and that the feedback collected is useful for models differing from the ones originally conversed with, e.g., if much larger models are used in the future.