Learning From Free-Text Human Feedback – Collect New Datasets Or Extend Existing Ones? (2023)

Abstract: Learning from free-text human feedback is essential for dialog systems, but annotated data is scarce and usually covers only a small fraction of error types known in conversational AI. Instead of collecting and annotating new datasets from scratch, recent advances in synthetic dialog generation could be used to augment existing dialog datasets with the necessary annotations. However, to assess the feasibility of such an effort, it is important to know the types and frequency of free-text human feedback included in these datasets. In this work, we investigate this question for a variety of commonly used dialog datasets, including MultiWoZ, SGD, BABI, PersonaChat, Wizardsof-Wikipedia, and the human-bot split of the Self-Feeding Chatbot. Using our observations, we derive new taxonomies for the annotation of free-text human feedback in dialogs and investigate the impact of including such data in response generation for three SOTA language generation models, including GPT-2, LLAMA, and Flan-T5. Our findings provide new insights into the composition of the datasets examined, including error types, user response types, and the relations between them.

## introduction if a dialog system generates a dissatisfying or erroneous response, e.g., a response with factually incorrect information, users tend to provide a textual descriptions of what went wrong or what they would have expected @xcite @xcite . this textual description is usually referred to as free-text human feedback, and it is an important source to improve dialog systems, to keep them engaging and socially acceptable @xcite @xcite . learning from such data requires annotations for errors 1 code and data are available on github. in system utterances and subsequent free-text human feedback, but available datasets are scarce and typically cover only a small subset of the error types known in conversational ai, such as in the cases of saferdialogues @xcite and fits @xcite . as a result, data collection is usually a first step for research on learning from free-text human feedback @xcite @xcite . to avoid this in future research, recent advances in synthetic dialog generation @xcite could be used to augment existing dialog datasets with annotations for errors and free-text human feedback. however, to assess the feasibility of such an effort, it is important to know the types and frequency of such data included in these datasets. in this work, we investigate this question for a variety of dialog datasets, including multiwoz @xcite , per-sonachat @xcite , wizards-of-wikipedia @xcite , sgd @xcite , babi @xcite , and the humanbot split from the self-feeding chatbot @xcite . for this, we follow a two-step approach in which we first use the integrated error taxonomy proposed by @xcite to study the types of errors in system utterances and subsequent user responses in subsets of these datasets. we use the insights gained during this process to derive a new user response type taxonomy and a modified integrated error taxonomy for the annotation of free-text human feedback in dialog data. in the second step, we use sentence-transformer @xcite to identify similar situations in the remaining dialogs of the datasets. for a subsequent statistical analysis, we manually annotate subsets of the identified dialogs with error and user response types. we also use this manually annotated data to investigate the impact of including this data in re-sponse generation, using three sota language generation models, i.e., gpt-2 @xcite , llama @xcite , and flan-t5 @xcite . we find that types and frequency of errors and user responses that include free-text human feedback largely depend on (1) whether the dialogs are human-human or human-bot, and (2) whether the dialogs are task-oriented, open-domain, or knowledge-grounded. our analysis and experiments show that using our taxonomies is beneficial for identifying free-text human feedback, and that including such data has a positive impact in response generation. ## related work datasets annotated with free-text human feedback as of today, dialog datasets with annotations for free-text human feedback are few and mostly focused on specific error types. for example, fits @xcite , a dataset of 14,000 human-bot dialogs, provides free-text human feedback for response quality and search query errors. saferdialogues @xcite consists of 7,000 human-bot dialogs, but only addresses toxicity in system utterances. the dataset published with the self-feeding chatbot @xcite provides 60,000 human-bot dialogs, partly annotated with response alternatives for dissatisfying system utterances. other works, such as @xcite and @xcite collected free-text human feedback for evaluating their approaches, but never made the data publicly available. in this work, we investigate the types of errors and subsequent user responses in six widely used dialog datasets, such as multiwoz @xcite , personachat @xcite , and sgd @xcite to assess their extendibility with annotations for learning from freetext human feedback as an alternative to data collection from scratch. error and user response type taxonomies the error taxonomies used to collect the datasets discussed above are very specific which limits their applicability, e.g., in saferdialogues @xcite , they only focus on toxicity. however, errors in conversational ai have long been the subject of research and more comprehensive error taxonomies are already available. for example, @xcite and @xcite focus on errors in task-oriented dialog systems and distinguish multi-ple error types with focus on content-related errors and practical aspects. more recently published error taxonomies, such as the integrated error taxonomy proposed by @xcite , distinguish levels of errors, which makes them more broadly applicable. the integrated error taxonomy covers 17 error types in four different levels, including utterance, response, context, and society. regarding user responses subsequent to errors in system utterances, @xcite proposed a taxonomy that distinguishes dissatisfaction and unclear user utterances. in this work, we use the integrated error taxonomy by @xcite to study the errors in the system utterances of 1,200 dialogs from six dialog datasets, based on which we determine (1) the different types of user responses to errors in system utterances, and (2) the limitations of the integrated error taxonomy, such as missing error types. for task-oriented dialog datasets, we consider multiwoz @xcite (mwoz), sgd @xcite , and babi @xcite . they mainly differ in the number of domains included in the dialogs. mwoz includes seven different domains, sgd 16, and babi only one (but with dialogs of increasing difficulty). in contrast to mwoz and sgd, babi consists of human-bot dialogs. for open-domain dialogs, we consider personachat @xcite (pc) and the human-bot split of the self-feeding chatbot @xcite . while pc consists of dialogs between two people who are trying to get to know each other, sfc consists of human-bot open-domain dialogs foot_0 . for knowledge-grounded dialogs, we focus on wizardsof-wikipedia @xcite (wow), which consists of human-human dialogs. ## datasets examined for simplicity, we do not distinguish between human or bot in the following. we always refer to the utterance of the partner as a system utterance. ## manual error type analysis and taxonomies we first study the errors in system utterances in a randomly sampled set of 1, 200 dialogs (200 dialogs from each of the six datasets), using the integrated error taxonomy proposed by @xcite . the taxonomy consists of 17 error types (i1-i17) across four levels: utterance, response, context, and society. they further categorize error types into content violation, i.e., if the error may cause a dialog breakdown, and form violation, i.e., if the system utterance is not interpretable due to massive grammatical problems. if we find an error, we analyze the following user response for an error-indicating phrase, a short text fragment of arbitrary length that directly addresses the error in the previous system utterance (e.g., how do you mean) or indicates user dissatisfaction (e.g., i don't like that), and add the surrounding sentence to our list of unique error-indicating sentences. we use these error-indicating sentences in section 5 to explore the remaining dialogs from each dataset for user responses (and thus errors) that are similar to the ones observed in this step foot_1 . overall, we found 79 errors in system utterances and collected a set of 67 error-indicating sentences with an average sentence length of approximately 6.52 words (see appendix b for all collected phrases and sentences). each sentence contains a unique error-indicating phrase with an average length of 3.52 words. contractions (two words that have been connected, e.g., don't or it's) are considered as one word. task-oriented open-domain know.-grounded dataset mwoz (hh) sgd (hh) babi (hb) pc (hh) sfc (hb) wow (hh) #sentences 7 0 5 9 36 10 we find most error-indicating sentences in opendomain and knowledge-grounded datasets, especially in sfc @xcite . ## automatic filtering for potentially relevant dialogs since our study in section 4 indicated that errors in system utterances are rare, we use sentence-transformer @xcite to facilitate the process of filtering the remaining dialogs of each datasets for potentially relevant ones, i.e., dialogs with user responses similar to the collected error-indicating sentences. for each dataset, we decompose every dialog into turns (alternating utterances), extract the user response, and segment it into sentences. next, we pair these sentences with each of the errorindicating sentences and use a pretrained sentence-transformer based on mpnet @xcite to calculate their cosine similarity (see appendix c for implementation details). we consider a dialog to be potentially relevant if at least one of these pairs has a cosine @xmath0. with 58.5%, mwoz @xcite contains most of the potentially relevant dialogs. pc @xcite and wow @xcite have the smallest number of such dialogs (8.9% and 7.57%, respectively). overall, only 25% of the data is potentially relevant, i.e., contains at least one user response that is similar to one of those observed in section 4. hereinafter, we refer to these dialogs as filtered dialogs. we provide a sentence-level analysis in appendix d. ## statistical analysis in this section, we conduct a statistical analysis of the distribution of error and user response types and their relations in the dialogs of the datasets examined. for this, we manually annotate 555 of the filtered dialogs (100 from each dataset, if available) with error and user response types, using the taxonomies proposed in section 4.1 and 4.2. to avoid bias from our filtering procedure in section 5, we also consider 600 randomly selected dialogs (100 from each dataset) that were not identified during this process (@xmath1. hereinafter, these dialogs are referred to as random dialogs. in section 6.4, we assess the impact of our filtering procedure on this analysis. overall, we manually annotate 1,155 dialogs with error and user response types. for annotation, we always consider the entire dialog (the context). ## evaluation and experiments in this section, we use the manually annotated dialogs from section 6 in a human evaluation to assess the impact of our modifications to the integrated error taxonomy @xcite . we also use these dialogs to investigate the performance impact of errors in system utterances and subsequent user responses as additional input signals for response generation in three sota language generation models, including gpt-2 (radford et al., 2019), llama @xcite , and flan-t5 (chung et al., 2022). ## discussion the goal of this work was to investigate the type and frequency of errors in system utterances and subsequent user responses included in the datasets examined to assess their extendibility with annotations for learning from free-text human feedback. we found that this mostly depends on whether the dialogs are human-human or human-bot. in humanhuman dialogs, we find that humans rather suggest disagreements in a very polite way instead of accusing the partner of a mistake (see appendix j for examples). accordingly, there is only little free-text human feedback available that could be used for learning (section 6.2 and 6.3). therefore, it might be hard and ineffective to extend these datasets with annotations for learning from such data. this is different in human-bot dialogs, where humans often react harshly and accusingly to errors in system utterances, resulting in more direct feedback. however, we also found that it depends on the dialog type. in general, we find that opendomain and knowledge-grounded dialogs contain a larger number of errors and user responses that are likely to contain free-text human feedback, making them more suitable for this purpose (section 6.1). using the manually annotated dialogs from section 6, our experiments in section 7.2 suggest that including user responses to errors in system utterances has a positive impact in response generation, which supports the findings from recent works on including free-text human feedback @xcite . additionally, our results suggest that including the error-annotated system utterance itself can have a positive impact. from our point of view, distinguishing between user response types could be an interesting alternative to binary signals, such as user satisfaction @xcite or thumbs-down @xcite , as an indicator of an error in a system utterance. however, the dialogs annotated in section 6 do not provide enough such data for a thorough analysis that also takes into account the different types of user responses. therefore, we leave this as a research question for future work. our human evaluation in section 7.1 shows that our proposed taxonomies may serve as a promising starting point to obtain the necessary annotaions, although they may not cover all possible error and user response types. ## conclusion in this work, we examined the dialogs of six datasets from various types, including multiwoz, sgd, babi, personachat, wizards-of-wikipedia, and the human-bot split from the self-feeding chatbot, for errors in system utterances and the types of subsequent user responses to assess their extendibility with annotations for learning from free-text human feedback. our results show that this largely depends on whether the dialogs are human-human or human-bot, and whether they are task-oriented, open-domain, or knowledgegrounded. we found that human-bot dialogs, contain more errors in system utterances that are addressed with free-text human feedback in subsequent user responses, especially in the case of opendomain and knowledge-grounded dialogs. therefore, it might be feasible to extend these datasets with the needed annotations to support research into methods for learning from free-text human feedback, e.g., by taking advantage of the recent developments in synthetic data generation. we also used the insights gained during this process to propose a new user response type taxonomy and a modified integrated error taxonomy for the annotation of free-text human feedback. our experiments show that including errors from system utterances and subsequent user responses has a positive impact in response generation.