Leveraging Few-Shot Data Augmentation and Waterfall Prompting for Response Generation (2023)

Abstract: This paper discusses our approaches for task-oriented conversational modelling using subjective knowledge, with a particular emphasis on response generation. Our methodology was shaped by an extensive data analysis that evaluated key factors such as response length, sentiment, and dialogue acts present in the provided dataset. We used few-shot learning to augment the data with newly generated subjective knowledge items and present three approaches for DSTC11: (1) task-specific model exploration, (2) incorporation of the most frequent question into all generated responses, and (3) a waterfall prompting technique using a combination of both GPT-3 and ChatGPT.

## introduction task-oriented dialogue (tod) systems are traditionally designed to facilitate users in achieving specific objectives, such as looking up train times or booking a flight in a dialogue setting. for these tasks, the models are often given access to a database of factual information to complete the task. however, other tasks necessitate not only factual but also subjective insights, which are derived from other users' opinions. handling subjective knowledge and using it for generating dialogue responses is the core of the subjective-knowledge-based task-oriented dialogue (sk-tod) @xcite challenge. the challenge is set up as conversations between users and artificial assistants, inquiring about and potentially booking hotels or restaurants. the organisers provided dialogue snapshots and a knowledge base with subjective reviews and faqs related to said hotels and restaurants. the challenge consists of three interlinked subtasks: 1) knowledge seeking turn detection, where it is determined whether a turn needs knowledge to create an appropriate response; 2) knowledge selection, where relevant items are selected from the knowledge base; and 3) knowledgegrounded response generation, where the dialogue history and the selected knowledge items must be aggregated to generate a concise response for the user. our primary interest was in sub-task 3, but due to the high correlation of performance with subtask 2, we simultaneously worked on improving the knowledge selection (see figure 1 ). furthermore, we experimented with splitting sub-task 3 into two smaller tasks. first, we perform few-shot data augmentation, supported by a detailed data analysis, to substantially increase the knowledge base and address generalisation to new domains. despite its marginal impact on the final test set, given no new domains were added, it forms a robust strategy for future generalisations. second, we undertook the exploration of task-specific models, we found that using a larger model leads to slight improvements in task performance. in addition, we augmented the generated responses by integrating the most frequently asked question, aiming to enhance conversation coherence and engagement. lastly, we also employed a waterfall prompting technique that integrated a combination of large language models. this strategy, though it scored lower in quantitative metrics compared to the baseline, possibly due to the abstractive nature of these models, is a promising avenue for further research foot_0 . waterfall prompting combining prompts or rerunning the same prompt has shown increased performance and robustness. for example, @xcite coin the term ensemble refinement which involves using a two-step prompting approach. in the first iteration, multiple answers are generated for a prompt, and in the second iteration, these answers are incorporated into the prompt to develop a final solution that is more robust for the given task. similar efforts have been explored such as self-consistency @xcite , recitation-augmentation @xcite , and rational-augmentation @xcite . @xcite expand on this framework by incorporating multiple rounds of prompting, each building upon the improvements made in the previous round. however, these still focus on prompting a single model and differ from a typical ensemble where different models' output is aggregated @xcite . to the best of our knowledge, there is limited work on waterfall prompting or assembling the responses from different language models. previous dstc editions according to system reports from the dstc9 and dstc10 challenges, enhancing the performance of sub-task 2 significantly affects the overall task performance @xcite . in the dstc9 iteration, it was found that ensemble model approaches are effective for knowledge selection @xcite . in dstc10, successful approaches used a separate entity tracking component for knowledge selection to narrow down the search space before document ranking @xcite . ## data analysis before developing our approach to the dstc tasks, we analyse the available data. we 1. inspect the correlation between features, 2. investigate the sentiment of the available data, and 3. examine the distribution of dialogue act types. ## results in the sub-task of knowledge seeking turn detection, our added knowledge induced a slight decrement in performance, with precision declining from 0.998 to 0.995 and f1 score from 0.998 to 0.997, albeit recall experienced a marginal increase (refer to conversely, knowledge selection saw an improvement with the augmentation, resulting in a small uplift across all metrics: precision from 0.790 to 0.796, recall from 0.788 to 0.794, f1 from 0.789 to 0.795, and exact match from 0.91 to 0.418. we submitted three different approaches for the response generation task. (1) our first submission is based on the model exploration mentioned in 4.2.1. after fine-tuning flan-t5-large this approach outperforms the baseline on all metrics, leading to a mean reciprocal rank (mrr) of 0.068 compared to 0.039 of the baseline. on bleu this approach ranked 7th out of 49 submissions. ## conclusion in conclusion, our work on the task-oriented conversational modeling with subjective knowledge task produced three key contributions: firstly, our detailed data analysis could serve as a basis for future dataset scoping within this domain. secondly, we significantly expanded the knowledge dataset size utilising few-shot data augmentation. lastly, our most successful model was a fusion of the baseline with augmented data and flan-t5-large. our waterfall prompting approach incorporating a blend of large language models demonstrated lower metrics compared to the baseline, but upon a qualitative assessment, the results were deemed satisfactory, albeit the absence of official human evaluation impedes a definitive judgement. a preliminary qualitative analysis shows that chatgpt can spot mistakes during knowledge selection. thus, future work could explore incorporating an initial step where chatgpt is employed to evaluate the relevance of the selected knowledge items. additionally, we plan to experiment with summarising positive and negative reviews separately before creating a consolidated summary. this method may enhance performance when dealing with polarised reviews.