Question Generation to Elicit Users’ Food Preferences by Considering the Semantic Content (2023)

Abstract: To obtain a better understanding of user preferences in providing tailored services, dialogue systems have to generate semi-structured interviews that require flexible dialogue control while following a topic guide to accomplish the purpose of the interview. Toward this goal, this study proposes a semantics-aware GPT-3 fine-tuning model that generates interviews to acquire users’ food preferences. The model was trained using dialogue history and semantic representation constructed from the communicative function and semantic content of the utterance. Using two baseline models: zero-shot ChatGPT and fine-tuned GPT-3, we conducted a user study for subjective evaluations alongside automatic objective evaluations. In the user study, in impression rating, the outputs of the proposed model were superior to those of baseline models and comparable to real human interviews in terms of eliciting the interviewees’ food preferences.

## introduction with interviews being used for various purposes, interview systems such as surveys @xcite , job interviews @xcite , and coaching @xcite have been developed. interviews are categorized into three types: structured, semi-structured, and unstructured. in terms of flexibility, semistructured interviews are between structured and unstructured. they are not completely planned but have a topic guide that needs to be covered. to build a dialogue system that can generate semistructured interviews, flexible dialogue control must be provided while following the topic guide. to address the issues involved in generating semistructured interviews, this study proposes an interview system to learn user food preferences. various dialogue control mechanisms have been studied in task-oriented dialogue systems to collect information from users, with the system responses are determined based on manually defined rules, pomdp @xcite , deep learning @xcite , and reinforcement learning @xcite . however, these systems have less flexibility in dialogue control because the dialogue states are defined as a set of slot-value pairs that are limited to the task domain. research on open-domain non-task-oriented dialogue generation has contributed to the development of chitchat systems that can produce system responses for various topics. initially, a simple sequence-to-sequence approach @xcite @xcite was employed to generate a response. this approach has been improved to produce appropriate and meaningful responses, considering the dialogue context @xcite , and generate knowledge-grounded responses @xcite @xcite . more recently, chatgpt @xcite has demonstrated remarkable performance in generating rich and natural dialogues. however, these techniques have not yet been designed to generate dialogues for user model acquisition. consequently, interview systems are required to generate responses that are aligned with the purpose of the interview. to overcome the problems discussed above and generate useful questions in semi-structured interviews to elicit user food preferences, this study proposes a gpt-3 based model trained to generate responses with its semantic representation, which is constructed from the utterance's communicative function and semantic content. semantic content refers to a structured sequence of labels for objects and their attributes. it is expected that using semantic content as part of the training targets would help constrain the generated responses towards eliciting the user food preferences. the contributions of this study are as follows: 1) a semantic representation is proposed for system responses; 2) a response generation model is created for the interviewer's role; and 3) the effectiveness of the proposed method in eliciting user preferences is demonstrated through an evaluation experiment. ## corpus collection to prepare the dataset used in this study, text-based dyad conversations were collected to interview participants regarding their food preferences. the participants were recruited through crowdsourcing. each participant was assigned the role of either interviewer or interviewee and communicated using a chat system on a web browser. the interviewer was instructed to elicit the partner's preference for food, whereby they exchanged messages taking turns, for a minimum of 40 turns. thus, a total of 118 japanese dialogues were collected. ## method to train a response generation model for the interviewer's role by considering the semantic representation of the interviewer's responses, we propose the method illustrated in figure 1 . first, the semantic representation of the interviewer's responses is presented, and subsequently, model training is explained. ## experiment and evaluation we evaluated the performance of the proposed cf+sc model using three comparison targets: the ground truth and two baselines. ground truth (gt): actual utterances of the interviewers were used as the ground truth. fine-tuned gpt-3 (seq2seq): this simple finetuning model uses gpt-3. the model was trained without semantic representation (cf and sc) of the prediction targets. a sequence of preceding utterances was provided as prompt, and the model output was the interviewer's response. zero-shot chatgpt (chatgpt): openai's chat-gpt model (reinforcement learning with human feedback and chat-optimized models @xcite ), specifically, gpt-3.5-turbo-0301, was adopted as the best general-purpose dialogue model. the zero-shot method was employed such that only the dialogue history and the system's role as an interviewer were provided as prompts 1 . the 1 we also tested the few-shot chatgpt, with prompts including two example responses accompanied by cf and sc, system was instructed to play the role of the interviewer and generate a response to elicit customer preferences by considering the context. the temperature parameter for the three gptbased models was set to 0. thus, the generation was almost deterministic. while the cf+sc model generates both semantic representation and text of the response, we used the system_output part to extract the system response text. the cf+sc and seq2seq models generate a single sentence. thus, in order to align the comparison conditions, when chatgpt model generates multiple sentences, the last sentence, which tends to contain the main claim, was used in comparing with the ground truth. the gpt-3 ("davinci" model) was fine-tuned using openai's api. the model was trained for four epochs. the batch size was eight, and the learning rate was 0.05. the validation loss remained constant after epoch two. the number of instances used for training and validation were 1671 and 206, respectively. ## conclusions and future directions this study proposed a response generation model aiming to extract user preferences for food. we trained the gpt-3 based model using a communicative function and semantic content. the results of the human impression evaluation experiment showed that the proposed model outperformed zeroshot chatgpt and fine-tuned gpt-3 model, and comparable to real human interviews in terms of eliciting the interviewee's preferences. one limitation of the current model is that it produces only a single sentence. in the future, this model should be improved to generate more complex responses using multiple sentences. moreover, it is necessary to evaluate the model's performance in interactions with users, and examine whether the interview system is useful for understanding users.