To Clarify or not to Clarify: A Comparative Analysis of Clarification Classification with Fine-Tuning, Prompt Tuning, and Prompt Engineering (2024)

Abstract: Misunderstandings occur all the time in human conversation but deciding on when to ask for clarification is a challenging task for conversational systems that requires a balance between asking too many unnecessary questions and running the risk of providing incorrect information. This work investigates clarification identification based on the task and data from (Xu et al., 2019), reproducing their Transformer baseline and extending it by comparing pre-trained language model fine-tuning, prompt tuning and manual prompt engineering on the task of clarification identification. Our experiments show strong performance with LM and a prompt tuning approach with BERT and RoBERTa, outperforming standard LM fine-tuning, while manual prompt engineering with GPT-3.5 proved to be less effective, although informative prompt instructions have the potential of steering the model towards generating more accurate explanations for why clarification is needed.

## introduction humans often communicate when they do not understand something and are able to collaboratively avoid and resolve misunderstandings by clarifying them. clarification questions can be used to establish common ground between interlocutors @xcite . effectively repairing misunderstandings would be a desirable feature for conversational systems, thereby keeping the conversation between user and system as natural and efficient as possible. as noted by @xcite , understanding users' underlying needs is critical for conversational systems, where the input is often limited to short questions. when system confidence in user intent is low, a clarification request (cr) should be generated to resolve ambiguity. however, handling uncertainty in conversational systems moves along a thin line between over-and under-generation of clarification @xcite . asking too many or unnecessary clarification questions can lead to user frustration @xcite , while asking too few runs the risk of providing the user with incorrect information. hence, clarification request identification is an important task for conversational systems and it may also rely on additional information coming from a knowledge base (kb), as in the claqua @xcite dataset used in this work. we focus on clarification in a knowledge-based question answering (kbqa) setting and compare three approaches for modelling clarification identification with claqua: pre-trained language model fine-tuning, prompt tuning and manual prompt engineering. ## related work while research on clarification in conversational systems was for a long time held back by a lack of datasets @xcite , @xcite now observe a growing number of research approaches and datasets on the topic. datasets for clarification in question answering (qa) systems include raocq @xcite and clarq @xcite , built from stackexchange posts. qulac is a dataset for conversational search introduced by @xcite and claqua @xcite supports clarification identification with a knowledge base. several approaches focus on identifying ambiguity in user queries to improve performance of kbqa systems. @xcite predict whether system confidence is high enough to answer the query before the user is asked to choose from a list of possible relevant entities. @xcite study to which extent neural models can generate crs in conversational qa and introduce the abg-coqa corpus for clarifying ambiguities in reading comprehension questions. the neurips nlp shared task @xcite also addresses the problem of when the agent should ask for clarification using a simulated minecraft environment to benchmark different models. ## experiments our work compares three different approaches of leveraging pre-trained language models (plms) for clarification identification in a kbqa system: language model (lm) fine-tuning, manual prompt engineering with large language models (llms) and prompt tuning. the task is to determine whether a cr is needed or the context already provides enough information to decide for which entity to answer the question. ## conclusion our comparative analysis of different approaches shows that lm fine-tuning and prompt+lm tuning lead to good task performance. the best clarification identification results on claqua are achieved with a joint lm and prompt tuning approach. the results indicate that the linguistic knowledge gained from pre-training can be leveraged with transformer-based lms, modelling the clarification identification task with only the conversation context and entity text descriptions. for future work, we consider the use of various other models, for example deberta @xcite or electra @xcite . other promising research directions include: (1) generating clarification questions and joint modeling of clarification identification and generation, (2) conducting a user study to investigate how users react to under-and over-represented clarification questions in dialogue and (3) analysing to what extent state-of-the-art dialogue systems can benefit from explicit clarification question identification. manual prompt engineering with gpt-3.5 was not competitive in terms of clarification identification scores. however, with informative prompt instructions, manual prompt engineering can be used for deeper analysis of the interaction between context and entity information and the reasoning process for why user questions need clarification or not. even though prompt responses leave room for improvement, they show a direction worth exploring further.