Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness (2024)

Abstract: Clarifying questions are an integral component of modern information retrieval systems, directly impacting user satisfaction and overall system performance. Poorly formulated questions can lead to user frustration and confusion, negatively affecting the system’s performance. This research addresses the urgent need to identify and leverage key features that contribute to the classification of clarifying questions, enhancing user satisfaction. To gain deeper insights into how different features influence user satisfaction, we conduct a comprehensive analysis, considering a broad spectrum of lexical, semantic, and statistical features, such as question length and sentiment polarity. Our empirical results provide three main insights into the qualities of effective query clarification: (1) specific questions are more effective than generic ones; (2) the subjectivity and emotional tone of a question play a role; and (3) shorter and more ambiguous queries benefit significantly from clarification. Based on these insights, we implement feature-integrated user satisfaction prediction using various classifiers, both traditional and neural-based, including random forest, BERT, and large language models. Our experiments show a consistent and significant improvement, particularly in traditional classifiers, with a minimum performance boost of 45%. This study presents invaluable guidelines for refining the formulation of clarifying questions and enhancing both user satisfaction and system performance.

## introduction asking clarifying questions (cqs) plays a pivotal role in enhancing both conversational search @xcite and web search experiences @xcite . timely and high-quality questions can significantly improve system performance @xcite as well * corresponding author as overall user experience @xcite . however, the adverse effects of poorly timed @xcite or inappropriate questions can be significant, often leading to user frustration and dissatisfaction @xcite . given these challenges, optimizing the formulation of cqs has become an area of growing research interest. much research has studied the effectiveness of cqs in improved retrieval performance @xcite @xcite @xcite . for example, @xcite studies different types of cqs and their answers, such as positive or negative answers, to characterize their impact on retrieval performance. trec cast, in its latest edition in 2022 @xcite , includes mixed-initiative conversation trajectories and features an independent mixedinitiative subtask, mainly focusing on search clarification. several models are proposed in the con-vai3 challenge @xcite , aiming to incorporate cqs in the ranking process, mostly proposed based on pre-trained language models. complementing this focus, some research integrates ranking and clarification features within learning objectives @xcite , while others explore the inherent risks by gauging the prospective retrieval gains @xcite . in the information retrieval (ir) community, there is a long-standing discussion suggesting that superior system performance in terms of relevance does not necessarily result in enhanced user experience or usefulness @xcite . this has catalyzed a distinct line of research focused on comprehending the user experience with cqs @xcite @xcite @xcite . it is pertinent to note that, in this study, we categorize "useful clarifying questions" as those that lead to higher user satisfaction. specifically, we argue that users' overall satisfaction depends on a variety of facets of a triad: the query, its cqs, and the corresponding candidate answers. this perspective is motivated by a recent study @xcite ) that focuses on user satisfaction in task-oriented dialogues, emphasizing the importance of utterance relevance and efficiency. while there is existing research, such as that by @xcite and @xcite , that models user interaction and engagement with clarification panes, these studies primarily offer observational insights and have produced publicly available datasets like mimics and mimics-duo. in contrast to these studies, our focus shifts toward predicting the practical valueusefulness and user satisfaction -of cqs, based on various attributes of search queries, cqs, and their candidate answers. in summary, much of the existing research has concentrated on the quality and effectiveness of cqs in the context of retrieval gain. however, there is a noticeable gap in characterizing and predicting the real-world applicability or 'usefulness' of these questions. the concept of usefulness is intricately connected to user satisfaction, as underscored by @xcite . addressing this gap is challenging due to the multitude of factors influencing user experience beyond mere relevance @xcite . to tackle this unexplored aspect of cqs, our study aims to answer the following research questions: rq1 what features of clarifying questions help achieve higher user satisfaction? rq2 for which search queries do users prefer to use clarification? rq3 what is the impact of each feature on the usefulness prediction of clarifying questions? to this end, we conduct a comprehensive analysis and demonstrate their effectiveness in predicting question usefulness. 1 in particular, we analyze the characteristics of cqs and user queries on two widely used real-world datasets, namely, mimics @xcite and mimics-duo @xcite . the choice of using these two datasets is grounded on a recent survey @xcite , indicating that mim-ics and mimics-duo are the only two datasets allowing the evaluation of clarifying question usefulness as per user satisfaction levels. leveraging these two datasets, we conduct a comprehensive evaluation over multiple dimensions, including the template structures of cqs, the number of candidate answers available, subjectivity and sentiment polarity of cqs, the length of both cqs and queries, query ambiguity, as well as the predicted relevance between cqs and queries. to augment the evaluation of useful cqs, we further conduct a user study over a number of features, such as question naturalness. in addition, to show the benefit of the learned relationships between numerous aforementioned features and cq usefulness, we leverage the extracted features and feed them to multiple classifiers to predict cq usefulness, leading to significant performance improvement. therefore, the main contributions of our work are as follows: ## related work in this section, we discuss the existing research that pertains to the domain of asking clarifying questions (acq) in a conversational informationseeking system. although there have been previous efforts in this area, none of them has specifically examined the potential features that contribute to the usefulness of clarifying questions. therefore, we reviewed the related literature to provide a background description of our contributions in this paper. benefiting from the released public datasets with available query-clarifying question relevance labels, such as qulac @xcite and clariq @xcite , many clarifying question ranking models have been introduced @xcite . for example, in @xcite , with concatenated embeddings of posts, clarifying questions as well as optional answers from a stackexchange-based dataset @xcite as input to a multi-layer neural model, they estimate the probability of a clarifying question being relevant or not. however, due to the diverse and complex nature of clarifying questions, it is challenging to effectively address this asking clarifying question task in a retrieval manner @xcite @xcite . in particular, to enable the generation of appropriate clarifying question, a good comprehension of the queries and their likely intents is required. for example, @xcite specifically designed a query aspect modelling module as well as multiple query aspect encoders to encompass the information within queries for clarification generation effectively. so far, the existing studies illustrate the effectiveness of their generated clarifying questions by comparing to the available ground-truth @xcite , or human annotators @xcite . however, there is limited effort in exploring the aspects or features about a useful clarifying question. a similar contribution is @xcite , which evaluate the aspects of dialogues that could improve the user satisfaction level in a conversational recommendation scenario. therefore, we argue that the investigation on revealing aspects for evaluating the usefulness of clarifying questions can guide the future development of clarifying question generation. ## experimental setup in this study, we investigate numerous features that likely contribute to the usefulness of clarifying questions. in conversational informationseeking systems, users often submit diverse types of queries, ranging from statements to questions, varying in length (short or long). a clarifying question can be returned by the corresponding system to better reveal users' true information needs based on the query-as-input from the end users. intuitively, to assess the usefulness of a clarifying question, we should not rely solely on the question itself. it is crucial to jointly model both the query and the corresponding clarifying question. meanwhile, to examine user satisfaction with the presented clarifying questions, we leverage two commonly used datasets, mimics and mimics-duo, which encompass the corresponding labels. for the question-based features, we consider (1) the question template variance, (2) clarifying question presentation with a varied number of candidate answers, (3) question subjectivity, (4) sentimental polarity of questions and (5) question length. as for the query-oriented features, we investigate the impact of (6) query length in words, (7) query types (ambiguous or faceted) and ( 8 ) query-question relevance. note that partial features, such as the length of questions and the number of candidate answers, were studied in @xcite . however, these features remain underexplored when it comes to providing comprehensive insights into the usefulness of cqs. therefore, in this study, we extend the observations to the two datasets, systematically explore many other potential features and develop classifiers for the prediction as promising guidance for the future development of clarifying questions. note that, for the quantification of each feature, we detail the strategy in each of their corresponding discussions. specifically, while comparing the contributions of features, we observe a common issue of data imbalance -the number of positive queries does not equal to negative ones. to address this issue, we normalize the scores of evaluated features based on the frequency of the corresponding groups. for instance, if 60 positive labels are assigned to 100 long clarifying questions and 15 positive labels are assigned to 50 short clarifying questions, we score the long and short questions 0.6 and 0.3, respectively, for comparison. in the second part of this study, we investigate the value of the learned features from the previous step on classifying the usefulness of clarifying questions. to do so, we develop and explore numerous machine learning classifiers to estimate the usefulness of a given clarifying question. for evaluation, we partition each dataset into 80% as the training set and the rest 20% as the test set. the experimented approaches are from traditional machine learning and recent neural classifiers. for the classic approaches, we consider decision tree classifier (dtc) @xcite , random forest classifier (rfc) @xcite and support vector classifier (svc) @xcite with a linear kernel. for neural approaches, we encode the input using pre-trained language models, including: traditional machine learning models take in tf-idf weighted bag-of-word features as input, which are extracted from the text data. we implemented these models using popular libraries such as scikit-learn foot_1 @xcite , hugging-face foot_2 @xcite , and pytorch foot_3 @xcite . to assess the performance of our models, we used standard evaluation metrics for supervised classification tasks, including precision, recall, and f1 score. all of the implementations, parameters, and datasets can be found on our github repository. ## clarification usefulness in this section, we aim to answer rq1 and rq2 at first by examining various potential factors and characteristics of cqs and queries that are pertinent to the effectiveness and usefulness of a clarifying that can be deployed to shape a clarifying question for optimised performance. in the literature, zamani et al. (2020a) recently proposed to generate a majority of clarification types in a pre-existing set of question templates. in this study, to identify the most effective templates, we analyze both datasets and focus on those clarifying questions with top frequent formats. ## clarifying question usefulness prediction after exploring the correlation between available features and the usefulness of clarifying questions (cqs), in this section, we aim to answer rq3 by evaluating the effectiveness of various features for predicting cq usefulness. we consider both traditional ml and recent neural approaches discussed in section 3 for the task of cq usefulness classification, using query-question-candidate answer triplets as input on the mimics and mimics-duo datasets. to demonstrate the effectiveness of including additional cq features for cq usefulness prediction, we concatenate observed related features from section 4, including cq length, rouge-precision, sentiment polarity, and subjectivity, which are positively correlated with the clarifying question usefulness, with the original input for comparison. for the use of gpt-4 model, we carefully crafted a prompt to ask the model to generate a label-only output (good, fair or bad) with the query and clarifying question as input or with the inclusion of additional features. the corresponding prompt is provided in appendix a. we present the experimental results in we observe that across the two datasets, incorporating our proposed features leads to large improvements on the traditional, neural approaches and large language models on both mimics and mimics-duo datasets. in particular, the improvements to the traditional classifiers are significant, especially on the mimics dataset, with a minimum of 69.6% and up to 151.4% increases in f1 score. the resulting performance can also be comparable with advanced neural models. on the other hand, on the mimics-duo dataset, by comparing the performance of the traditional classifiers with and without additional features as well as the basic neural models, their classification performances are less promising, which equally gives lower than 40% of f1 scores (even the additional features can improve the basic traditional approaches with a minimum 45% increase of f1 scores). however, by incorporating the positively correlated features into the neural model, we observe a significant impact (minimum 120% improvement) on the model's performance, resulting in nearly perfect classification accuracy. meanwhile, as for the performance of the gpt-4 model, we observe that it does not perform competitively with the other two groups of approaches. the low accuracy of the gpt-4 model can be caused by its autoregressive nature of label generation, which does not guarantee a good classification outcome without fine-tuning. however, the use of additional features can still contribute to an improved performance of gpt-4, which further validates the effectiveness of using these positively correlated features. ## user study evaluation after observing promising performance improvements by including clarifying question features for usefulness estimation, we further conduct a user study to examine user opinions towards potential usefulness features by leveraging the expertise of domain experts in identifying potential relevant features for usefulness prediction. we identify eight additional features that can potentially advance usefulness prediction, divided into two groups: clarification features (i.e., naturalness, grammar, and fluency) that evaluate the text quality of a clarifying question and query-question features (i.e., coverage, novelty, efficiency, relevance, and question template) that measure if a clarifying question can effectively aid a query by addressing missing aspects, identifying novel but useful aspects, we present 50 sampled query-clarifying question-feature triplets to seven domain experts to annotate the usefulness of cqs. we then ask them to label which features are most essential for considering a cq useful. also, we ask them to select the minimum-required features for a cq to be deemed useful. we summarize and present the user study results in figure 3 (b) . the results of the study show that a high textual quality question is necessary for a cq to be considered useful, especially in terms of naturalness. additionally, among the query-question features, relevance is commonly considered an issue that needs to be addressed to present useful cq. this observation aligns with previous efforts in the literature that link query aspects with cqs to generate them effectively @xcite . another interesting finding is that coverage is one of the lowest-scored features, which also aligns with our previous consistent findings on using specific, rather than high aspect-recall clarifying questions. therefore, we conclude that the user study further highlights the value of the text quality of cqs and their relevance to queries, in addition to the features such as length, subjectivity and specificness that we previously identified as useful through experimental results. ## conclusion and future work this paper analyzed the usefulness of clarifying questions using two well-known real-world clarifying question datasets. specifically, we studied the impact of various features related to both clarification questions and the corresponding query on the usefulness of clarifying questions with respect to the level of user satisfaction. the analytical results indicate the positive impact of having specific, positively sentimental-oriented, lengthy and subjective clarification questions. by leveraging such analysis, we introduce these positively correlated features to the usefulness estimation of clarification questions. as per the classification accuracy, we observed a consistent improvement in applying the additional features, especially on the traditional approaches, with a minimum 45.7% improvement. furthermore, the performance-boosting on the neural approaches enables the classifiers to achieve a consistent, nearly perfect performance with over 94% classification precision. the results of our usefulness prediction models proved our hypothesis that incorporating different feature types would help improve the prediction by a large margin. in addition, we augment our contributions with another user study, which uses users' opinions in examining the usefulness of clarification questions from various perspectives, and we also observed close conclusions with our experimental findings. in the future, we plan to further study the impact of the features on pre-trained language models and explore various methods such as prompting large language models to generate useful and satisfying clarifying questions. furthermore, we plan to finetune open-source large language models such as llama @xcite to select the more relevant and useful clarifying questions between several questions when a model generates more than one clarifying question to clarify users' ambiguity.