User Impressions of Questions to Acquire Lexical Knowledge (2020)

Abstract: For the acquisition of knowledge through dialogues, it is crucial for systems to ask questions that do not diminish the user’s willingness to talk, i.e., that do not degrade the user’s impression. This paper reports the results of our analysis on how user impression changes depending on the types of questions to acquire lexical knowledge, that is, explicit and implicit questions, and the correctness of the content of the questions. We also analyzed how sequences of the same type of questions affect user impression. User impression scores were collected from 104 participants recruited via crowdsourcing and then regression analysis was conducted. The results demonstrate that implicit questions give a good impression when their content is correct, but a bad impression otherwise. We also found that consecutive explicit questions are more annoying than implicit ones when the content of the questions is correct. Our findings reveal helpful insights for creating a strategy to avoid user impression deterioration during knowledge acquisition.

## introduction structured knowledge bases are not only crucial for providing various services such as information search and recommendations but also effective for non-task-oriented dialogue systems to avoid generic or dull responses @xcite @xcite . however, it is impractical to presuppose a perfect knowledge base @xcite in an early stage of system development. therefore, being able to acquire knowledge from users and thereby enhance knowledge bases through dialogues is one of the most important abilities that dialogue systems should possess. although knowledge acquisition can be done by asking people to input information on guis or spreadsheets, knowledge acquisition through dialogues good indonesian restaurants have opened around here recently. i have been there! "nasi goreng" belongs to "indonesian" it seems indonesian … step 1 estimate an attribute of w i will try to cook nasi goreng today. ## user study design we assume a system that obtains an attribute value for an unknown term. that is, when an unknown term appears in a dialogue, we try to make the system acquire its attribute from the user through the dialogue. a pair consisting of the term and its attribute can then be stored as new system knowledge. more specifically, we assume the pair of an unknown food name and its cuisine. first, the cuisine of a food name is estimated from its character sequence @xcite , and next, the estimated cuisine is verified by asking either form of question. we focus here on the types of questions for verifying the estimated cuisine. ## analysis with linear regression we analyzed the effect of each question type by using the coefficients of a linear regression model that predicts the collected impression scores. first, we describe the basic regression model and its refinement to make the multiple correlation coefficients (r) higher. after that, we discuss the effect of each question type on user impression and analyze results when the same question types were repeated. ## conclusion we investigated how five types of question affect user impression and clarified the order of the ques- user the panna cotta was very sweet and good. system is panna cotta italian? user yes. it's italian. system i understand. thank you. user the pad thai was refreshing and delicious. system is pad thai a thai food? user that's right. it's a thai food. system i understand. thank you. user i ate a bagna cauda. it was similar to shabu-shabu. system is bagna cauda italian? user yes. it is an italian pot cuisine. system i understand. thank you. tion types in terms of two impression scores: intelligent and annoying. we also found that repeating the same question type annoys the user and degrades user impression even when the content is correct. the implicit questions tend to give a better user impression than the explicit ones as long as the content is correct. on the other hand, the impression by the implicit questions with wrong content is worse. these results justify an adaptive strategy using a confidence measure of the cuisine estimation in order to lower the risk of user impression deterioration. specifically, it is reasonable that the system should select an implicit question if the confidence is high, and select an explicit one (or a whquestion) if the confidence score is low. this is consistent with results discussed for task-oriented dialogues using confidence measures of automatic speech recognition, such as @xcite . a new finding here, based on the results of our analysis in section 4.3, is that the designer of the dialogue system also needs to avoid repeating the same type of questions in non-task-oriented dialogues. the system should have multiple choices of question types in order to prevent users from becoming irritated. that is, it is necessary to change question types appropriately by considering not only the confidence of the estimation but also the history of the dialogue. this will help the dialogue to continue with less degradation of the user's impression and enable the system to acquire knowledge through dialogues. several issues remain as future work. our experiment was limited in terms of the number of turns and the domain where it was tested. the results need to be verified with non-task-oriented systems that can engage in longer dialogues in various domains. we are planning to implement user gelato looks gorgeous. system ice confections are perfect for eating on a hot day. user that's right. it's refreshing. system i think so too. user i ate paella for dinner. system many japanese people like spanish food. user japanese people like such taste. system i think so too. user ushiojiru is a nice, tasty soup. system many japanese dishes are lightly seasoned. user that's right. they're so good. system i think so too. a non-task-oriented dialogue system that has the function to acquire knowledge. the subdialogue shown in this paper can be embedded within a longer dialogue. the implicit confirmation can be implemented by preparing the expressions of implicit questions for each category (cuisine type, in this paper) to be estimated. a further user study will be conducted with the implemented system. another issue is that answers from users may be different; e.g., some users may say that "mapo doufu" is sichuan, but others may say it is chinese. this is caused by the different concept granularity of individual users, which appears in the answers. a knowledge graph that can have different nodes representing the both concepts may be a possible solution for this issue. incorporating the utility of each question type for acquiring knowledge @xcite would be another interesting extension of the strategy.