Interactive Classification by Asking Informative Questions (2020)

Abstract: We study the potential for interaction in natural language classification. We add a limited form of interaction for intent classification, where users provide an initial query using natural language, and the system asks for additional information using binary or multi-choice questions. At each turn, our system decides between asking the most informative question or making the final classification pre-diction. The simplicity of the model allows for bootstrapping of the system without interaction data, instead relying on simple crowd-sourcing tasks. We evaluate our approach on two domains, showing the benefit of interaction and the advantage of learning to balance between asking additional questions and making the final prediction.

## introduction responding to natural language queries through simple, single-step classification has been studied extensively in many applications, including user intent prediction @xcite , and information retrieval @xcite . typical methods rely on a single user input to produce an output, and do not interact with the user to reduce ambiguity and improve the final prediction. for example, users may under-specify a request due to incomplete understanding of the domain; or the system may fail to correctly interpret the nuances of the input query. in both cases, a low quality decision could be mitigated by further interaction with the user. in this paper we take a low-overhead approach to add limited interaction to intent classification. our goal is two-fold: (a) study the effect of interaction on the system performance, and (b) avoid the cost and complexities of interactive data collection. we build an interactive system that poses a sequence of binary and multiple choice questions follow-what is the bill length of the bird: shorter, similar, or longer than head? shorter than head. is the bird underpart orange? the identified bird is: ## technical overview our goal is to classify a natural language query to a label through an interaction. notation we treat the classification label y, interaction question q and the user response r as random variables. we denote an assignment of a random variable using subscripts, such as @xmath0. we use superscripts for the observed value of the random variable at a given time step, for example, q t is a question asked at time step t. when clear from the context, we write y i instead of @xmath1. for example, p(r |q j , y i ) denotes the conditional distribution of r given @xmath2, and p(r k |q j , y i ) further specifies the corresponding probability when @xmath3. an interaction starts with the user providing an initial user query x. at each turn t, the system selects a question q t , to which the user responds with r t , or returns a label y to conclude the interaction. we consider two types of questions: binary and multiple choice questions. the predefined set of possible answers for a question q t is r(q t ), where r(q t ) = {yes, no} for binary questions, or a predefined set of question-specific values for multiple choice questions. we denote an interaction up to time t as x @xmath4, (q 1 , r 1 ), . . . , (q t , r t ), and the set of possible class labels as @xmath5, . . . , y n }. figure 1 shows example interactions in our two evaluation domains. ## related work human feedback has been leveraged to train natural language processing models, including for dialogue @xcite , semantic parsing @xcite @xcite ) and text classification @xcite . these methods collect user feedback after the model-predicting stage and treat user feedback as additional offline training data to improve the model. in contrast, our model leverages user interaction to increase prediction performance. human feedback has been incorporated in reinforcement learning as well, for example to learn a reward function from language as reflecting human preferences @xcite . language-based interaction has been studied in the context of visual question answering @xcite @xcite @xcite , sql generation @xcite , information retrieval @xcite and multi-turn textbased question answering @xcite @xcite . most methods require learning from recorded dialogues @xcite @xcite or conducting wizard-of-oz dialog annotations @xcite . instead, we limit the interaction to multiple-choice and binary questions. this simplification allows us to reduce the complexity of data annotation while still achieving effective interaction. our task can be viewed as an instance of the popular 20-question game (20q), which has been applied to a celebrities knowledge base @xcite . our approach differs in using natural language descriptions of classification targets, questions and answers to compute our distributions, instead of treating them as categorical or structural data. our question selection method is related to several existing methods. @xcite refine image search by asking to compare visual qualities against selected reference images, and @xcite perform object identification in an image by posing binary questions about the object or its location. both methods, as well as ours use an entropy reduction criterion to select the best next question. we use a bayesian decomposition of the joint distribution, which can be easily extended to other model-driven selection methods. @xcite propose a learning-toask approach by modeling the expected utility of asking question. our selection method can be considered as a special case when entropy is used as the utility. in contrast to @xcite , we model the entire interaction history instead of a single turn of follow-up questioning. our model is trained using crowdsourced annotations, while @xcite uses real user-user interaction data. alternatively to asking questions, @xcite and @xcite present to the user the most likely image in an image retrieval scenario. the user compares it with the ground-truth image and provides feedback using relevance score or natural language describing the discrepancy between them. ## method we maintain a probability distribution p(y |x t ) over the set of labels y. at each interaction step, we first update this belief, decide if to ask a question or return the classification output using a policy controller and, if needed, select a question to ask using information gain. ## data collection we design a crowdsourcing process to collect data for the faq task using amazon mechanical turk. 3 for the birds domain, we re-purpose an existing dataset. we collect initial queries and tags for each faq document. appendix a.1 describes the worker training process. ## experimental setup task i: faq suggestion we use the faq dataset from @xcite . the dataset contains 517 troubleshooting documents from sprint's technical website. we collect 3,831 initial queries and 118,640 tag annotations using the setup described in section 5. we split the data into 310/103/104 documents as training, development, and test sets. only the queries and tag annotations of the 310 training documents are used for pre-training and learning the policy controller, leaving the queries and tag annotations in the development and test splits for evaluation only. task ii: bird identification we use the caltech-ucsd birds dataset @xcite . the dataset contains 11,788 bird images for 200 different bird species. each bird image is annotated with a subset of 27 visual attributes and 312 attribute values pertaining to the color or shape of a particular part of the bird. we create categorical questions from attributes with less five possible values, providing eight categorical questions in total. the remaining 279 attributes are converted to binary questions. each image is annotated with 10 image captions describing the bird in the image @xcite . we use the image captions as initial user queries and bird species as labels. since each caption contains only partial information about the bird species, the data is naturally noisy and provides challenging user interactions. we do not use the images from the dataset for model training. the images are only provided for grounding during human evaluation. baselines we compare with four methods: evaluation we use human evaluation, and further analyze performance using our simulator. for human evaluation, users interact with our systems and baseline models using a web-based interactive interface. each interaction starts with a user scenario: foot_2 a bird image or a device-troubleshooting scenario described in text. the user types an initial query and answers follow-up questions selected by the system. once the system returns its prediction, we measure its accuracy, and the user is asked to rate the whole interaction according to rationality and naturalness. 5 the user does not know the correct target label. we use a five-points likert score for the followup questions. for faq suggestion, we consider two evaluation setups: (1) assuming the model has access to tags in the development and test set for interaction, and (2) using only tags in the training set annotation. the former is equivalent to adding tags for new documents not seen during training time. the latter zero-shot evaluation setup allows us to investigate the model's performance on unseen targets with no additional tags associated with them. appendix a.4 provides further details of the human evaluation setup. we do further analysis with the user simulator . we evaluate classification performance using accuracy@k, which is the percentage of time the correct target appears among the top-k predictions of the model. ## results our simulated analysis shows that the sru rnn text encoder performs better or similar to the other encoders. this encoder is also the most lightweight. therefore, we use it for the majority of our experiments. human evaluation figure 2 and ## conclusion we propose an approach for interactive classification, where the system can inquire missing information through a sequence of simple binary or multi-choice questions when users provide underspecified natural language queries. our expertguided, incremental design of questions and answers enables easy extension to add new classes, striking the balance between simplicity and extendability. our modeling choices enable the system to perform zero-shot generalization to unseen classification targets and questions. our method uses information gain to select the best question to ask at every turn, and a lightweight policy to efficiently control the interaction. we demonstrate that the system can be bootstrapped without any interaction data and show effectiveness on two tasks. a potential future research direction is to bridge the gap between this simple bootstrapping paradigm and the incorporation of user free-form responses to allow the system to handle free-text responses. we hope our work will encourage more research on different possibilities of building interactive systems that do not necessarily require handling full-fledged dialogue, but still benefit from user interaction.