Combining Structured and Unstructured Knowledge in an Interactive Search Dialogue System (2022)

Abstract: Users of interactive search dialogue systems specify their preferences with natural language utterances. However, a schema-driven system is limited to handling the preferences that correspond to the predefined database content. In this work, we present a methodology for extending a schema-driven interactive search dialogue system with the ability to handle unconstrained user preferences. Using unsupervised semantic similarity metrics and the text snippets associated with the search items, the system identifies suitable items for the user’s unconstrained natural language query. In crowd-sourced evaluation, the users chat with our extended restaurant search system. Based on objective metrics and subjective user ratings, we demonstrate the feasibility of using an unsupervised low latency approach to extend a schema-driven search dialogue system to handle unconstrained user preferences.

## introduction we extend a schema-driven dialogue system with the ability to handle unconstrained user queries and to allow users to specify preferences flexibly as they would when using a search engine. interactive search dialogue systems, such as search for restaurants, hotels, trains, books, shows, venues, are task-oriented systems that provide a natural language interface for interactive search and information extraction. in these systems, a user typically starts by typing (or speaking) a search query. next, the system's policy chooses an optimal action, which may be either asking the user to provide additional information or presenting one or more search result options. once the system presents an option, the user may provide another query, narrowing down or changing the search criteria, or ask a question about the presented option(s). in a schema-guided approach to designing a dialogue interface @xcite , a set of 'informable' and 'requestable' slots derived from the fields of the underlying database table (or schema), define the natural language interface capability. a user can specify the values of 'informable' fields as search criteria and ask questions to retrieve information stored in the 'requestable' fields. the schema-guided method simplifies authoring dialogue systems for new domains. with this approach, a dialogue interface for a new database may be bootstrapped from the schema/ontology and database content of the domain. 1 one of the drawbacks of the schema-guided approach is that the criteria by which the user can search for an item and the types of questions that the user may ask are limited by the database schema. for example, a restaurant search query 'find a romantic place that serves great wines' cannot be handled by a schema-driven system unless the schema includes the relevant properties of the restaurant atmosphere and wine quality. it is possible to design a system to notify the user of its limitations using help messages @xcite , but the constraint on the interaction remains, as the user is unable to retrieve items using criteria other than those defined by the 'informable' slots. given that in many domains, additional unstructured information beyond the database fields may be available, it is natural to extend schema-guided systems to use this unstructured information. @xcite describe a system that extends the schema-guided functionality with the ability to ask follow-up questions. in this work, we propose to extend the information search dialogue interface functionality to retrieve items for unconstrained user queries. to handle user queries that cannot be grounded in terms of a domain schema and ontology, we propose to use semantic similarity metrics to retrieve search results from unstructured data. we evaluate the proposed approach with crowd-sourced users interacting with the restaurant search system through a chat interface. in previous work, restaurant search systems were evaluated by giving users predefined 'goals' which primed users and lead to rigid interactions. in our evaluation, the users are given a general instruction to find an ideal restaurant and are free to specify any search query. the results show the users' preference for the proposed flexible system that allows the use of unconstrained search queries. we release the dialogues with the automatically annotated intents and the subjective user judgements collected during the evaluation to the research community. foot_1 ## related work interactive search can be modelled as task-oriented dialogue using structured knowledge, symbolic dialogue state representation, and a statistical policy that addresses both task and conversational phenomena, such as clarifications and social dialogue acts @xcite . however, users of dialogue systems that are based only on structured knowledge are limited in expressing their preferences by the underlying database schema. in response to an out-of-schema user request, a task-oriented dialogue system may produce an informative help message guiding the user to adapt to its limitations @xcite . alternatively, system capabilities may be extended beyond a domain api. for example, @xcite proposes a method for handling user's follow-up questions in task-oriented dialogue systems. to support pragmatic interpretation, @xcite explores users' indirect responses to questions. to extend a task-oriented system to handle natural preferences, a corpus of natural requests for movie preferences was collected using preference elicitation @xcite . end-to-end approaches to dialogue, where the system generates a response without explicitly modelling intent or storing a dialogue state, have been successfully applied to open-domain chitchat @xcite . the use of unstructured knowledge was shown to improve open domain chitchat systems @xcite @xcite . in recent work, interactive search has been modelled as end-to-end gener-ation task using text and images as the knowledge source @xcite . search tasks in natural human-human dialogues can be complex and are often resolved interactively @xcite , motivating the need for methods capable of handling natural conversational phenomena as well as extracting information and generating knowledge-grounded responses. in this work, we evaluate a task-oriented dialogue system with a semantic-level policy extended with the use of unstructured knowledge. task-oriented dialogue systems require accurate models to extract information from unstructured text. pretrained transformer models, such as bert @xcite , have shown to be effective in extracting information from text, leading to significant improvements on many nlp tasks, including open-domain question answering, faq retrieval, and dialogue generation @xcite @xcite . following previous work, we use bert in an unsupervised setting to extract relevant information from text @xcite . ## evaluation our goal is to evaluate the use of unstructured data in an interactive task-oriented dialogue system. the proposed approach involves two statistical models: 1) intent classifier and 2) relevance ranker, which accesses unstructured data, depending on the prediction of the intent classifier. we first show the performance of the intent classifier on the collected dataset and then describe the human evaluation of the overall system. ## conclusions in this work, we propose a hybrid design for information navigation dialogue systems combining structured and unstructured knowledge. we present a restaurant search dialogue system where the users specify preferences flexibly as they would to a search engine. the proposed system uses the structured knowledge in a database to extract matching restaurants when a user's natural language search query mentions one of the database fields and unstructured text when it does not. the system is evaluated in the interactive experiments with crowdsourced users. the results show a preference for the proposed approach. in future work, we will extend the system to answer follow-up questions and introduce a response generation model.