V oi S e R : A New Benchmark for Voice-Based Search Refinement (2021)

Abstract: Voice assistants, e.g., Alexa or Google Assistant, have dramatically improved in recent years. Supporting voice-based search, exploration, and refinement are fundamental tasks for voice assistants, and remain an open challenge. For example, when using voice to search an online shopping site, a user often needs to refine their search by some aspect or facet. This common user intent is usually available through a “filter-by” interface on online shopping websites, but is challenging to support naturally via voice, as the intent of refinements must be interpreted in the context of the original search, the initial results, and the available product catalogue facets. To our knowledge, no benchmark dataset exists for training or validating such contextual search understanding models. To bridge this gap, we introduce the first large-scale dataset of voice-based search refinements, VoiSeR, consisting of about 10,000 search refinement utterances, collected using a novel crowdsourcing task. These utterances are intended to refine a previous search, with respect to a search facet or attribute (e.g., brand, color, review rating, etc.), and are manually annotated with the specific intent. This paper reports qualitative and empirical insights into the most common and challenging types of refinements that a voice-based conversational search system must support. As we show, VoiSeR can support research in conversational query understanding, contextual user intent prediction, and other conversational search topics to facilitate the development of conversational search systems.

## introduction modern voice assistants, such as amazon alexa or apple siri, make use of natural language understanding (nlu) techniques to perform several tasks. some of the most popular functions offered by these systems are based on voice-search: millions use voice assistants to access information or search for music, products or local restaurants and stores. however, search experience with a voice assistant remains limited. the current generation of these systems mostly supports single-turn interactions, and does not naturally support more complex search needs, which often require refinements to narrow, broaden or change the initial search. supporting refinement is a fundamental aspect of search systems, and it is done in a variety of ways in web-based user interfaces, e.g., through query suggestion or explicit facets navigation or filtering. for example, in an e-commerce search, a user may want to refine their search with respect to some facet or attribute (e.g., brand or price); this critical functionality is supported on most e-commerce websites. however, this kind of interaction is challenging to support via voice-based dialogue interfaces, as interpreting such refinements requires modeling the original search intent, the initial results, and the available result facets. to the best of our knowledge, no large scale dataset exists for training and validating nlu models for multi-turn voice-based search. to bridge this gap, we present a new voice-based search refinement dataset, voiser 1 to enable research into contextual understanding of multi-turn voice-based search. the voiser dataset contains 9,810 utterances of voice refinements in the e-commerce domain, paired with contextual information. specifically, our dataset includes refinement utterances intended to filter results from a previous search with respect to some facet or attribute (e.g., brand, color, review rating, etc.). the dataset was collected through crowdsourcing via amazon mechanical turk, between february and june 2020 in the us and india. we designed the task to minimize any bias towards partic1 the dataset is available for download at https://registry.opendata.aws/ amazon-conversational-product-search/ . we annotated the dataset to highlight some important aspects characterizing a voice refinement of product search. in particular, we annotated (i) the products and attributes mentioned in each utterance, if present; (ii) the specific refinement intent of each utterance (e.g., refinement by exact attribute value). in addition to the new voiser benchmark dataset, our contributions include (i) an analysis of the data, where we highlight some linguistic aspects characterizing how people express the refinement intent, and (ii) an empirical investigation to demonstrates that voiser can be successfully used to bootstrap nlu models for handling voicebased search refinements. furthermore, we show that contextual information is beneficial for such nlu tasks. next, §2 provides details about the data collection and annotation. §3 provides a detailed analysis of the dataset, while §4 reports the empirical investigation. in §5, we discuss the related works. finally, §6 discusses our conclusions. ## voiser data collection in order to collect a large number of voice search refinements from multiple participants, we designed a crowdsourcing task on amazon mechanical turk foot_0 . in §2.1, we provide details of the crowdsourcing experiment to collect refinement utterances. these utterances were annotated for intent and relevance, as described in §2.2. ## voice refinements dataset in this section, we provide the analysis of the data collected through the crowdsourcing experiment. first, we describe how we conducted the mechanical turk experiment and the statistics of the collected data in §3.1; then, we discuss some of the linguistic properties emerging in the context of voice refinements in §3.2. ## empirical investigation in this section we present a set of experiments on intent detection, specifically the recognition of attribute and product categories in a search refinement utterance. we aim to show that the voiser dataset enables building models for intent recognition in voice search refinement. moreover, we show the contribution of contextual information, e.g., the previous utterance, is beneficial, highlighting the need for large scale datasets like voiser for developing contextual intent recognition models. experimental setup. in order to train and test the model, we split the data into train, validation and test portions, as reported in model. we model the recognition of attribute and product mentions in a refinement utterance as a sequence tagging problem in an iob2 foot_3 tagging schema. for example, given the utterance "show me a vega brand hair curler", its correct tagging is "o o o b-attribute i-attribute b-product i-product". notice that we do not tag the attribute type, but only a general category attribute. this is to enable the model to generalize over the linguistic expressions that are similar among different attribute types. the model we implemented is a bert-based @xcite sequence tagger. bert is used as the encoder to obtain the contextualized embeddings of each token w i of a sentence, i.e., h @xmath0. after applying dropout on h i , a linear classifier is used to obtain the c i distribution over the iob categories for each token w i , i.e., c @xmath1, where softmax refers to the function to transform the scores in output probabilities and w and b are the weights and bias of the classifier, respectively. the model is trained by optimizing the cross entropy between the predicted categories and the true one-hot distributions for each token. we run experiments on two settings: (i) w/o context, i.e., the input to the model is only the refinement utterance; (ii) w/ context, i.e., the model receives as input both the original search query and the refinement utterance. the two utterances are separated by the [sep] token, as usual in bert @xcite . the latter experiment will provide insights about the utility of contextual information to the model. ## related work human-computer information retrieval (hcir) @xcite combines the fields of humancomputer interaction (hci) and information retrieval (ir) to create systems that improve search by taking into account the human context, or through a multi-step search process that provides the opportunity for human feedback. modern search engines implement several hcir strategies including relevance feedback @xcite , automatic query completion @xcite and faceted search @xcite . in particular, most of the e-commerce websites provide faceted search by allowing users to filter on several product attributes. with our work we would like to study how a similar experience can be made available in a voice-based interaction. due to the increasing availability of smartphones and voice assistants like amazon alexa or google home, voice-based search is becoming ubiquitous @xcite . voice-based web search is significantly different from text-based search @xcite , notably for voice-based query reformulation or refinement @xcite . the voice-based interaction is especially challenging when no visual interface is available and both inputs and outputs are entirely provided by voice @xcite . to improve user experience and engagement in voice-search some work has been recently done. @xcite propose a query rewriting approach for handling mispronounced, misexpressed, and misunderstood customer queries in voice shopping. @xcite describe the problem of reformulating answers from a community question answering system to make them suitable for a voice interaction. there have been prior efforts in creating open multi-turn voice-based search datasets, but because of the lack of effective automated systems for these tasks, the datasets were collected in a lab using a wizard-of-oz approach, where a hidden human participant playing the part of the search engine, e.g., @xcite @xcite , and @xcite . the dataset we contribute complements prior efforts by providing a large-scale collection of voice-based query reformulation, collected in a realistic environment with a real search engine, using automated asr, thus providing a critical resource for training robust, high-performance models for voice refinement. recently, several dialog-related datasets have been @xcite proposed. they target task-oriented conversations but do not specifically focus on search tasks. ## conclusions in this paper, we discussed the problem of search refinement, a fundamental component for supporting multi-turn voice-based complex search tasks. we presented the challenges in the voice refinement problem, and introduced a large-scale, critically needed benchmark for training and evaluating models in this setting. specifically, we introduced the first benchmark dataset, voiser, specifically developed for analyzing and measuring the linguistic phenomena underlying the voice refinements in second-turn searches in the e-commerce domain. we emphasize that the target search facets and attributes are (by design) general, and thus the data and the resulting models can be used, with or without adaptation, for a wide range of conversational search refinement and intent prediction tasks. we provided a detailed description of the data collection and annotation processes, and identified interesting statistical and linguistic phenomena in the dataset. we complement the data release with an extensive empirical investigation, which demonstrates that (i) our dataset is a valuable resource for training nlu models for voice-based search and (ii) using contextual information for recognizing product and attribute mentions is beneficial. together, the new voiser dataset and the analysis in this paper enable productive research for developing systems for voice-based complex search tasks.