Bootstrapping a Music Voice Assistant with Weak Supervision (2021)

Abstract: One of the first building blocks to create a voice assistant relates to the task of tagging entities or attributes in user queries. This can be particularly challenging when entities are in the tenth of millions, as is the case of e.g. music catalogs. Training slot tagging models at an industrial scale requires large quantities of accurately labeled user queries, which are often hard and costly to gather. On the other hand, voice assistants typically collect plenty of unlabeled queries that often remain unexploited. This paper presents a weakly-supervised methodology to label large amounts of voice query logs, enhanced with a manual filtering step. Our experimental evaluations show that slot tagging models trained on weakly-supervised data outperform models trained on hand-annotated or synthetic data, at a lower cost. Further, manual filtering of weakly-supervised data leads to a very significant reduction in Sentence Error Rate, while allowing us to drastically reduce human curation efforts from weeks to hours, with respect to hand-annotation of queries. The method is applied to successfully bootstrap a slot tagging system for a major music streaming service that currently serves several tens of thousands of daily voice queries.

## introduction music listening is among the top-5 reasons of daily usage of voice assistants in the us. 1 users can have different goals when formulating a musicrelated query to their home voice assistant or mobile phones. for instance, users may look for a specific entity, which can be either explicit (e.g., "play led zeppelin") or implicit (e.g., "play the latest album by foo fighters"). they may also ask queries without having a specific entity in mind (e.g., "play some reggae music"), or make open-ended requests like "play something that i like" @xcite . given a transcribed voice query, a fundamental task towards its understanding is to identify entities and musical attributes in it. however, this can be a non-trivial task, especially when the catalog is composed of possibly millions of different entities. in such situations, the chances that the name of one entity will overlap even partially with another entity are non-negligible. it is even more likely to find overlaps between entities and musical attributes, or between entities and other commonly-used natural language phrases in the query @xcite . for example, the word "happy" is at the same time a song by pharrell williams and an attribute belonging to the "mood" category in our taxonomy of musical attributes. mislabeling entities in a user query can potentially lead to awkward user experiences. slot tagging, or slot filling, is traditionally tackled as a supervised sequence labeling problem and it is often based on methods such as recurrent neural networks @xcite , conditional random fields @xcite or pretrained language models like bert @xcite . in real-world industrial applications, however, the choice and optimization of the machine learning architecture is just the tip of the iceberg. most of the time and cost are actually spent in gathering sufficient accurately-labeled training data. this process generally requires the manual annotation of up to millions of user queries, a process that should be routinely repeated to keep up with natural drifts in user queries due to, e.g., new interests from users or items that are added or removed from the catalog of searchable products. manual annotation can be complemented, or even replaced, with synthetically generated training data based on patterns curated by experts @xcite . while synthetic generation unlocks the possibility of gathering nearly infinite labeled training sam-ples, it still requires solid domain expertise to create a sufficiently rich set of patterns to cover as many query variations as possible. both manual annotation and generation of training data requiring a significant financial and human resources; another line of thoughts is to exploit unlabeled query data, which is generally cheap and abundant, and to label it via weak supervision. weak supervision -or distant supervision-has demonstrated its suitability to a number of natural language processing tasks such as relation extraction @xcite or entity recognition @xcite . moreover, it has been shown as a useful method to bootstrap conversational systems, being applied to intent detection @xcite or slot tagging @xcite tasks. given this success, flexible frameworks like snorkel @xcite have been created to help on building weak supervision pipelines at scale. however, these frameworks are not easily adaptable to sequence labeling tasks @xcite . in this paper, we present our own methodology inspired by weak supervision to label large sets of transcribed voice queries with entities and attributes from a catalog with millions of entries. the resulting labels are, albeit noisy, sufficiently accurate to be used for training slot tagging models. we show how our methodology allows us to control the amount of noisy labels injected in the training dataset by combining weak supervision and human filtering, and provide experimental evidence of how it was exploited to successfully bootstrap a slot tagging system that now serves tens of thousands of voice queries every day in a major music streaming platform. it is worth noticing that the proposed methodology, while defined and tested specifically for the music domain, is generic enough to be applied to other voice search applications that face similar challenges, like e.g. video on-demand @xcite or online shopping. ## training data creation methodology starting with large amounts of unlabeled voice query transcripts, foot_0 we automatically label selected terms with respect to both a set of music entities (i.e., artists, albums and tracks) and to attributes from an in-house taxonomy of musical attributes (e.g., genres, instruments, moods, etc.). some of these annotated queries are then discarded, while the remainder are selected for training purposes (see section 3). this methodology requires the following basic components: there are two main steps to our methodology. first, a heuristic labeling function makes use of corpus statistics and string matching rules to fullyautomatically label queries, while discarding some queries whose annotations cannot be established with sufficient confidence. then, query patterns are extracted from this first set of labeled queries, and leveraged in a human filtering task where erroneously-labeled queries are discarded. ## experimental setup we evaluate different quantities of weaklysupervised labeled queries as training data for slot tagging models, from a few thousands up to millions, on a sample of actual voice traffic collected from our application. with the goal of showing the potential of weakly-annotated training data, we compare it to models trained on a dataset of humanannotated queries and different synthetically generated datasets. all datasets are described in ## results and discussion the test set presents a representative sample of the queries that are effectively asked by users using our voice assistant and hence provide valuable insights of how the system will likely perform online. the ser obtained by the bilstm-crf model trained on each of the training datasets is shown in we can immediately notice that the man baseline is outperformed by all the syn, ws and ws(f) variants that we tested. the improvement brought by weak supervision over the baseline grows noticeably with size. when training using small datasets, syn s outperforms man and ws s . however, the ser of models trained on syn data increases with training set size. we hypothesize this behavior to be due to an amplification of the bias coming from the relatively small amount of patterns used in data generation. indeed, in our experimentation setup, we are focusing on a relatively small amount of human-labeled queries. one way of overcoming this issue would be to add more hand-curated queries or patterns -with the corresponding implications in terms of cost and time. further, research should be also be dedicated to evaluate different ways to generate synthetic data. baselines are in italic. the best test ser overall is highlighted in bold. in opposition, the weakly-supervised ws datasets are able to feed the network with a set of query patterns and annotations whose variety naturally grows with size. this characteristic turns out to be extremely beneficial, as evidenced by the incremental reduction in ser with dataset size. when considering the largest datasets, the models trained on the ws xl dataset show a reduction in ser of 9.7 w.r. we can however notice that the reduction in ser slows down as size grows. one possible explanation can be that the inherent noise in the data. as shown in moreover, the curation task followed to create ws(f) is much faster than query annotation, not only because the number of words to review is smaller than the number of raw queries, but also because the task itself is easier. we measured that an expert trained annotator is able to manually annotate approximately 100 queries per hour, whereas the pattern vocabulary can be curated following our methodology in less than 4 hours. therefore, annotating 7000 queries by two annotators took approximately 140 hours, which is orders of magnitude more than the time needed for the vocabulary curation proposed here. ## summary and future work this paper presents a methodology to create training data for training slot tagging models inspired by weak supervision. the methodology consists of two steps. first, a simple heuristic query labeling process is applied, that leverages corpus statistics obtained from query logs and comparing them with entity popularity metrics. second, a pattern extraction and filtering process is applied to the labeled queries, that makes use of human curation. our experimental evaluation clearly shows the value of weak supervision for building training datasets to bootstrap slot tagging models. we show that training with large amounts of weaklysupervised data generated from unlabeled voice queries using the proposed methodology outperforms smaller yet reasonable amounts of handannotated data. it also outperforms training with large amounts of synthetic data generated from the same hand-annotated data. we showed that the proposed methodology can be combined with a much less time-consuming word vocabulary curation task with very significant reduction in the end model sentence error rate. future work should include experimenting with other manual curation tasks, such as manual cleaning of remaining patterns after vocabulary curation, which could help to increase even more model accu-racy with a task still much faster than manual annotation of queries. in addition, further study should focus on how synthetic generated queries can complement weakly-supervised datasets, which may help not only in terms of accuracy, but also to assess the quality of labeling of less frequently queried entities or attributes. moreover, other model architectures could be experimented with, such as pre-trained language models. finally, experiments could evaluate the applicability of our weakly-supervised methodology to other domains with very large catalogs of entities.