Improving Spoken Language Understanding by Wisdom of Crowds (2020)

Abstract: Spoken language understanding (SLU), which converts user requests in natural language to machine-interpretable expressions, is becoming an essential task. The lack of training data is an important problem, especially for new system tasks, because existing SLU systems are based on statistical approaches. In this paper, we proposed to use two sources of the “wisdom of crowds,” crowdsourcing and knowledge community website, for improving the SLU system. We firstly collected paraphrasing variations for new system tasks through crowdsourcing as seed data, and then augmented them using similar questions from a knowledge community website. We investigated the effects of the proposed data augmentation method in SLU task, even with small seed data. In particular, the proposed architecture augmented more than 120,000 samples to improve SLU accuracies.

## introduction recent advances in speech applications running on smartphones and smart speakers increase the importance of spoken language understanding (slu). slu is a task to predict an appropriate system function with its arguments, given a user request written or spoken in natural language. various slu benchmarks have been proposed: air travel information services (atis) @xcite , restaurant information navigation @xcite , and other speech applications @xcite . adaptation of slu to newly defined tasks is an important problem @xcite . the number of training data directly affects the slu accuracy because most of the recent slu systems are based on statistical machine learning approaches. some existing work tackled this problem based on transfer learning @xcite , which uses a pre-trained model on different domain data. however, it is still challenging to make the slu accurate with no or fewer data. data augmentation has been applied to solve this problem, which generates pseudo training samples @xcite . however, such methods often generate unnatural training samples that will decrease slu accuracy. another problem is the ambiguity of user utterances; it is difficult to generate such ambiguous examples with generative approaches. some existing work tackled this problem by using paraphrasing models @xcite . on the other hand, collecting text data from the web is a widely used approach to building language models of automatic speech recognition systems @xcite @xcite . web texts are expected to be more natural than generated pseudo sentences because users handcraft most of them. however, web texts contain diverse domain texts; thus, we need some criteria to select appropriate texts to be used for the augmented training data. test-set perplexity @xcite or semantic similarity @xcite were widely used as criteria to select appropriate sentences for the training data augmentation. such a selective approach using large-scale web data has been applied to the data augmentation of dialogue systems @xcite . crowdsourcing is a common way to collect human-annotated data at low-cost @xcite . however, accurate slu systems based on neural networks require a large-scale dataset. it is not easy to collect sufficient amount of training data only using crowdsourcing even if the cost of crowdsourcing is less than normal annotators. in this paper, we utilize two sources of the "wisdom of crowd" for collecting a large-scale dataset to train the slu system. we firstly collect a small amount of seed data by using crowdsourcing and then augment the dataset with similar texts extracted from the web. as the target web texts for the extraction, we focus on the online knowledge community website as another "wisdom of crowds." online knowledge community websites often contain qualified question-style sentences. we choose sentences similar to the seed data from the qualified sentences and use them for slu training. we conducted experiments to investigate the relationship between the accuracies of slu systems and their training dataset augmented by sentences from the knowledge community website. as the result, over 120,000 sentences were extracted from the web, and we got 35 points improvement in accuracy on domain selection of slu. ## data augmentation using online knowledge community website crowdsourcing is a promising way to collect initial data; however, it costs linearly to the amount of data. it is not realistic to collect all the data using crowdsourcing. thus, we propose to use the crowdsourced data as a seed data to augment the dataset. another "wisdom of crowds," knowledge community website has the potential to be the augmented training data, because it consists of questions and answers in spoken language style, which are similar to user queries. it is reported that such website data is useful for dialogue systems @xcite ; we expect that the data also will be useful for slu. we calculate similarities between any seed queries and any question sentences extracted from the knowledge community website for finding the best alignment between augmented sentences and user intents. the ith user intent f i has corresponding q i,j (1 ≤ @xmath0, which is a possible query collected in crowdsourcing (j is the number of queries assigned to intent f i ). we calculate similarities between each q i,j and c k , which is a question sentence extracted from the knowledge community website, for finding similar sentence ĉk to q i,j . ĉk which will be assigned as an agumented training sample of f i . converting sentences to vector representations is an common approach to calculate similarities: vector space model @xcite , means of distributed representation of words @xcite and bi-directional long short-term memory neural networks (bi-lstm) @xcite . recently, bidirectional encoder representations from transformers (bert) @xcite is known as a better sentence encoder, which is based on masked word prediction in surrounding sentences. we used the bert model trained using japanese wikipedia @xcite on the task of masked word prediction because we would like to extract semantically similar sentences to seed queries. the task of masked word prediction is based on the distributional hypothesis @xcite ; thus, the resultant model trained in the task can embed semantically similar sentences into close points on the latent space. we note the vector of sentence q i,j as q i,j . because both vectors q i,j and c k have the same vector size, we define their similarity as the cosine between them as, we investigated the effect of each data augmentation method in experiments in this section. we describe the slu system that we used, experimental setting, and the results. ## conclusion in this paper, we proposed to use two sources of the "wisdom of crowds," crowdsourcing and knowledge community website, to augment the training data for slu. we used bert to calculate similarities between seed queries and queries extracted from a knowledge community website, as a converter from utterances to vectors. experimental results showed that using both crowdsourcing and knowledge community website for data augmentation will improve the accuracy and robustness of the slu system at low-cost. the proposed architecture is evaluated only on written texts collected by crowdsourcing; thus, experimental evaluation with real recognized speech is essential for future work. in recent end-to-end slus, acoustic features are also important; however, it is difficult to collect such acoustic features from web texts. one way to solve the problem is using a machine speech chain @xcite , which can generate pseudo acoustic features for the augmented query texts. disambiguation to queries that have several meanings, and using other embedding methods such as roberta @xcite will be another future work.