T RANSFERRING SLU M ODELS IN N OVEL D OMAINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Spoken language understanding (SLU) is a critical component in building dialogue systems.
When building models for novel natural language domains, a major challenge is the lack of data in the new domains, no matter whether the data is annotated or not.
Recognizing and annotating “intent” and “slot” of natural languages is a time-consuming process.
Therefore, spoken language understanding in low resource domains remains a crucial problem to address.
In this paper, we address this problem by proposing a transfer-learning method, whereby an SLU model is transferred to a novel but data-poor domain via a deep neural network framework.
We also introduce meta-learning in our work to bridge the semantic relations between seen and unseen data, allowing new intents to be recognized and new slots to be ﬁlled with much lower new training effort.
We show the performance improvement with extensive experimental results for spoken language understanding in low resource domains.
We show that our method can also handle novel intent recognition and slot-ﬁlling tasks.
Our methodology provides a feasible solution for alleviating data shortages in spoken language understanding.
I NTRODUCTION
The recent surge of artiﬁcial intelligence motivates the technical and applicable exploration of novel human-computer interactions.
Spoken dialogue systems are widely studied and used in various mobile devices.
Well-known commercial applications driven by dialogue systems include intelligent personal assistants and intelligent robots for customer services.
As more and more products and scenarios integrate dialogue systems in intelligent services, the ability for model adaptation in dialogue systems is critically needed.
Many state-of-the art dialogue systems follow a learning pipeline that includes components such as spoken language understanding (SLU), dialogue management (DM) and spoken language generation or retrieval (SLG).
Since both dialogue management and spoken language generation systems rely on knowledge or information learned from spoken language understanding, research and industry community have paid much attention to the SLU area.
In this paper, we focus on the problem of data shortage for SLU in a new domain.
Spoken language understanding typically deﬁnes and represents user utterances in terms of semantic frames comprised of domains, intents and slots (Tur & Mori, 2011).
A spoken language understanding task involves classifying the domain, detecting the intent of user utterance and identifying token sequences corresponding to slot values in the semantic frame.
A learning model is often designed to handle the above subtasks separately and sometimes simultaneously.
Deep learning models are the cutting-edge techniques for achieving impressive performance in spoken language understanding compared with conventional machine learning models.
However, deep neural-network models often require the collection and manual annotation of data, which is a time and labor intensive process.
These problems pose as a major obstacle to building accurate learning models in new domains.
To address this problem, we propose to perform domain adaption of SLU models by transferring the SLU knowledge and model parameters learned in an auxiliary domain where a model is already built.
A related problem is that only few examples of some output intent and slot classes are available during the training phrase in a new domain, because training sets for a new domain are small.
To deal with this limitation, we propose a novel few shot learning model in a meta learning paradigm, we call it the Transfer Semantic Similarity Model (TSSM).
The TSSM achieves few shot learning by exploiting the semantic embeddings of both seen and unseen intent and slots.
In the related work section, we argue that the current spoken language understanding tasks often do not take the relationship between intents and slots into consideration, even when these works classify the intents and slots in a novel SLU domain simultaneously in a single model.
We propose a “intent2slot” mechanism which consider such relationship and show that the TSSM can indeed achieve better performance in natural language understanding.
In summary, the challenges of spoken language understanding include:
• Expanding SLUs to novel domains is necessary as the development of artiﬁcial intelligent products extends to more areas in our society.
But it is difﬁcult to get sufﬁcient data or spoken language sentence in novel domains.
• Without sufﬁcient labeled data, deep learning methods are difﬁcult to train and thus it is hard to obtain satisfactory performance in a new domain.
• Unseen intents and slots exist because small annotated training set might not cover all intents and slots.
Thus, the generalization ability of the models are crucial for dealing with unseen classes.
Our contributions are summarized as follows:
• We design a novel and efﬁcient SLU model, the TSSM.
It incorporates transfer learning and meta learning for SLU tasks in low resource domains.
• The proposed TSSM can improve model performance with only a small amount of training data in new domains and it can handle unseen classes easily.
The experiments illustrate that our proposed methods outperform the state-of-the-art baselines in many experimental domains.
• Our work can treat intent and slot detection as a structured multi-objective optimization problem, which further increases the accuracy of learning tasks in new domains.
R ELATED W ORK
Spoken language understanding is usually deﬁned as a supervised learning problem, involving conventional machine learning models (Young, 2002; Hahn et al., 2011;
Wang et al., 2005) and deep learning models (Mesnil et al., 2015; Kurata et al., 2016; Sarikaya et al., 2011) on massive amount of annotated training data.
Although unsupervised learning and semi-supervised learning based approaches have been proposed as well (Chen et al., 2013; 2015b;a), deep learning approaches were shown to outperform most others.
Besides the requirement of a large amount of annotated data being available, the intents and slots are usually predeﬁned.
Thus they are inﬂexible to expand to new domains.
Some researchers have paid attention to alleviate this limitation.
An example is Korpusik et al. , which work retrains its models to cover new intents and associated slots while redesigning a semantic schema (Korpusik et al., 2014).
Considering that models for different domains are usually trained independently, some attempts have been made to learn shared latent semantic representations or shared model parameters for multiple domains via transfer learning.
A survey for transfer learning with applications is Pan & Yang (2010).
In general, the goal of transfer learning is to improve performance in low data domains and reduce model re-training efforts by bridging the knowledge from data-rich source tasks to new target tasks.
Transfer learning in deep learning setting has been applied to recommendation systems and computer vision domains.
In natural language processing (NLP), researchers have studied cross-domain NLP problems but have mainly focused in cross-lingual problem settings (Mou et al., 2016; Buys & Botha, 2016; Kim et al., 2017a;b).
In Hakkani-Tr et al.
(2016), multi-task and transfer learning approaches are proposed to learn shared implicit feature representation across various domains for spoken language understanding.
Another study Jaech et al.
(2016) follows the similar idea in cross-domain NLP and showed signiﬁcant performance improvement.
Researchers in Amazon have shown in a recent study that DNN-based natural language engines can re-use the existing models through transfer learning (Goyal et al., 2018).
When certain classes do not show up in the training data, one can exploit meta learning to use fewshot learning to help with knowledge transfer Fei-Fei et al.
; Palatucci et al.
; Vinyals et al.
; Ravi & Larochelle (2018).
For example, Vinyals et al.
(2016) learns a network that maps a small labeled support set and unlabeled examples to its labels, saving the effort in ﬁne-tuning to adapt to new class types.
Besides image object recognition tasks, in SLU some experiments have been conducted.
The objective of Ferreira et al. (2015) is to predict the semantic tag sequences of a user query without using any target-domain user utterances and thus in-context semantic tags.
The research studies build a statistical model to predict the SLU for unseen data in Yazdani & Henderson (2015);
Chen et al.
An Action Matching algorithm is proposed by Zhao & Esk´enazi (2018) to learn a cross-domain embedding space that models the semantics of dialog responses, which, in turn, lets a neural dialog generation model generalize to new domains.
Recent publication by
Google (Bapna et al., 2017) explores semi-supervised slot-ﬁlling based on deep learning that can use the slot labels without the need for any labeled or unlabeled data in domain examples or the need for any explicit schema alignment, to quickly bootstrap the model in a new domain.
O UR P ROPOSED A PPROACH FOR M ODEL T RANSFER
In this section, we introduce our transfer learning approach to address the cross-domain SLU problem.
We consider a source domain D s and a target domain D t .
Typically, we have a large annotated dataset for D s , while only a few labeled data for D t .
The tasks of interest in both D s and D t are of the same types, namely they are both slot ﬁlling and intent detection.
However, D s and D t have different intent label spaces T s and T t , because the target domain might have new intent values (slot faces the same situation).
The architecture of our model is illustrated in Fig.
The model contains: (1) a word hashing layer to convert one-hot word representations into embedding vectors, (2) a bi-directional Long Short Term Memory networks(LSTMs) (Hochreiter & Schmidhuber, 1997) layer to capture features from word embeddings, (3) a intent speciﬁc bi-LSTMs layer combined with an semantic network to detect intent, (4) a slot speciﬁc bi-LSTMs layer powered with an attention component and semantic network to conduct slot ﬁlling.
We ﬁrst pre-train this model in a source domain with large amount of annotated data.
We wish to allow the model to extend to new slot names, slot values and intent names in the target domain with little annotated data.
The number of possible intents/slots grows rapidly as we get into new domains with new intents and slots.
One natural approach to this task is to train one binary classiﬁer for each possible label, to decide whether or not to include it in the output.
However, this would require training a large number of classiﬁers.
It also would be impossible to generalize to target domains that include intents and slots that do not show up in the training set, since there will not be any parameter sharing among label classiﬁers.
Instead of encoding slot names and intent names as discrete features in the target domain, we encode them using a continuous representation.
To be speciﬁc, we learn a representation of slot names, slot values and intent names from their constituent words using semantic networks.
We then check to see if these representations match the representations of utterances.
In the following, we explain in details how this representation works in our transfer learning algorithm.
[Figure]
Let w
=
( w 0 , w 1 , w 2 , . .
. , w T +1 ) represent the input word sequence with w 0 and w T +1 being the beginning-of-sequence ( (cid:104) bos (cid:105) ) and end-of-sequence ( (cid:104) eos (cid:105) ) tokens, respectively.
T is the number of words.
Let s =
( s 0 , s 1 , . .
, s T ) be the slot label sequence, where s 0 is a padded slot label that maps to the beginning-of-sequence token (cid:104) bos (cid:105) and s i ∈ S .
Let τ ∈ T be the intent class.
S and T are the slot and intent label space, respectively.
Let sn =
( sn 1 , sn 2 , . .
, sn m ) represent the m slot names.
sn i = ( w 1 , . . .
, w m i ) is a vector of the words for the i -th slot name, where m i is its number of words.
Each slot s i has a list of slot values sv i =
( sv i, 1 , sv i, 2 , . . .
, sv i,n i ) , where n i is the number of slot values for the i -th slot.
A slot value is allowed to be a phrase like ‘fried chicken’ for slot ’Food’.
Let sv
i,j = ( w 1 , . . . , w n i,j ) be the words of the j -th slot value for i -th slot and n i,j is the number of words for this slot value.
3. B ASE N ETWORKS
We use a Pretrained DNN model as our baseline model (Goyal et al., 2018).
The model has three layers of bi-LSTMs.
The ﬁrst layer is the common bi-LSTMs layer, which learns task related representations from input utterances.
The learned representations are taken as input to the upper two bi-LSTMs layers, which are optimized separately for the slot-ﬁlling task and intent-detection task. Our model modify the baseline model in three aspects.
The main difference is that Goyal et al. (2018) treat intent and slot as discrete symbols and train classiﬁers to make predictions. Such a practice limits the knowledge transfer between two domains as the classiﬁer layers following the upper two bi-LSTMs layers need to be re-initialized when ﬁne-tune on new domains (the output labels are different). In our model, we encode intent and slot as continuous representations by their constituent words.
The classiﬁcation problems are transformed to semantic similarity problems and the whole model parameters could be transferred to new domains.
Section 3.2 will introduce the semantic network that encode intent name and slot name into semantic vectors.
The second difference is the usage of gazetteers (lists of slot values for a slot).
Goyal et al.
(2018) use gazetteer features as an additional input.
Such features are binary indicators of the presence of an n-gram in a gazetteer.
In our model, we use attention network (Section 3.3) to encode external slot values into a semantic vector that represents the slot from the value perspective, which suits our semantic framework naturally.
Finally, there are no connections between the upper two layers in Pretrained DNN (Goyal et al., 2018).
However, we believe that the output of intent speciﬁc bi-LSTMs layer could beneﬁt the slot detection task.
As one of our contributions, in our network, we concatenate the output of common bi-LSTMs layer and intent bi-LSTMs layer to feed to the slot bi-LSTMs layer.
For the intent speciﬁc bi-LSTMs layer, we concatenate the last hidden state of each direction to acquire the global representation of the whole utterance h intent =
[ −→ h intentT +1 , ←− h intent] .
Such an operation converts utterances with variable lengths into a ﬁxed-length vector, with which the information throughout the entire utterance can be captured.
Representation h intent will then be used to calculate the similarity scores with representations of candidate intents.
We have also tried other methods like take Mean-Pooling or Max-Pooling of bi-LSTM hidden states to acquire h intent but get almost the same results.
3. S EMANTIC N ETWORKS
We wish to build a representation-learning mechanism for the SLU task that can generalize to unseen words and labels.
In order to represent words, we use word embeddings, which are a form of vector space model.
In various NLP tasks (Baroni et al., 2014; Collobert & Weston, 2008; Mikolov et al., 2013), word embeddings have proven to be effective models of semantic representation of words.
Because word embeddings trained on large corpora of unlabeled data reﬂect the similarity between words, they help the model generalize from the words in the original training domains to the words in the new target domains.
For every utterance we learn how to compose the word embeddings to reﬂect the semantics of that utterance.
Furthermore, we learn how to compose the semantic representation of each intent/slot from the embeddings of the words used to name that intent/slot.
This enables us to generalize to unseen intents/slots.
For slot names, slot values and intent names, we use the same semantic network and transform it to a ﬁxed-length representation vector.
Take slot name
sn i = ( w 1 , w 2 , . . .
, w m i ) as an example, the semantic network ﬁrst extracts the word embedding E ( w i ) for each word.
E ( w i ) is then fed to a feed forward layer to get the non-linear semantic features.
The last is a Mean-Pooling layer, which applies the mean operations over each dimension of semantic features across all words.
The output is treated as the semantic representation for sn
i ,
[Equation]
where W slot transfrom and b slot transfrom are the learned linear projection matrix and bias for slot name semantic network respectively.
In the same way, we could get the semantic representation r slot value i,j for slot value
sv i,j and r intent name i for the intent name τ i .
More sophisticated vector-space semantic representations of the slots/intents are an area for future work, but we believe the simple Mean-Pooling would be a proper choice here since intent names, slot names and slot values are typically very short and composed by entity names.
3. S LOT V ALUE ATTENTION N ETWORKS
With previously introduced semantic networks, we can formulate a semantic representation r slot value i,j of each slot value sv i,j from the i -th candidate slot s i .
The whole slot value representation list is (cid:0) r slot value i,, . . . ,
[Equation]
slot value i [Equation] n i (cid:1) .
At each decoding step t , the slot bi-LSTMs layer output state h slott is connected to each slot’s slot values’ representations to calculate the similarity scores.
Take slot s i as an example, we compute an attention weight α t,i,j for slot value
sv i,j via a bilinear operator, which reﬂects how relevant or important slot value sv i,j is to the current slot hidden states h slott ,
[Equation]
where W v is a parameter matrix.
The overall slot value representation for slot s i would be
[Equation]
3. T RAINING P ROCEDURE
The source domain data containing utterances and associated intents/slots is used to train the model.
The idea behind this model is to learn the representation for both utterances and intents/slots such that utterances with the same intents/slots are close to each other in the continuous semantic space.
We deﬁne a semantic score between an utterance and an intent/slot using the similarity between their representation embeddings h and r by Sim ( h , r )
= h (cid:124) W s r .
W s is the corresponding similarity matrix for slot or intent task.
The posterior probability of a possible intent given an utterance is computed based on the semantic score through a softmax function
[Equation]
where τ i is an intent candidate.
Similarly, we can get the posterior probability of a possible slot at each time.
Since we have the representations of both slot name and slot value, we ﬁrst add the semantic scores from slot name and slot value and then feed to the softmax function to get the overall slot posterior probability
[Equation]
where s i is an slot candidate.
For model training, we maximize the likelihood of the correctly associated intents/intents given all training utterances.
The parameters W of the model are optimized by an joint negative log likelihood objective:
[Equation]
where l is the total number of training utterances, w k is the k -th input utterance.
The model is optimized using mini-batch stochastic gradient descent(SGD)
(Huang et al., 2013).
3. T RANSFER TO A N EW D OMAIN
We train our TSSM using labeled data from source domain D s .
This stage is called pre-training, where the networks learn to extract semantic relationships between utterance representations and intent/slot representations.
The model is able to adapt to new domains, since the seen and unseen intents/slots representations are in the same semantic space, use the same shared composition semantic network with the same unsupervised word embeddings as input.
It has been shown that such word embeddings can capture word similarities and hence the classiﬁer is no longer ignorant about any new intent or slot.
After the pre-training, we ﬁne-tune our model on the target domain D t with less annotated data and with unseen intents/slots.
The model for the new domain D t are the same as that for D s and are initialized by the pre-trained network parameters.
We tried ﬁne-tune different components of the model.
We ﬁx some parts and ﬁne-tune on the other parts.
After experiments on several combinations of components, we found that ﬁne-tune the whole model gives us the best performance.
For zero-shot learning where no ﬁne-tune data is avaiable, in order to predict new intents/slots on new utterances, the TSSM calculates semantic similarites.
We transform each input utterance w into a vector h intent using the base network, and then estimate its semantic similarity with all intents including seen and unseen intents.
The vector representations for new intents can be generated from the trained semantic network by feeding the word embedding vectors of new intents as the input.
For the utterance w , the estimated semantic score of the i -th intent is deﬁned as Sim ( h intent , r intenti ) in Equation 3.4.
Then predicted intent for each utterance is decided according to the estimated semantic scores.
Predicting slot sequences is similar to predicting intents except that the calculation is conducted step by step at each position.
E XPERIMENTS AND R ESULTS
In this section, we ﬁrst conduct extensive experiments to verify the transfer ability of the TSSM with different number of training data in the target domain.
Then we verify the generalization ability of the TSSM with different number of new labels in the target domain.
Finally we verify the zerosshot learning ability of the TSSM model and verify the effectiveness of the proposed “intent2slot” mechanism.
4. D ATASET
In this experiment, we use ﬁve popular public dialogue datasets from ﬁve domains and three datasets obtained from practical applications.
The CamHotel dialogue dataset (Hotel) (Wen et al., 2015) records some dialogues about booking a hotel room in Cambridge.
The CamRestaurant dialogue dataset (Restaurant) (Wen et al., 2016c;a) contains dialogue records about booking a restaurant in
Cambridge.
The Laptop and TV dialogue dataset (Wen et al., 2016b) are dialogues about buying Laptops and TVs.
The Airline Travel Information System dataset (Atis) (Price, 1990) contains dialogues about airline travel, such as the ﬂigh time, the destination, etc.
Weilidai, Huobijijin and Weizhongyouzhe are three ﬁnancial products from WeBank. While Weilidai is very popular and its online customer service accumulated a lot of queries, Huobijijin and Weizhongyouzhe are newly developed products and have only few queries. We would like to pretrain on Weilidai queries and ﬁne-tune on Huobijijin and Weizhongyouzhe queries to improve the online customer service’s experience of the last two products.
In order to thoroughly evaluate the transfer learning ability on more diversiﬁed domains, we make ﬁve different pairs of source and target domains, as shown in table 1.
In case when we have multiple source domains, all source domains will be combined into a large source domain.
For each pair of the source and target domain pair, we ﬁrst pretrain the model on the source domain.
Then we ﬁne-tune the model on the target domain training data, ﬁnally we evaluate the ﬁne-tuned model on the target domain testing dataset.
We use accuracy to evaluate the intent classiﬁcation task and we use accuracy and F1 score to evaluate the slot ﬁlling task. [Table]
4. B ASELINES
We compare the proposed TSSM with the Pretrained DNN (Goyal et al., 2018) and two popular baselines that do not use transfer learning.
The Pretrained DNN (Goyal et al., 2018) (DNN) shares low-level word features between the intent classiﬁation task and the slot ﬁlling task.
The Max Entropy model (MaxEntropy) is a popular baseline for intent classiﬁcation task, which uses bag-of-words features.
This model cannot beneﬁt from the source domain since the source and the target domains have different intents.
The Conditional Random Field model (CRF) is a popular baseline for slot ﬁlling task.
It leverages the pretrained word-embedding models and it can model the transformation probability among sequential labels.
This model also cannot beneﬁt from the source domain.
4. R ESULTS UNDER DIFFERENT SIZES IN TARGET - DOMAIN DATA SETS
In order to evaluate the transfer learning performance for the proposed model, we compare the proposed model with the baselines when we have different number of target domain training data.
After pretraining on the source domain, we ﬁne-tune all models on different percentage of target domain data, and we evaluate the performance of the models on the test set.
We use 50% of the target domain data as testset, and we vary the percentage of trainset used from 5% to 46% .
The results are shown in Figure 2.
As a higher percentage of target domain data is used to ﬁne-tune the model, the performance of all models improves.
Generally, transfer learning models perform better than non-transfer models, which implies that transfer learning can help improve the performance in the target domain.
The proposed TSSM model outperforms the baseline model signiﬁcantly in almost every pair of source and target domains, and for both intent classiﬁcation task and slot ﬁlling task.
As the amount of target domain training data increases, the improvement of the TSSM model remain very signiﬁcant, which demonstrates the TSSM model can effectively adapt to new domains. In some cases, the proposed TSSM is not better than Pre-trained DNN.
This might because that our model has more parameters than the baseline DNN model and in some cases simper model performs better.
Compared with intent classiﬁcation task, the slot ﬁlling task beneﬁt more from transferring from the source domain.
This is because the intent has larger domain differences compared with the slots. For example, the intent in the hotel domain contains verbs such as “inform” “inform no match” while the intent in the atis domain contains nouns such as “ﬂight” “aircraft”. While the slots in all domains mostly contain nouns. This suggest that it is better to transfer from a more similar source task.
The improvement of transferring from Atis to Hotel dataset is worse than the other source and target pairs.
We think this is because the intent and slot difference between “Atis” and ’Hotel’ domains are bigger than the domains difference between other tasks.
[Figure]
4. R ESULTS UNDER DIFFERENT NUMBERS OF NEW LABELS IN THE TARGET DOMAIN
In order to evaluate the generalization ability of the proposed TSSM model, we test the model on previously unseen testing labels.
More speciﬁcally, we evaluate the proposed model under situations when the target domain test set have different number of new labels that is not seen in the target domain training set.
We remove zero, one and two labels and their corresponding training instances from the target domain training set.
The results are shown in Figure 3.
Here we use 10% of all target domain data as the training set, when the percentage of target domain training data increases to 20% or 30% , the results are similar.
As the number of unseen label increases in the target domain testset, the performance of all model drops signiﬁcantly.
Transfer learning methods again out-performans none-transfer learning baselines, which demonstrate that transferring knowledge from source domain is generally beneﬁcial to the target domain.
As the number of unseen label increases, the performance drop of the TSSM model is signiﬁcantly slower than the other baselines models.
Compared with a softmax classiﬁcation layer, the semantic similarity module works better since it can still computes semantic similarity [Figure]
for unseen labels. With a pretrained word embeddings, the TSSM can theoretically compute semantic similarity for arbitrary label as long as the label has a word embedding.
The experimental results demonstrate that the TSSM model can effectively deal with cold start labels that is not seen in the target domain training data.
4. Z ERO - SHOT T RANSFER L EARNING
In the extreme case, there could be no training instances in the target domain at all.
In order to test the proposed TSSM model on such extreme case, we report the performances of the TSSM model when there is no taraget domain training instances.
The result is shown in Table 2.
The random guess performance is 0 .
for intent classiﬁcation accuracy, 0 . 02 for slot accuracy and 0 .
02 for slot f1.
The
TSSM have an performance much better than random guess.
In such case, both the Pretrained DNN, the Max Entropy model and the CRF model could not work, since all of them require target domain training instances to initialize their softmax layer.
[Table]
4. A BLATION E XPERIMENT
One of the improvements we have made is to use the intent bi-LSTMs output states as input features to the slot bi-LSTMs, we call this mechanism “intent2slot”.
In order to evaluate whether the “intent2slot” mechanism can actually improve the performance of the model, we compare the proposed
TSSM with its ablation variant without “intent2slot”, and the result is shown in Table 3 As shown in the result, the TSSM outperforms the TSSM-abl without “intent2slot” in most cases.
The results demonstrate that the “intent2slot” mechanism helps to improve intent classiﬁcation performance.
The parameters in intent bi-LSTMs layer might have obtained additional supervision from the upper slot ﬁlling task. The improvements for transferring from Restaurant, Laptop, TV, Atis to Hotel is not obvious. The reason needs further experiments.
[Table] [Table]
C ONCLUSIONS
In this paper [Table] we introduce a deep semantic similarity network to transfer from domains with sufﬁcient labeled data to low resource domains.
The experiments illustrate that our proposed methods outperform the state-of-the-art baselines in several experimental settings.