T RANSFERRING SLU M ODELS IN N OVEL D OMAINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Spoken language understanding (SLU) is a critical component in building dialogue systems.
When building models for novel natural language domains, a major challenge is the lack of data in the new domains, no matter whether the data is annotated or not.
Recognizing and annotating “intent” and “slot” of natural languages is a time-consuming process.
Therefore, spoken language understanding in low resource domains remains a crucial problem to address.
In this paper, we address this problem by proposing a transfer-learning method, whereby a SLU model is transferred to a novel but data-poor domain via a deep neural network framework.
We also introduce meta-learning in our work to bridge the semantic relations between seen and unseen data, allowing new intents to be recognized and new slots to be ﬁlled with much lower new training effort.
We show the performance improvement with extensive experimental results for spoken language understanding in low resource domains.
We show that our method can also handle novel intent recognition and slot-ﬁlling tasks.
Our methodology provides a feasible solution for alleviating data shortages in spoken language understanding.
I NTRODUCTION
The recent surge of artiﬁcial intelligence motivates the technical and applicable exploration of novel human-computer interactions.
Spoken dialogue systems are widely studied and used in various mobile devices.
Well-known commercial applications driven by dialogue systems include intelligent personal assistants and intelligent robots for customer services.
As more and more products and scenarios integrate dialogue systems in intelligent services, the ability for model adaptation in dialogue systems is critically needed.
Many state-of-the art dialogue systems follow a learning pipeline that includes components such as spoken language understanding (SLU), dialogue management (DM) and spoken language generation or retrieval (SLG).
Since both dialogue management and spoken language generation systems rely on knowledge or information learned from spoken language understanding, research and industry community have paid much attention to the SLU area.
In this paper, we focus on the problem of data shortage for SLU in a new domain.
Spoken language understanding typically deﬁnes and represents user utterances in terms of semantic frames comprised of domains, intents and slots (Tur & Mori, 2011).
A spoken language understanding task involves classifying the domain, detecting the intent of user utterance and identifying token sequences corresponding to slot values in the semantic frame.
A learning model is often designed to handle the above subtasks separately and sometimes simultaneously.
Deep learning models are the cutting-edge techniques for achieving impressive performance in spoken language understanding compared with conventional machine learning models.
However, deep neural-network models often require the collection and manual annotation of data, which is a time and labor intensive process.
These problems pose a major challenge to building high-quality learning models in new domains.
To address this problem, we propose to perform domain adaption of SLU models by transferring the SLU knowledge and model parameters learned in an auxiliary domain where a model is already built.
A related problem is that only few examples of some output intent and slot classes are available during the training phrase in a new domain, because training sets for a new domain are small.
To deal with this limitation, we propose a novel few shot learning model in a meta learning paradigm, we call it the Transfer Semantic Similarity Model (TSSM).
The TSSM achieves few-shot learning by exploiting the semantic embeddings of both seen and unseen intent and slots.
In summary, the challenges of spoken language understanding include: (1) Expanding SLUs to novel domains is necessary as the development of artiﬁcial intelligent products extends to more areas in our society.
But it is difﬁcult to get sufﬁcient data or spoken language sentence in novel domains.
(2) Without sufﬁcient labeled data, deep learning methods are difﬁcult to train and thus it is hard to obtain satisfactory performance in a new domain.
(3) Unseen intents and slots exist because small annotated training set might not cover all intents and slots.
Thus, the generalization ability of the models is crucial for dealing with unseen classes.
Our contributions are summarized as follows:
• We design a novel and efﬁcient SLU model, the TSSM.
It incorporates transfer learning and meta learning for SLU tasks in low resource domains.
• The proposed TSSM can improve model performance with only a small amount of training data in new domains and it can handle unseen classes easily.
The experiments illustrate that our proposed methods outperform the state-of-the-art baselines in many experimental domains.
• Our work can treat intent and slot detection tasks as a structured multi-objective optimization problem, which further improves the accuracy of learning tasks in new domains.
R ELATED W ORK
Spoken language understanding is usually deﬁned as a supervised learning problem, involving conventional machine learning models (Young, 2002; Hahn et al., 2011;
Wang et al., 2005) and deep learning models (Mesnil et al., 2015; Kurata et al., 2016; Sarikaya et al., 2011) on massive amount of annotated training data.
Although unsupervised learning and semi-supervised learning based approaches have been proposed as well (Chen et al., 2013; 2015b;a), deep learning approaches were shown to outperform most others.
Besides the requirement of a large amount of annotated data being available, the intents and slots are usually predeﬁned.
As a result, they are inﬂexible to expand to new domains.
Some researchers have paid attention to alleviate this limitation.
An example is the work by Korpusik et al. , which retrains models to cover new intents and associated slots while redesigning a semantic schema (Korpusik et al., 2014).
There have also been some attempts to learn shared latent semantic representations or model parameters for multiple domains via transfer learning.
A survey for transfer learning with applications is Pan & Yang (2010).
In general, the goal of transfer learning is to improve performance in novel domains that only has small data. In these target domains, it is criticasl to reduce the model re-training efforts by bridging the knowledge from source tasks to the new target tasks.
In natural language processing (NLP), researchers have studied cross-domain NLP problems but have mainly focused in cross-lingual problem settings (Mou et al., 2016; Buys & Botha, 2016; Kim et al., 2017a;b).
In Hakkani-Tr et al.
(2016), both multi-task and transfer learning approaches are proposed to learn shared implicit feature representation across various domains for spoken language understanding.
Researchers in Amazon have shown in a recent study that DNN-based natural language engines can re-use the existing models through transfer learning (Goyal et al., 2018).
When the training data is missing some task labels, one can exploit meta learning to apply few-shot learning to help with the knowledge transfer Fei-Fei et al.
; Palatucci et al.
; Vinyals et al.
; Ravi & Larochelle (2018).
For example, Vinyals et al.
(2016) learns a network that maps a small amount of labeled support set and unlabeled examples to its labels, saving the effort in ﬁne-tuning to new class types.
Besides image object-recognition tasks, in SLU some experiments have been conducted.
The objective of Ferreira et al. (2015) is to predict the semantic tag sequences of a user query without using any target-domain user utterances and thus in-context semantic tags.
The research builds a statistical model to predict the SLU for unseen data in Yazdani & Henderson (2015);
Chen et al.
An action-matching algorithm is proposed by Zhao & Esk´enazi (2018) to learn a cross-domain embedding space that models the semantics of dialog responses, which, in turn, lets a neural dialog generation model generalize to new domains.
Recent publication by
Google (Bapna et al., 2017) explores semi-supervised slot-ﬁlling based on deep learning that can use the slot labels without the need for any labeled or unlabeled data in domain examples or the need for any explicit schema alignment, to quickly bootstrap the model in a new domain.
O UR P ROPOSED A PPROACH FOR M ODEL T RANSFER
In this section, we introduce our transfer learning approach to address the cross-domain SLU problem.
We consider a source domain D s and a target domain D t , where our goal is to learn a model in the target domain.
Typically, we have a large annotated dataset for D s , while only a few labeled data for D t .
The tasks in both D s and D t are of the same types, namely they are both slot ﬁlling and intent detection tasks.
However, D s and D t have different intent label spaces T s and T t , because the target domain might have new intent values (slot faces the same situation).
The number of possible intents and slots grows rapidly as we get into new domains with new intents and slots.
One natural approach in solving the learning problem is to train one binary classiﬁer for each possible label, and decide whether or not to include the classiﬁcation model in the output.
However, this would require training a large number of classiﬁers.
It also would be impossible to generalize to target domains that include intents and slots that do not show up in the training set, since there will not be any parameter sharing among the classiﬁers.
Instead of encoding slot names and intent names as discrete features in the target domain, we encode them using a continuous representation.
To be speciﬁc, we learn a representation of slot names, slot values and intent names from their constituent words using semantic networks.
We then check to see if these representations match the representations of utterances.
The architecture of our model is illustrated in Fig. More speciﬁcaslly, the model incluces the following major components. (1) a word hashing layer to convert one-hot word representations into embedding vectors, (2) a bi-directional Long Short Term Memory networks(LSTMs) (Hochreiter & Schmidhuber, 1997) layer to capture features from word embeddings, (3) a intent speciﬁc biLSTMs layer combined with an semantic network to detect intent, (4) a slot speciﬁc bi-LSTMs layer powered with an attention component and semantic network to conduct slot ﬁlling. We ﬁrst pretrain this model in a source domain with large amount of annotated data.
We then wish to allow the model to extend to new slot names, slot values and intent names in the target domain with little annotated data.
[Figure]
Let w
=
( w 0 , w 1 , w 2 , . .
. , w T +1 ) represent the input word sequence with w 0 and w T +1 being the beginning-of-sequence ( (cid:104) bos (cid:105) ) and end-of-sequence ( (cid:104) eos (cid:105) ) tokens, respectively.
T is the number of words.
Let s =
( s 0 , s 1 , . .
, s T ) be the slot label sequence, where s 0 is a padded slot label that maps to the beginning-of-sequence token (cid:104) bos (cid:105) and s i ∈ S .
Let τ ∈ T be the intent class.
S and T are the slot and intent label space, respectively.
Let sn
=
( sn 1 , sn 2 , . .
, sn m ) represent the m slot names.
sn i = ( w 1 , . . .
, w m i ) is a vector of the words for the i -th slot name, where m i is its number of words.
Each slot s i has a list of slot values sv i =
( sv i, 1 , sv i, 2 , . . .
, sv i,n i ) , where n i is the number of slot values for the i -th slot.
A slot value is allowed to be a phrase like ‘fried chicken’ for slot ’Food’.
Let sv
i,j = ( w 1 , . . . , w n i,j ) be the words of the j -th slot value for i -th slot and n i,j is the number of words for this slot value.
3. B ASE N ETWORKS
We use a Pretrained DNN model as our baseline model (Goyal et al., 2018).
The model has three layers of bi-LSTMs.
The ﬁrst layer is the common bi-LSTMs layer, which learns task related representations from input utterances.
The learned representations are taken as input to the upper two bi-LSTMs layers, which are optimized separately for the slot-ﬁlling task and intent-detection task.
Our model differs from the baseline model by Goyal et al. (2018) in three aspects.
First, the most important difference is that Goyal et al.
(2018) treated intents and slots as discrete symbols and train classiﬁers to make predictions on these symbols.
Such an approach limits the knowledge transfer between two domains as the classiﬁer layers (afﬁne transform leading to the softmax) following the upper two bi-LSTMs layers need be re-initialized when we transfer the model via ﬁne-tuning in new domains, where the output labels are different.
In our model, we encode intents and slots as continuous representations via their constituent words.
The classiﬁcation problems are transformed to semantic similarity problems and the whole model parameters could be transferred to new domains.
The second difference is the usage of gazetteers (lists of slot values for a slot).
Goyal et al.
(2018) used gazetteer features as an additional input.
Such features are binary indicators of the presence of an n-gram in a gazetteer.
In our model, we used an attention network (Section 3.3) to encode external slot values into a semantic vector that represents the slot from the value perspective, which suits our semantic framework naturally.
Finally, there are no connections between the upper two layers in the baseline by Goyal et al.
However, we believe that the output of intent speciﬁc bi-LSTMs layer could beneﬁt the slot detection task.
As one of our contributions, in our network, we concatenate the output of common bi-LSTMs layer and intent bi-LSTMs layer to feed to the slot bi-LSTMs layer.
For the intent speciﬁc bi-LSTMs layer, we concatenate the last hidden state of each direction to acquire the global representation of the whole utterance h intent =
[ −→ h intentT +1 , ←− h intent] .
Such an operation converts utterances with variable lengths into a ﬁxed-length vector, with which the information throughout the entire utterance can be captured.
Representation h intent will then be used to calculate the similarity scores with representations of candidate intents.
We have also tried other methods like take Mean-Pooling or Max-Pooling of bi-LSTM hidden states to acquire h intent but get similar results.
3. S EMANTIC N ETWORKS
We wish to build a representation-learning mechanism for the SLU task that can generalize to unseen words and labels.
For slot names, slot values and intent names, we use the same semantic network and transform them to ﬁxed-length representation vectors.
Take slot name
sn i = ( w 1 , w 2 , . . .
, w m i ) as an example, the semantic network ﬁrst extracts the word embedding E ( w i ) for each word.
E ( w i ) is then fed to a feed forward layer to get the non-linear semantic features.
The last is a Mean-Pooling layer, which applies the mean operations over each dimension of semantic features across all words.
The output is treated as the semantic representation for sn
i ,
[Equation]
where W slot transfrom and b slot transfrom are the learned linear projection matrix and bias for slot name semantic network respectively.
In the same way, we could get the semantic representation r slot value i,j for slot value
sv i,j and r intent name i for the intent name τ i .
More sophisticated vectorspace semantic representations of the slots/intents are an area for future work, but we believe the simple Mean-Pooling would be a proper choice here since intent names, slot names and slot values are typically very short and composed by entity names.
3. S LOT V ALUE ATTENTION N ETWORKS
With previously introduced semantic networks, we can formulate a semantic representation r slot value i,j of each slot value sv i,j from the i -th candidate slot s i .
The whole slot value representation list is (cid:0) r slot value i,, . . .
, r slot value i,n i (cid:1) .
At each decoding step t , the slot bi-LSTMs layer output state h slott is connected to each slot’s slot values’ representations to calculate the similarity scores.
Take slot s i as an example, we compute an attention weight α t,i,j for slot value
sv i,j via a bilinear operator, which reﬂects how relevant or important slot value sv i,j is to the current slot hidden states h slott ,
[Equation]
where W v is a parameter matrix.
The overall slot value representation for slot s i would be
[Equation]
3. T RAINING P ROCEDURE
The source domain data containing utterances and associated intents/slots is used to train the model.
The idea behind this model is to learn the representation for both utterances and intents/slots such that utterances with the same intents/slots are close to each other in the continuous semantic space.
We deﬁne a semantic score between an utterance and an intent/slot using the similarity between their representation embeddings h and r by Sim ( h , r )
=
h (cid:124) W s r .
W s is the corresponding similarity matrix for slot or intent task.
The posterior probability of a possible intent given an utterance is computed based on the semantic score through a softmax function
[Equation]
where τ i is an intent candidate.
Similarly, we can get the posterior probability of a possible slot at each time.
Since we have the representations of both slot name and slot value, we ﬁrst add the semantic scores from slot name and slot value and then feed to the softmax function to get the overall slot posterior probability
[Equation]
where s i is an slot candidate.
For model training, we maximize the likelihood of the correctly associated intents/intents given all training utterances.
The parameters W of the model are optimized by an joint negative log likelihood objective:
[Equation]
where l is the total number of training utterances, w k is the k -th input utterance.
The overall objective function combines the intent classiﬁcation task and slot ﬁlling task objectives.
We observe that this multitask architecture achieves better results than separately training intent and slot models.
The model is optimized using mini-batch stochastic gradient descent(SGD)
(Huang et al., 2013).
3. T RANSFER TO A N EW D OMAIN
We train our TSSM using labeled data from source domain D s .
This stage is called pretraining, where the networks learn to extract semantic relationships between utterance representations and intent/slot representations.
The model is able to adapt to new domains, since the seen and unseen intents/slots representations are in the same semantic space, use the same shared composition semantic network with the same unsupervised word embeddings as input.
After the pretraining, we ﬁne-tune our model on the target domain D t with less annotated data and with unseen intents/slots.
The model for the new domain D t are the same as that for D s and are initialized by the pretrained network parameters.
We tried ﬁne-tune different components of the model.
We ﬁx some parts and ﬁne-tune on the other parts.
After experiments on several combinations of components, we found that ﬁne-tune the whole model gives us the best performance.
For zero-shot learning where no ﬁne-tune data is available, in order to predict new intents/slots on new utterances, the TSSM calculates semantic similarities.
We transform each input utterance w into a vector h intent using the base network, and then estimate its semantic similarity with all intents including seen and unseen intents.
The vector representations for new intents can be generated from the trained semantic network by feeding the word embedding vectors of new intents as the input.
For the utterance w , the estimated semantic score of the i -th intent is deﬁned as Sim ( h intent , r intenti ) in Equation 3.4.
Then predicted intent for each utterance is decided according to the estimated semantic scores.
Predicting slot sequences is similar to predicting intents except that the calculation is conducted step by step at each position.
E XPERIMENTS AND R ESULTS
In this experiment, we use ﬁve popular public dialogue datasets from ﬁve domains and three datasets obtained from practical applications.
The CamHotel dialogue dataset (Hotel) (Wen et al., 2015) records some dialogues about booking a hotel room in Cambridge.
The CamRestaurant dialogue dataset (Restaurant) (Wen et al., 2016c;a) contains dialogue records about booking a restaurant in
Cambridge.
The Laptop and TV dialogue dataset (Wen et al., 2016b) are dialogues about buying
Laptops and TVs.
The Airline Travel Information System dataset (Atis) (Price, 1990) contains dialogues about airline travel, such as the ﬂight time, the destination, etc.
Weilidai, Huobijijin and Weizhongyouzhe are three ﬁnancial products from WeBank, an internet bank in China.
While Weilidai is very popular and its online customer service accumulated a lot of queries, Huobijijin and Weizhongyouzhe are newly developed products and have only few queries.
We would like to pretrain on Weilidai queries and ﬁne-tune on Huobijijin and Weizhongyouzhe queries to improve the online customer service’s experience of the last two products.
In order to thoroughly evaluate the transfer learning ability on more diversiﬁed domains, we make ﬁve different pairs of source and target domains, as shown in table 1.
In case when we have multiple source domains, all source domains will be combined into a large source domain.
For each pair of the source and target domain pair, we ﬁrst pretrain the model on the source domain.
Then we ﬁnetune the model on the target domain training data, ﬁnally we evaluate the ﬁne-tuned model on the target domain test set.
We use accuracy to evaluate the intent classiﬁcation task and we use accuracy and F1 score to evaluate the slot ﬁlling task.
Apart from the Pretrained DNN (Goyal et al., 2018), we also compare TSSM to Max Entropy model and Conditional Random Field model (CRF).
[Table]
4. R ESULTS UNDER DIFFERENT SIZES IN TARGET - DOMAIN DATA SETS
In order to evaluate the transfer learning performance of the proposed model, we compare the proposed model with the baselines when we have different number of target domain training data.
After pretraining in the source domain, we ﬁne-tune the model on different percentages of the target domain data, and we evaluate the performance of the models on the test set.
We use 50% of the target domain data as test set, and we vary the percentage of train set used from 5% to 46% .
The results are shown in Figure 2.
As a higher percentage of target domain data is used to ﬁne-tune the model, the performance of all models improves.
Generally, transfer learning models perform better than non-transfer models, which implies that transfer learning can help improve the performance in the target domain.
The proposed TSSM model outperforms the baseline model signiﬁcantly in almost every pair of source and target domains, and for both intent classiﬁcation task and slot ﬁlling task.
As the amount of target domain training data increases, the improvement of the TSSM model remain very signiﬁcant, which demonstrates that the TSSM model can effectively adapt to new domains.
In some cases, the proposed TSSM is not better than Pretrained DNN.
This might because our model has more parameters than the baseline DNN model and in some cases simper model performs better.
Compared with intent classiﬁcation task, the slot ﬁlling task beneﬁt more from transferring from the source domain.
This is because the intent has larger domain differences compared with the slots.
For example, the intent in the hotel domain contains verbs such as “inform” “inform no match” while the intent in the atis domain contains nouns such as “ﬂight” “aircraft”.
While the slots in all domains mostly contain nouns.
This suggest that it is better to transfer from a more similar source task.
The improvement of transferring from Atis to Hotel dataset is worse than the other source and target pairs.
We think this is because the intent and slot difference between “Atis” and ’Hotel’ domains are bigger than the domains difference between other tasks.
4. R ESULTS UNDER DIFFERENT NUMBERS OF NEW LABELS IN THE TARGET DOMAIN
In order to evaluate the generalization ability of the TSSM model, we test the model on previously unseen labels.
More speciﬁcally, we remove zero, one and two labels and their corresponding training instances from the target domain training set. Then we evaluate TSSM on the target domain test set which has full lists of labels.
The results are shown in Figure 3.
Here we use 10% of all target domain data as the training set, when the percentage of target domain training data increases to 20% or 30% , the results are similar.
As the number of unseen label increases in the target domain test set, the performance of all model drops signiﬁcantly.
Transfer learning methods again out-performs none-transfer learning baselines, which demonstrate that transferring knowledge from source domain is generally beneﬁcial to the target domain.
As the number of unseen label increases, the performance drop of the TSSM model is signiﬁcantly slower than the other baselines models.
Compared with a softmax classiﬁcation layer, the semantic similarity module works better since it can still compute semantic similarity for unseen labels.
With a pretrained word embeddings, the TSSM can theoretically compute semantic similarity for arbitrary label as long as the label has a word embedding.
The experimental results demonstrate that the TSSM model can effectively deal with cold start labels that is not seen in the target domain training data.
4. Z ERO - SHOT T RANSFER L EARNING
In the extreme case, there could be no training instances in the target domain at all.
In order to test the proposed TSSM model on such extreme case, we report the performances of the TSSM model when there is no target domain training instances.
The result is shown in Table 2.
The random guess performance is 0 .
for intent classiﬁcation accuracy, 0 . 02 for slot accuracy and 0 .
02 for slot f1.
The TSSM have an performance much better than random guess.
In such case, both the Pretrained DNN, the Max Entropy model and the CRF model could not work, since all of them require target domain training instances to initialize their softmax layer. [Table]
[Table]
A BLATION E XPERIMENT
One of the improvements we have made is to use the intent bi-LSTMs output states as input features to the slot bi-LSTMs, we call this mechanism “intent2slot”.
In order to evaluate whether the “intent2slot” mechanism can actually improve the performance of the model, we compare the proposed
TSSM with its ablation variant without “intent2slot”, and the result is shown in Table 3 As shown in the result, the TSSM outperforms the TSSM-abl without “intent2slot” in most cases.
The results demonstrate that the “intent2slot” mechanism helps to improve intent classiﬁcation performance.
The parameters in intent bi-LSTMs layer might have obtained additional supervision from the upper slot ﬁlling task.
The improvements for transferring from Restaurant, Laptop, TV, Atis to Hotel is not obvious.
The reason needs further experiments.
[Table]
C ONCLUSIONS
In this paper, we introduced a deep semantic similarity network to transfer from domains with sufﬁcient labeled data to low resource domains.
The experiments illustrated that our proposed methods outperform the state-of-the-art baselines in several experimental settings.
In the future, we will consider different regularization choices to make our model more robust in changing data situations.
We will also consider different data set types for dialog systems.