A Multitask Active Learning Framework for Natural Language Understanding (2020)

Abstract: Natural language understanding (NLU) aims at identifying user intent and extracting semantic slots. This requires sufficient annotating data to get considerable performance in real-world situations. Active learning (AL) has been well-studied to decrease the needed amount of the annotating data and successfully applied to NLU. However, no research has been done on investigating how the relation information between intents and slots can improve the efficiency of AL algorithms. In this paper, we propose a multitask AL framework for NLU. Our framework enables pool-based AL algorithms to make use of the relation information between sub-tasks provided by a joint model, and we propose an efficient computation for the entropy of a joint model. Experimental results show our framework can achieve competitive performance with less training data than baseline methods on all datasets. We also demonstrate that when using the entropy as the query strategy, the model with complete relation information can perform better than those with partial information. Additionally, we demonstrate that the efficiency of these active learning algorithms in our framework is still effective when incorporate with the Bidirectional Encoder Representations from Transformers (BERT).

## introduction voice assistant becomes increasingly intelligent on the mobile phone. given an utterance from the user, the task of natural language understanding (nlu) aims at identifying the user's intent and extracting semantic information. this can help a voice assistant to convert the utterance to an executable instruction. the task can be separated as two sub-tasks called intent detection (id) and slot filling (sf) @xcite . ## multitask active learning for nlu our main goal is to enable al algorithms to select samples for both tasks simultaneously. in order to achieve this, the model should be able to directly provide the joint probability of intents and slots given the input sequence. this joint probability contains the aforementioned relation information for id and sf. any model that can provide above components can directly fit into our framework. for the above reasons, we build our joint model from @xcite and (huang et al., 2015) as shown in figure 1 . this section briefly introduce the necessary components in our joint model, and show how to apply al algorithms by these components. bilstm-tricrf joint model: @xcite introduced triangular chain conditional random field (tricrf) for nlu joint model. tricrf jointly represents the slots and intents in probabilistic graphic model, which can encode their dependencies and uncertainty. in tricrf, the potential of input sequence @xmath0, ..., x t ), slot sequence @xmath1, ..., y t ) and intent z is described as after getting the output z and y from equation( 2 ), the loss of joint model is defined as where (y * , z * ) = argmax y,z p θ (y, z|x) is the most likely sequence slots and intent for joint probability distribution. argmax2 ←return two largest values and indexs in the input output: top1score -top2score 1: for @xmath2, z 2 , ..z n do 2: here, (y * 1 , z * 1 ), (y * 2 , z * 2 ) are the first and second best sequence slots and intents under the current model respectively. this method aims to select the instance with the smallest margin between the posteriors of its top two most likely sequence slots and intents. this instance can be considered as the learner has much doubt in distinguishing two possible intents and slot paths. when we select best two joint probabilities on sequence slots and intents, we need to consider the joint probability over all intents rather than considering the former two joint probability on one most possible intent. considering the increasing number of intents and sequence slots paths, we adopt the algorithm 1 to accelerate the speed of calculation. another more common uncertainty-based method is entropy(h) @xcite : where (y, z) ranges over all possible slots and intents for a sequence. this method represents the information needed to "encode" a joint distribution (y, z), and it can be thought as posteriors of our model over its slots and intents. in practice, since the number of possible labels increases sharply with the length of sequence, the quantity of candidate intents and labels combinations grows exponentially. previous work by @xcite has employed n-best sequence entropy (nse) that the entropy of the n parses with the highest probabilities. in our scenario, it can be rewritten as: where @xmath3, y * 2 , ...y * n ), the set of the n-best state sequences in a certain intent. it is an approxima-algorithm 2 calculate joint entropy for nlu input: @xmath4, z 2 ...z n ← all intents n ←the length of sequence p z z ← marginal distribution for intent z ; // the detail in equation ( 15 ) p x z i ← partition function in one intentz i ; using this decomposition, we can define a dynamic programming over entropy. we introduce the detailed computation and proof of algorithm 2 in nlu to the appendix a. ## conclusion in this paper, we propose a multitask active learning framework for nlu that focuses on making use of the relation information between id and sf. we implement representative pool-based query strategies that include least confidence, margin sampling and entropy in our framework. we also perform an efficient computation for the entropy of a joint-model. experimental results show above query strategies with our framework can achieve competitive performance with less training data than baseline methods on all datasets. the results also demonstrate that making use of the relation information between tasks may achieve better performance rather than only consider intents. additionally, our framework is still useful when the model is changed. these results suggest that the framework has the potential to be applied for industrial use.