Adversarial Semantic Decoupling for Recognizing Open-Vocabulary Slots (2020)

Abstract: Open-vocabulary slots, such as file name, album name, or schedule title, significantly degrade the performance of neural-based slot filling models since these slots can take on values from a virtually unlimited set and have no semantic restriction nor a length limit. In this paper, we propose a robust adversarial model-agnostic slot filling method that explicitly decouples local semantics inherent in open-vocabulary slot words from the global context. We aim to depart entangled contextual semantics and focus more on the holistic context at the level of the whole sentence. Experiments on two public datasets show that our method consistently outperforms other methods with a statistically significant margin on all the open-vocabulary slots without deteriorating the performance of normal slots.

## introduction slot filling is a critical component of spoken language understanding (slu) in task-oriented dialogue systems. it aims at extracting semantic constituents from the user queries. given an immense amount of labeled training data, recent neural networks @xcite @xcite @xcite @xcite have been actively applied to slot filling task and achieved good results. although most previous neural-based models achieve state-of-the-art performance across a wide range of slot filling datasets, they often suffer from poor slot filling accuracy while dealing with 'openvocabulary' slots. open-vocabulary slots signify slot types that can take on values from a virtually unlimited set, such as file name, album name, text body, or schedule title. typically, these slot values add the song don't drink the water to my playlist add the song don't drink the water to my playlist @xcite . here "water" is mistakenly recognized as "entity name" type by the baseline model @xcite due to the local context "don't drink the water". however, it represents a playlist at the level of the whole sentence. @xcite . we display the top10 slot types of the highest error rates. in this paper, we propose a robust adversarial slot filling approach that explicitly decouples local semantic representation inherent in open-vocabulary slot words from the global context. our approach aims to focus more on the holistic semantics at the level of the whole sentence, not only the vicinity of the local context within open-vocabulary slots. specifically, our approach generates modelagnostic adversarial worst-case perturbations to the inputs in the direction that significantly increases the model's loss. our main contributions are threefold: (1) we dive into the issues of open-vocabulary slots in slot filling task and propose a novel adversarial semantic decoupling method which distinguishes local semantics from the global context. fig 3 shows the overall architecture of our method. here we adopt bilstm @xcite as our backbone. 2 our method includes three core steps: forward, backward, and decoupling forward. we first feed each word to an embedding layer to get word embeddings e @xmath0. then in the forward step, we adopt a bilstm layer and softmax output layer to calculate the classification cross-entropy loss l(f (e; θ), y ) for each word. then in the second backward step, we perform adversarial attacks @xcite @xcite @xcite to explicitly shift the local semantics of open-vocabulary slot words and decouple them from the global context. theoretically, we need to compute a decoupling vector v dec that effectively degrades the current model's performance (i.e., maximum the loss function): here, the gradient g is the first-order differential of the loss function l w.r.t. e, representing the direction that rapidly increases the loss function. we perform normalization to g and then use a small to ensure the approximate is reasonable. finally, we perform a mask operation to filter out normal words and add the decoupling vector to the original token embeddings e. hence, the updated word embeddings are @xmath1. ablation study proves that only adding the decoupling vector to open-vocabulary slot words achieves better improvement. in the third decoupling forward step, we feed e to the same bilstm model and calculate a new adversarial loss l . the final loss is a weighted sum of l and l controlled by a hyperparameter α foot_2 : baselines for a fair comparison, we use the same slot filling architecture bilstm @xcite as @xcite . @xcite proposes two model variants, where random noise means adding random noise in the embeddings of all slot words and cw represents concatenating the context word window as input. note that the random noise in @xcite is independently sampled regardless of the global context, which is significantly different from our method. our adversarial semantic decoupling method can take into account the impact of different contexts (global semantics) on local semantics, thereby enabling more accurate decoupling. @xcite proposes greedy delex and iterative delex methods for open-vocabulary slots. we also validate our method in the bert-based models @xcite for comprehensive analysis. evaluation we evaluate the performance of slot filling using the f1 metric @xcite . specially, we report the f1 score over all open-vocabulary slots, noted as f1-ov. we followed the set-ups in @xcite , and re-implement the baseline bil-stm, +random noise and +random noise,cw based on the same settings. we report the original results of greedy delex and iterative delex from @xcite . we also show the results of bert models. ## conclusion in this paper, we dive into the issues of openvocabulary slots in slot filling task and propose a novel model-agnostic adversarial semantic decoupling method which distinguishes local semantics inherent in open-vocabulary slot words from the global context. experiments confirm the effectiveness of semantic decoupling. we hope to provide new guidance for the future slot filling work.