PSSAT : A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling (2022)

Abstract: Most existing slot filling models tend to memorize inherent patterns of entities and corresponding contexts from training data. However, these models can lead to system failure or undesirable outputs when being exposed to spoken language perturbation or variation in practice. We propose a perturbed semantic structure awareness transferring method for training perturbation-robust slot filling models. Specifically, we introduce two MLM-based training strategies to respectively learn contextual semantic structure and word distribution from unsupervised language perturbation corpus. Then, we transfer semantic knowledge learned from upstream training procedure into the original samples and filter generated data by consistency processing. These procedures aims to enhance the robustness of slot filling models. Experimental results show that our method consistently outperforms the previous basic methods and gains strong generalization while preventing the model from memorizing inherent patterns of entities and contexts.

## introduction the slot filling (sf) task in the goal-oriented dialog system aims to identify task-related slot types in certain domains for understanding user utterances. traditional supervised slot filling models and sequence labeling methods @xcite @xcite @xcite have shown remarkable performance. however, these models tend to memorize inherent patterns of entities and contexts @xcite . faced with uncertainty and diversity of human language expression, the perturbation of entities and contexts will lead to a decrease in the generalization ability of the sf model, which hinders its further application in practical dialog scenarios. * the first four authors contribute equally. weiran xu is the corresponding author. email: dongguanting@bupt.edu.cn due to the variety of expression habits, users may not interact with the dialogue system abiding by a rigid input mode in real dialog scenarios. instead, the expression styles of users would be of high lexical and syntactic diversity while users express their intentions. an interesting finding is that, every expression retains the key semantic information of the sentence to ensure consistency of the intention, but it inevitably damages the semantic structure of the context. as shown in figure 1 , the original sentence comes from training data, while the other two sentences are real queries of users with different language habits. firstly, paraphrase and simplification perturb the contextual semantic structure of the original sentence to various degrees. secondly, some slot entities also suffer from word perturbations. however, they all retain pricerelated information to express the same intention. we refer to the above two perturbations collectively as spoken language perturbation. the previous slot filling model, which tends to memorize entity patterns, has a significantly reduced generalization ability when faced with these situations. therefore, it is necessary to train a robust slot filling model against perturbations in practical application. recently, improving the robustness of nlp systems against input perturbations has attracted increasing attention. most existing studies @xcite @xcite that explored the robustness problem are only about rule-based synthetic datasets, which have certain limitations. further, @xcite focused on the robustness of the ner model against optical character recognition (ocr) disturbance and misspellings. however, real-world dialogue systems face more diverse perturbations due to frequent interactions with users. @xcite proposed language understanding augmentation, which contains four data augmentation methods, to simulate natural perturbations. nevertheless, each method is designed for a specific perturbation, which cannot generalize for other unknown perturbations. to solve the above issues, in this paper, we propose a perturbed semantic structure awareness transferring method (pssat). it can generate augmented data based on human diversity expressions. in fact, it is not difficult to obtain unsupervised corpora containing spoken language perturbations in real-world scenarios (e.g. social media). therefore, we extract the texts from two multi-modal datasets @xcite and construct an unsupervised language perturbation corpus, which helps the model learn the semantic structure of perturbed data. to be specific, we introduce a perturbed structure pre-training stage, which guides the model to directly learn contextual semantic structure and words distribution from unsupervised language perturbation corpus through two different mlm-based training strategies, respectively. to better eliminate the distribution gap between upstream and downstream data, we design a semantic structure transferring generation stage to transfer the upstream learned semantic structure knowledge to downstream original train-ing samples. by doing so, the generated augmented samples are more in line with the spoken language perturbation. however, as there are mixed perturbations existed in upstream corpus, the model may generate some low-quality samples. to alleviate this problem, we introduce consistency processing to filter generated samples. our contributions are three-fold: (1) to the best of our knowledge, this is the first work to investigate spoken language perturbation of slot filling tasks and validate the vulnerability of existing rulebased methods in the condition of diverse language expressions. (2) we propose a perturbed semantic structure awareness transferring method, which transfers the learned contextual semantic structure and word distribution into the original samples through the mlm-based method. (3) experiments demonstrate that our method outperforms all baseline methods and gains strong generalization while preventing the model from memorizing inherent patterns of entities and contexts. ## conclusion in this paper, we propose a perturbed semantic structure awareness transferring method for perturbation-robust slot filling task. specifically, we design the perturbed structure pre-training and the semantic structure transferring generation to transfer the upstream learned semantic structure knowledge to downstream original training samples. further, we filter low-quality samples through a consistency processing module. sufficient experiments and error analysis demonstrate the effectiveness and generalization of our methods, and also prove that pssat alleviates the problem of memorizing inherent patterns of entities and contexts.