Self-Training using Rules of Grammar for Few-Shot NLU (2021)

Abstract: We tackle the problem of self-training networks for NLU in low-resource environment—few labeled data and lots of unlabeled data. The effectiveness of self-training is a result of increasing the amount of training data while training. Yet it becomes less effective in low-resource settings due to unreliable labels predicted by the teacher model on unlabeled data. Rules of grammar, which describe the grammatical structure of data, have been used in NLU for better explainability. We propose to use rules of grammar in self-training as a more reliable pseudo-labeling mechanism, especially when there are few labeled data. We design an effective algorithm that constructs and expands rules of grammar without human involvement. Then we integrate the constructed rules as a pseudo-labeling mechanism into self-training. There are two possible scenarios regarding data distribution: it is unknown or known in prior to training. We empirically demonstrate that our approach substantially outperforms the state-of-the-art methods in three benchmark datasets for both scenarios.

## introduction deep learning for natural language understanding (nlu) achieves satisfactory performance in various tasks such as intent detection (id) or slot filling (sf) @xcite @xcite . given a sentence @xmath0, id is a task to find an intent i of s among several possible intents, and sf is to identify keywords in s and tag a correct slot sequence s 1 s 2 • • • s n of s in bio format. recent studies have focused on the data scarcity problems for id and sf tasks. in low-resource settings with few labeled data, the model performance often becomes poor if it is not properly trained * the first two authors contributed equally to this work. to cope with the problem. @xcite suggest several approaches for few-shot learning such as data augmentation and self-training (st). data augmentation has been popular in image processing and recently is being used in nlp applications. however, it is still an open issue if data augmentation is suitable for nlu since the quality of augmented data might not be reliable even if it improves the overall performance @xcite . on the other hand, st @xcite shows promising performance for low-resource settings @xcite . nevertheless, classic st is often unreliable since the pseudo-labeling mechanism heavily depends on the teacher model of its own. this motivates researchers to add more reliable mechanisms to the teacher model and to make better st models @xcite . rules of grammar are one of the oldest techniques to represent knowledge in nlp @xcite , and recently have become popular again for few-shot learning due to the reliability and explainability of rules @xcite . on the other hand, naive rule extraction is easy to overfit on the source corpus and is impossible to recognize data outside of the corpus. generalizing the rules can solve this issue. we propose to use rules of grammar to enhance the teacher model in st. because we have formal grammars as rules, we can expand rules by grouping similar fragments and represent data more precisely with an easy grammar modification. we investigate the usefulness of rules of grammar from a quantitative perspective and study how good these rules for id and sf, especially in low-resource settings. it is well-known that reflecting a real-world data distribution among different classes makes better models when such information is available in prior training labeled data (l) label u with m id/sf dnn module m figure 1: an overview of our self-training procedure using rules of grammar for accurate id and sf when there are few labeled data. the rules of grammar are expanded by edit-distance, n-gram, and slot-centric methods. the models @xcite . thus, we consider two scenarios: when the distribution is unknown, or known in prior. in summary, our research questions are rq1: are rules of grammar useful in st for few-shot nlu? rq2: which rule expansion gives the best performance? rq3: does it make a difference to know the data distribution in prior for rules of grammar in st? it parses a sentence transportation on monday, and identifies its intent as ground_transport. ## experiments and analysis we run experiments on two scenarios: 1) when the data distribution among classes is unknown and 2) when it is known in prior. we first evaluate how well rules of grammar perform in low-resource settings for rq1. we then verify the effectiveness of various grammar expansions for rq2, and examine how the data distribution affects rules of grammar for rq3. dataset: we use three benchmark datasets: atis @xcite , snips @xcite , and facebook @xcite . each sentence in the datasets has an intent and a sequence of slot labels. we split a dataset into train, validation, and evaluation datasets with the ratio of 64%, 16%, and 20%, respectively. for the first scenario when the data distribution is unknown among classes, we build an initial corpus of 10, 20 and 30 sentences for each intent from both the train and the validation sets at random. for the second scenario when the distribution is known in prior, we take 1%, 5% and 10% of our the train and the validation datasets according to the data distribution from the original corpus. we then erase labels in the remaining train dataset, which becomes the unlabeled dataset for st. baseline models: in our empirical tests, we have observed that recent text classification models for few-shot settings such as ust @xcite or delta-training @xcite show competitive performance for id but extremely poor performance for sf due to the nature of the sf task. thus, we compare our model with the following two baselines, which are designed for the id/sf tasks explicitly. ## conclusions recently, rules of grammar have become popular again in deep learning due to their reliability and explainability. we have adapted rules of grammar in st for nlu in few-shot settings. a major problem when using rules of grammar is the fact that it might not be flexible to cope with exceptions or variant data from the current rules. we have resolved this problem by expanding rules based on the data/grammar similarity. this gives rise to more precise pseudo-labels in st and better performance. we have demonstrated that the combination of slot-centric and edit-distance shows the best performance. for both scenarios when knowing data distribution in prior or not, our algorithm achieves a substantial improvement. remark that rules of grammar do not depend on specific neural network models, and, therefore, learning with rules of grammar is a complementary solution for the few-shot problem in nlu.