PROTAUGMENT : Unsupervised diverse short-texts paraphrasing for intent detection meta-learning (2021)

Abstract: Recent research considers few-shot intent detection as a meta-learning problem: the model is learning to learn from a consecutive set of small tasks named episodes. In this work, we propose ProtAugment, a meta-learning algorithm for short texts classification (the intent detection task). ProtAugment is a novel extension of Prototypical Networks, that limits overfitting on the bias introduced by the few-shots classification objective at each episode. It relies on diverse paraphrasing: a conditional language model is first fine-tuned for paraphrasing, and diversity is later introduced at the decoding stage at each meta-learning episode. The diverse paraphrasing is unsupervised as it is applied to unlabelled data, and then fueled to the Prototypical Network training objective as a consistency loss. ProtAugment is the state-of-the-art method for intent detection meta-learning, at no extra labeling efforts and without the need to fine-tune a conditional language model on a given application domain.

## introduction intent detection, a sub-field of text classification, involves classifying user-generated short-texts into intent classes, usually for conversational agents applications @xcite . since conversational agent applications are domain-specific, intent detection is a challenging task because of labeled data scarcity and the number of classes (intents) it usually involves @xcite . as a consequence, recent research @xcite considers few-shot intent detection as a meta-learning problem: the model is trained to classify user utterances from a consecutive set of small tasks named episodes. each episode contains a limited number of c classes alongside a limited number of k labeled data for each of the c classes -this is usually referred to as a c-way k-shots setup. at test time, the algorithm is evaluated on classes that were not seen during training. that is the reason why meta-learning is sometimes referred to as learning to learn: it mimics human abilities to learn iteratively from different and small tasks. meta-learning has successfully been applied to a wide set of nlp tasks: hypernym detection @xcite , low resource machine translation @xcite , machine understanding tasks @xcite or structured query generation @xcite . most metalearning algorithms (section 2) were developed in the course of the last 5 years. it has recently been empirically demonstrated that comparative studies in follow-up papers of @xcite are debatable -for short texts classification -because of the two following main issues @xcite . first, comparative studies involve simple and limited datasets in terms of number and separability of classes (snips @xcite , a very popular dataset, includes only 7 classes, with the current best model performing over 99% accuracy @xcite ). second, as we further better understand @xcite , fine-tune @xcite and refine @xcite ) bert-derived models, it is not clear if the different meta-learning frameworks can be considered state-of-the-art due to their architecture or due to the improvements of available text encoders at the time of conception. @xcite concludes that prototypical networks @xcite (that were using lstm-based text encoders when introduced in nlp) are actually the state-of-the-art for intent detection when equipped with a fine-tuned bert text encoder model. ultimately, improving prototypical networks have therefore been proven to be a very challenging task in reality. a cornerstone challenge is that meta-learning models can easily overfit on the biased distribution introduced by a few training examples @xcite . in order to prevent overfitting and inspired by @xcite , we introduce an unsupervised diverse paraphrasing loss in the prototypical networks framework. a key idea is consistency learning: by augmenting unlabeled user utterances, protaugment enforce a more robust text representation learning. unfortunately, back-translation is a poor data augmentation strategy for short-texts: neural machine translation provides very similar (if not the same) sentences to the original ones, which hinders its ability to provide diverse augmentations (section 5.3). consequently, in this work, we transfer a denoising autoencoder pre-trained on the sequence-to-sequence task @xcite to the paraphrase generation task and then use it to generate paraphrases. as fine-tuning is very efficient for such a model, it is not easy to optimize it for diverse paraphrasing. @xcite presents an approach for diverse paraphrasing that reorders the original sentence to guide the conditional language model to generate diverse sentences. the diversity in that work is provided by the reordering of the elements, which surprisingly affects the attention mechanism. in @xcite , expression diversity is part of the unsupervised paraphrasing system supported by simulated annealing. both approaches imply domain transfer, and consequently, as many diverse paraphrasing models to maintain as the number of considered application domains, which do not scale very well. in this work, we instead introduce diversity in the downstream decoding algorithm used for paraphrase generation. diverse decoding methods are mostly extensions to the beam search algorithm, including noise-based algorithms @xcite , iterative beam search @xcite , clustered beam search @xcite and diverse beam search @xcite . there is no clear optimal solution, the choice is task-specific and dependent on one's tolerance for lower quality outputs as a diversity/fluency trade-off @xcite . while diverse beam search allows controlling the diversity/fluency trade-off partially, we further demonstrate that adding constraints to diverse beam search in order to generate tokens not seen in the input sentence (that is, constrained diverse beam search) is a simple yet powerful strategy to further improve the diversity of the paraphrases. paired with paraphrasing user utterances and its consistency loss incorporated in prototypical net-works, our model is the best method for intent detection meta-learning on 4 public datasets, with neither extra labeling efforts nor domain-specific conditional language model fine-tuning. we also show that protaugment, having access to only 10 samples of each class of the training data, still significantly outperforms a prototypical network which is given access to all samples of the same training data. ## neural architectures for meta-learning past works on meta-learning for classification tasks investigate how to best predict a query point's class at an episode scale. this process is bounded to the set of the c classes considered in a given episode. matching networks @xcite predict the class of a query point as the average cosine distance between the query vector and all support vectors for each class. prototypical networks @xcite extend matching networks: after obtaining support vectors from the encoder, a class prototype is produced via a class-wise vector averaging operation. all query points are then predicted with respect to their distance (cosine or euclidean) to all prototypes. like prototypical networks, relation networks @xcite emerged from computer vision application and were later successfully applied to nlp @xcite . they introduce a relation module, which captures the relationship between data points: instead of using a pre-defined distance (euclidean or cosine most of the time), this approach allows such networks to learn this metric by themselves. this is achieved using either a shallow feed-forward sub-network or a neural tensor layer relation module @xcite (intermediate learnable matrices). another extension to prototypical networks is provided in @xcite . unlabeled data are incorporated using two distinct approaches: i) taking unlabeled data from the same classes as the episode or ii) using any unlabeled data and incorporating both a distractor cluster and masking strategy to minimize the impact of distant unlabeled points. the first approach is unrealistic for meta-learning, as it implies knowing the unlabeled data class. the second method assumes that all the noise is centered around a single distractor cluster and introduces an additional hyperparameter for masking -which is hardly fine-tuneable for small few-shot datasets. ## protaugment in this section, we present our semi-supervised approach protaugment. along with the labeled data randomly chosen at each episode, this approach uses u unlabeled data randomly drawn from the whole dataset -that is, data from training, validation, and test labels. we first do a data augmentation step from this unlabeled data, where we obtain m paraphrases for each unlabeled sentence. the m th paraphrase of x will be denoted xm . then, given unlabeled data and their paraphrases, we compute a fully unsupervised loss. finally, we combine both the supervised loss l (the prototypical network loss using labeled data) and unsupervised loss (denoted l) and run back-propagation to update the model's parameters. ## conclusion in this work, we proposed protaugment, an architecture for meta-learning for the problem of classifying user-generated short-texts (intents). we first introduced an unsupervised paraphrasing consistency loss in the prototypical network's framework to improve its representational power. then, while the recent diverse beam search algorithm was designed to enforce diversity between the generated paraphrases, it does not ensure diversity between the generated paraphrases and the original sentences. to make up for the latter, we introduce constraints in the diverse beam search generation, further increasing the diversity. our thorough evaluation demonstrates that protaugment offers a significant leap in accuracy for the most recent and taugment the new state-of-the-art for this task. we provide the source code of protaugment as well as code for evaluations reported in this paper on a public repository 4 a diverse paraphrase samples orig: are you considered a bot? back: do you consider yourself a bot? dbs 0: is there any way to be considered a bot? dsb 1: is there any way to know if someone is bot? dbs 2: is there any way to know if a person is bot? orig: what is the square root of 31 back: what is the square root of 31 dbs 0: how can i find square roots of 31 in mathematics? dsb 1: how can i find quadratic roots of 31? dbs 2: what is square roots and what is its effect on 31 orig: how long will my transfer be pending for? back: how long will my transfer be on hold? dbs 0: how long will my transfer be pending? i am in first year. dsb 1: when are all transfers coming up and how many days are they expected? dbs 2: if i have a transfer for a while, how long should i wait for it? orig: how to cancel a transaction back: how to cancel a transaction dbs 0: what are some ways to cancel a transaction (in any way)? dsb 1: what are some ways of cancelling an account (in some cases also involving transaction ). dbs 2: is there any way in which i can cancel an existing transaction? orig: i am not sure where my phone is. back: i don't know where my phone is. dbs 0: i am not really sure where my phone is located dsb 1: how can i find the location of any android mobile dbs 2: i don't know where is my cell phone orig: what do i need to do for a refund? back: what do i need to do to get a refund? dbs 0: what should i do now that i am not getting a refund? dsb 1: what are things that should be done before resumption of service ( reimbursed)? dbs 2: what should you do if you want to refund your period of data? orig: does pizza hut have my order ready back: does pizza hut has my order ready dbs 0: does the pizza hut have all your orders ready? dsb 1: does pizza hut have all your orders made up? dbs 2: does the pizza hut have all your orders ready for delivery? orig: go silent for a day back: quiet for a day. dbs 0: do you stay silent for a day or go silent for another day dsb 1: to the rest of the day, stay peaceful and collected. dbs 2: so, to the rest of the day, go silent only. orig: what's the recipe for fish soup back: what is the recipe for fish soup dbs 0: how do you make fish soup? how is the recipe determined? dsb 1: how can you recipe for fish-sugary food? dbs 2: what are the recipes for fish soup and how is it prepared? orig: find easy recipe for almond milk back: find an easy recipe for almond milk dbs 0: what are some good recipe for almond milk? dsb 1: what are some good ways of making almond milk? dbs 2: how do i make almond milk for a beginner? orig: will i need to wear a coat today? back: should i wear a coat today? dbs 0: today, do i need to put on a coat dsb 1: should i wear a coat and what kind of coat dbs 2: what should i wear to work today, and why orig: can you play m3 file back: can you read m3 file dbs 0: m3 files: can i play the entire m3 file? dsb 1: is there any way to play 3m files on earth without downloading it dbs 2: is there any way to play m3 files on windows?