MGIMN: Multi-Grained Interactive Matching Network for Few-shot Text Classiﬁcation
Abstract
Text classiﬁcation struggles to generalize to unseen classes with very few labeled text instances per class.
In such a few-shot learning (FSL) setting, metric-based meta-learning approaches have shown promising results.
Previous studies mainly aim to derive a prototype representation for each class.
However, they neglect that it is challenging-yet-unnecessary to construct a compact representation which expresses the entire meaning for each class.
They also ignore the importance to capture the inter-dependency between query and the support set for few-shot text classiﬁcation.
To deal with these issues, we propose a metalearning based method MGIMN which performs instance-wise comparison followed by aggregation to generate class-wise matching vectors instead of prototype learning.
The key of instance-wise comparison is the interactive matching within the class-speciﬁc context and episode-speciﬁc context.
Extensive experiments demonstrate that the proposed method signiﬁcantly outperforms the existing SOTA approaches, under both the standard FSL and generalized FSL settings.
Introduction
Few-shot text classiﬁcation has attracted considerable attention because of signiﬁcant academic research value and practical application value (Gao et al., 2019; Yin, 2020; Brown et al., 2020; Bragg et al., 2021; Liu et al., 2021).
Many efforts are devoted towards different goals like generalization to new classes (Gao et al., 2019; Ye and Ling, 2019;
Nguyen et al., 2020; Bragg et al., 2021), adaptation to new domains and tasks (Bansal et al., 2020;
Brown et al., 2020; Schick and Schütze, 2020; Gao et al., 2020; Bragg et al., 2021).
Low-cost generalization to new classes is critical to deal with the growing long-tailed categories, which is common for intention classiﬁcation (Geng et al., 2019;
[Figure]
Krone et al., 2020) and relation classiﬁcation (Han et al., 2018; Gao et al., 2019).
To prevent over-ﬁtting to few-shot new classes and avoid retraining the model when the class space changes, metric-based meta learning has become the major framework with signiﬁcant results (Yin, 2020; Bansal et al., 2020).
The core idea is that episode sampling is employed in meta-training phase to learn the relationship between query and candidate classes (Bragg et al., 2021).
A key challenge is inducing class-wise representations from support sets because nature language expressions are diverse (Gao et al., 2019) and highly informative lexical features are unshared across episodes (Bao et al., 2019).
Under metric-based meta learning framework, many valuable backbone networks have emerged.
Snell et al.
(2017) presented prototypical network that computes prototype of each class using a simple mean pooling method.
Gao et al.
(2019) proposed hybrid attention mechanism to ease the negative effects of noisy support examples.
Geng et al.
(2019) proposed an induction network to induce better prototype representations.
Ye and Ling (2019) obtained each prototype vector by aggregating local matching and instance matching information.
Bao et al.
(2019) proposed a novel method to compute prototypes based on both lexical features and word occurrence patterns.
All these previous works ﬁrst obtain class-wise representations and then perform class-wise comparisons.
However, it is challenging-yet-unnecessary to construct a compact representation which expresses the entire meaning for each class (Parikh et al., 2016).
In text matching research, compareaggregate methods which perform token-level comparisons followed by sentence-level aggregation has already been successful (Parikh et al., 2016;
Tay et al., 2017; Yang et al., 2019).
Besides backbone networks,there are still some work that can be further combined.
Luo et al.
(2021) utilized classlabel information for extracting more discriminative prototype representation.
Bansal et al.
(2020) generated a large-scale meta tasks from unlabeled text in a self-supervised manner.
In this paper we propose M ulti- g rained
I nteractive M atching N etwork, a backbone network for few-shot text classiﬁcation.
The core difference between us with previous efforts is that our work performs instance-level comparisons followed by class-wise aggregation.
Speciﬁcally, ﬁrst, all text sequences including query and all support instances are encoded to contextual representations.
Second, as depicted in Figure 1, we design a novel multi-grained interactive matching mechanism to perform instance-wise comparisons which capture the inter-dependency between query and all support instances.
Third, class-wise aggregate layer obtains class-wise matching vector between query and each class.
Finally, a prediction layer predicts ﬁnal results.
In contrast to standard FSL setting, generalized FSL setting is a more challenging-yet-realistic setting where seen classes and new classes are coexistent (Nguyen et al., 2020; Li et al., 2020a).
In such a setting, we analyze the relationship between inference speed and the number of classes, and verify the necessity of retrieval, which is ignored by previous studies.
Our contributions are listed as follows:
• We propose MGIMN which is more concerned with matching features than semantic features through multi-grained interactive matching.
• We verify the necessity of retrieval for realistic applications of few-shot text classiﬁcation when the number of classes grows.
• We conduct extensive experiments and achieve SOTA results under both the standard FSL and generalized FSL settings.
Background 2. Few-Shot Learning
Few-shot learning focuses on construct a classiﬁer G ( S, q ) →
y which maps the query q to the label y given the support set S .
Few-shot learning is significantly different from traditional machine learning.
In traditional machine learning tasks, we directly sample batches of training data during training.
Unlike traditional machine learning models, fewshot learning models usually adopt episode training strategy which means we build meta tasks using sampled training data in each training step.
Speciﬁcally, during each training step, N different classes are randomly chosen. After the classes are chosen, we sample R samples as query set Q and K samples for each class as support set S where Q (cid:84) S ∈ ∅ .
For training, given the support set S =
{ S kn ; i = 1 , .., N, j = 1 , .., K } , and query set Q = { ( q i , y i ); i = 1 , 2 , .., R, y i ∈ 1 , .., N } which has R samples in each training step, the training objective is to minimize:
[Equation]
For evaluation, as we described in introduction section, there are two settings.
In standard FSL setting, we do N-way K-shot sampling on the classes for validation and test (which are unseen during training) to construct episodes for validation and test.
In generalized FSL setting, we reformulate the FSL task as C -way K -shot classiﬁcation where C is the count of all classes for training, validation and test, and usually is far greater than N.
In this setting, we construct episodes by sampling on all classes for test.
2. Matching Network
Matching Network (Vinyals et al., 2016) is a typical few-shot learning approach, which leverages the cosine similarity to perform few-shot classiﬁcation.
Speciﬁcally, for the query instance q and each support instance S kn ∈ S , the cosine similarity
[Figure]
Class N 𝑆 1.
𝑆 1.
𝑆 𝑁.
𝑆 𝑁.
Query ……
𝑆 2. 𝑆 2.
Pre-trained Encoder
Instance Matching Layer ……
𝑆 1.
𝑆 1.
𝑆 𝑁.
𝑆 𝑁.2 𝑆 2.
𝑆 2.2 ……
Instance-wise Matching Vector
Class-wise Aggregation Layer ……
Class [Figure] Matching Vector
Prediction Layer …… between q and S kn is computed as follow:
[Equation]
Then, we compute the probability distribution of the label y of the query q using attention:
[Equation]
Finally, for any query instance q , we regard the class with the maximum probability as its label y :
[Equation]
2. Text Classiﬁcation
As a basic task in NLP, text classiﬁcation has attracted much attention.
In previous works, different model architectures, including RNN (Zhou et al., 2016) and CNN (Kim, 2014) are used for text classiﬁcation.
After the appearance of pretrained language models like BERT (Devlin et al., 2019), they have become the mainstream method for text classiﬁcation.
In such methods, the input sentence is encoded into its representation using the Transformer (Vaswani et al., 2017) architecture through adding [ CLS ] token before the original input sentence x and then computing the output representation of [ CLS ] using the model.
[Equation]
where θ represents the model’s parameters.
Then, according to the representation, the probability distribution of y can be computed as follow:
[Equation]
where W softmax is the parameters of the softmax layer.
Method
As illustrated in the Figure 2, MGIMN consists of four modules: Encoder Layer, Instance Matching
Layer, Class-wise Aggregation Layer and Prediction Layer.
3. Encoder Layer
We employ transformer encoder from pre-trained
BERT as encoder layer.
Similar to the original work, we add a special token
[CLS] before original text.
Then the encoder layer takes a token sequence as input and outputs token-wise sequence representation.
Instead of using the vector of [CLS] token as sentence-wise representation, we adopt ﬁnal hidden states of the rest tokens for further ﬁne-grained instance-wise matching.
We denote x = { w 1 , w 2 , ..., w l } as a token sequence.
Encoder Layer outputs the token-wise representation h
= { h 1 , h 2 , ..., h l } , where l denotes length of the token sequence.
Query and each support instance are encoded individually.
We denote q as encoded result of query and s nk as encoded result of the k th support instance of n th class.
3. Instance Matching Layer
This is the core of our model.
Instance-wise matching vectors are obtained by comparing query with each support instance.
3.2. Bidirectional Alignment
Following previous works(Parikh et al., 2016; Yang et al., 2019) in text matching , we use bidirectional alignment to capture inter-dependency between two text sequences.
[Equation]
where a and b denote the token-wise sequence representations and BiAlign denotes the bidirectional alignment function deﬁned as follows:
[Equation]
where a i and b j denotes representation of i th and j th token of a and b , respectively, (cid:98) a i and (cid:98) b j denote the aligned representations, and F is a single-layer feed forward network.
3.2. Multi-grained Interactive Matching
In few-shot text classiﬁcation, judging whether query and each support instance belong to the same category cannot be separated from class-speciﬁc context and episode-speciﬁc context.
There are three components including alignment, fusion and comparison.
For alignment, besides local alignment between query and each support instance, we also consider their alignments with global context information.
We denote S n = concat ( { s nj } Kj =1 ) as class-speciﬁc context and S = concat ( { S i } Ni =1 ) as episode-speciﬁc context.
The multi-grained alignments for query and each support instance are performed as follows:
[Equation]
where q (cid:48) nk , q (cid:48)(cid:48) n and q (cid:48)(cid:48)(cid:48)
denote instance-aware, classaware and episode-aware query representations respectively, s (cid:48)
nk , s (cid:48)(cid:48)
nk and s (cid:48)(cid:48)(cid:48)
nk denote query-aware, class-aware and episode-aware support instance representations respectively.
For fusion, we fuse original representation and three kinds of aligned representations together as follows:
[Equation]
[Table]
[Equation] [Equation] [Table]
[Equation]
where H 1 , H 2 , H 3 , H are feed forward networks with single-layer and initialize with independent parameters, (cid:12) denotes the element-wise multiplication operation, and ; denotes concatenation operation.
For comparison, the instance-wise matching vector is computed as follows:
[Equation]
where q nk is a l q × D matrix, s nk is a l s × D matrix, (cid:126)q nk and (cid:126)s nk are vectors with shape ( 1 × 2 D ), G is single-layer feed forward networks, l q and l s are sequence length of query and support instance respectively, and D is hidden size.
3. Class Aggregation Layer
This layer aggregate instance-wise matching vectors into class-wise matching vectors for ﬁnal prediction. [Equation] where (cid:126)c n denotes the ﬁnal matching vector of n th class, “ ; ” denotes the concatenation of two vectors and (cid:126)m nk denotes instance-wise matching vector produced by instance matching layer.
3. Prediction Layer
Finally, prediction layer, which is a two-layer fully connected network with the output size is 1 , is applied to the matching vector (cid:126)c n and outputs ﬁnal predicted result.
[Equation]
Experiments 4.
Setup
4.1. The preparation of dataset
The proposed method has been evaluated on ﬁve diverse corpora: OOS (Larson et al., 2019), Liu (Liu et al., 2019), FaqIr (Karan and Šnajder, 2016),
Amzn (Yury., 2020) and Huffpost (Misra., 2018).
Among them, OOS, Liu and FaqIr datasets are all intent classiﬁcation datasets.
Amzn dataset is designed for ﬁne-grained classiﬁcation of product reviews.
Huffpost dataset is constructed to identify the types of news based on headlines and short descriptions.
The dataset characteristics is listed in Table 1.
For the standard FSL setting, we construct the “support & query" set by sampling the unique
N classes and K samples each class, and R samples for each of classes, respectively.
We conduct two groups of experiments using N = [5 , 10] , K =and R = 5 .
In the evaluation phase, we sample 500 episodes and report the average accuracy.
In generalized FSL setting(GFSL for short), we train the model with episode sampling of 5-way 5-shot.
And then we evaluate the model performance with C-way K-shot.
For all experiments, we divide all datasets for 5 times using different random seeds, just like the way of cross validation, to remove the impact of dataset division.
And we conduct 3 experiments for each model by using different random seeds for model initialization.
The ﬁnal results are reported by averaging 5 × 3 = 15 runs.
For fair comparison, all models implemented in this paper adopt BERT-Tiny 1 as encoder layer which is a 2-layer 128-hidden 2-heads version of BERT.
Meanwhile, these models initialized their paramaeters using the PTM published by Google and ﬁne-tuned during training procedure.
Besides, the encoder layer and all parameters of other layers are randomly initialized.
We ﬁx some hyperparameters with default values such as the hidden size 128 , we also exploit Adam optimizer in all experiment settings.
The learning rate is tuned from 1 e − 5 to 1 e − 4 on validation dataset.
Dropout with a rate of 0.1 is applied before each fully-connected layer.
The feed-forward networks described in section 3 (e.g. F, H 1 , H 2 , H 3 , H and G ) are all single fully-connected layers.
The prediction layer is a two-layer fully-connected layer.
4.1. Baselines
It is vital to compare the introduced method with some strong baselines with two evaluation metrics mentioned above.
Note that we re-implement all methods with the same pre-trained encoder for fairly comparison.
• Prototypical Network (Proto) (Snell et al., 2017) is the ﬁrst designed and applied to image classiﬁcation and has also been used to deal with the text classiﬁcation issue in recent studies.
• Matching Network (Matching) (Vinyals et al., 2016) computes the similarity both on each query and per support samples, and then averages them as ﬁnal prediction score.
• Induction network (Induction) (Geng et al., 2019) proposes an induction module to induce the prototype by using dynamic routing.
• Proto-HATT (Gao et al., 2019) is introduced to deal with the issue of noisy and diverse by leveraging instance-level attention and featurelevel attention.
• MLMAN (Ye and Ling, 2019), can be regarded as one of the variants of Proto, encodes query and support in an interactive way.
4. Main results Overall Performance
Our key experiment results are given in Table 2, 3 and 4. We report [Table] [Table]
[Table] [Table] [Table] [Table]
Methods
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table]
[Table]
the averaged scores over 15 runs (different seenunseen class splits and random seeds as introduced in section 4.1.1) for each dataset and model. Our method remarkably better than all baselines on the ﬁve diverse corpora, especially in more challenging generalized FSL setting: the improvements on Huffpost and Amzn datasets are 2.83% and 2.75% respectively.
Generalized FSL
In most studies of text classiﬁcation (Bao et al., 2019; Gao et al., 2019) with few-shot manner, N-way K-shot accuracy is the standard evaluation metric.
There are two problems: (1) The metric is not challenging, usually N = 5 or N = 10 , much smaller than C .
We also can see that high scores are often reported in some work(Bao et al., 2019; Gao et al., 2019).
It is unable to reﬂect the real application scenarios where we usually face the entire class space (both seen classes and unseen classes).
Consequently, the more challenging generalized FSL evaluation metric is employed to focus on the problems.
As shown in Table 2, 3 and 4, the performance of generalized
FSL evaluation is worse and more challenging than standard FSL.
It is very meaningful in realistic scenario and can contribute to the further research.
It is noteworthy that, comparing with Proto, our proposed approach makes bigger improvement in the challenging generalized FSL metric (GFSL) than the improvements in standard FSL metric(FSL), e.g. OOS dataset: 14 .
29% of GFSL vs 6 . 09% of FSL and FaqIr dataset: 15 .
03% of GFSL vs 9 .
13% of FSL.
Obviously, it can be implied from the experiment results that, the presented approach has higher effectiveness among such challenging scenarios.
Huffpost Dataset
Samples of the same class are more diverse and scattered on Huffpost.
For instance, "green streets are healthy streets", "the real heroes of Pakistan" and "what next for Kurdistan ?" are from the same class:"WORLD NEWS".
In this scenario, a single class-wise prototype is difﬁcult [Table] to represent the entire class semantic.
Interestingly, our approach improves more signiﬁcantly than other datasets, 2 .
22% of 5-way 5-shot standard FSL metric, 1 . 9% of 10-way 5-shot standard FSL metric and 2 .
83% of generalized FSL metric.
In our approach, richer matching features gained through interacting from low level with multi-grained interaction, are effective on the dataset with diverse expressed samples.
4. Ablation Study
To further validate the effect of different interaction levels and the effectiveness of instance matching vector, we make some ablation studies on both the datasets Liu and Huffpost.
4.3. Different Interaction Levels
We respectively take out the single-level interaction layer and see how the speciﬁc alignment feature affects the performance.
As shown in the Table 5, when taking out the speciﬁc interaction layer, the performance decreases in varying degrees, which explains that each alignment layer has positive effect on the performance and can complement each other.
It is noted that class-level interaction layer has the greatest impact.
The model can pay attention to the whole class context through class-level interaction, which makes the model encode more precise class semantic information.
It is the key to judge the relationship between query and class.
4.3. Instance-wise Matching Vector
We remove all interaction layers in our model, named ‘w/o instance & class & episode’ in Table 5.
Then it is the same as matching network except that the scalar matching score is replaced by instance-wise matching vector.
We make comparison with matching network.
As given in the Table 5, 2 and 4, it performs better than matching network, e.g. Liu dataset improves 3 .
49% of 10-way 5-shot FSL score and Huffpost dataset improves 2 .
30% of GFSL score.
Unlike scalar comparison in matching network, our approach can make ﬁne-grained instance vector comparisons in dimensional level.
4. Number of Classes and Inference Speed
As shown in Table 6, the inference speed increases linearly with the increase of the number of classes, from 315 ms/query to 1630 ms/query when c increases from 50 to 318 .
It is challenging for deploying the model to the online application.
To address the problem of inference speed, motivated by the idea of retrieval in traditional search system, we propose the retrieval-then-classify method (RTC-mgimnbert).
We construct the class-wise representation by averaging the vector of each support instance, produced by MGIMN encoder and then calculate the similarity between query and class-wise representation.
In our experiments, we retrieve top N = 10 classes with K = 5 instances per class.
(2)Stage2-classify: Retrieved support instances by stage1 are classiﬁed by MGIMN proposed in this paper.
The C -way 5 -shot task is reduced to a 10 -way 5 -shot which can greatly save the inference time and computation cost.
As shown in Table 6, in addition to the generalized FSL metric score, we also report the inference speed (processing time per query) to verify the effectiveness of retrieval-then-classify.
We regard the inference speed of retrieval-then-classify is greatly increased by 5 × to 23 × , with a small amount of loss.
The proposed method bring some positive effects on model performance compared with other retrieval methods such as BM25 and original BERT.
Related Work 5. Few-shot Learning
Intuitively, the few-shot learning focus on learn a classiﬁer only using a few labeled training examples, which is similar to human intelligence.
Since the aim of few-shot learning is highly attractive, researchers have proposed various few-shot learning approaches.
Generally, the matching network encodes query text and all support instances independently (Vinyals et al., 2016), then computes the cosine similarity between query vector and each support sample vector, and ﬁnally computes average of all scores for per class. The prototypical network basi- [Table]
[Table] cally encodes query text and support instances independently (Snell et al., 2017), then computes classwise vector by averaging vectors of all support instances in each class, and ﬁnally computes euclidean distances as ﬁnal scores.
Sung et al. (2018) introduced relation network (Sung et al., 2018) that exploits a relation module to model relationship between query vector and class-wise vectors rather than leveraging the distance metric (e.g., euclidean and cosine).
Geng et al. (2019) introduced the induction network and leverage the induction module that take the dynamic routing as a vital algorithm to induce and generalize class-wise representations.
In the few-shot scenario, the model-agnostic manner usually viewed as the improved version of fewshot, and deﬁned as MAML (Finn et al., 2017), which can be exploited in different ﬁelds MT (Gu et al., 2018; Li et al., 2020b), dialog generation (Qian and Yu, 2019; Huang et al., 2020).
For few-shot text classiﬁcation, researchers have also proposed various techniques to improve the existing approaches for few-shot learning.
Basically, the one of our strong baselines Proto-HATT is introduced by Gao et al.
that leverages the attention with instance-level and feature-level then highlight both the vital features and support point. Ye and Ling (2019) also tries to encode both query and per support set by leveraging the interactive way at word level with taking the matching information into account.
5. Text Matching
Text matching model aims to predict the score of text pair dependent on massive labeled data.
Before
BERT, related work focuses on deriving the matching information between two text sequences based on the matching aggregation framework.
It performs matching in low-level and aggregates matching results based on attention mechanism.
Many studies are proposed to improve performance.
The attend-compare-aggregate method (Parikh et al., 2016) which has an effectiveness on alignment, meanwhile aggregates the aligned feature by using feed-forward architecture.
The previous work extracts ﬁne-grained matching feature with bilateral matching operation by considering the multiperspective case(Wang et al., 2017).
Tay et al.
(2017) exploit the factorization layers to enhance the word representation via scalar features with an effective and strong compressed vectors for alignment.
Yang et al.
(2019) present a straightforward but efﬁcient text matching model using strong alignment features.
After the PTM (e.g., BERT) is presented (Devlin et al., 2019), it has became commonly used approach on the various ﬁelds of NLP.
Thus, many text matching methods are also leveraging the PTM.
For example, Reimers and Gurevych (2019) use the sentence embeddings of BERT to conduct text matching, and Gao et al.
(2021) use contrastive learning to train text matching models.
Additionally, to handle the issue of few-shot learning architecture, we employ the similar idea of comparison, aggregation and introduce new architecture multi-grained interactive matching network.
Conclusion
In this research investigation, we introduce the M ulti- g rained I nteractive M atching
N etwork( MGIMN ) for the text classiﬁcation task with few-shot manner.
Meanwhile, we introduce a two-stage method retrieval-then-classify (RTC) to solve the inference performance in realistic scenery.
Experiment results illustrate that the presented method obtains the best result in all ﬁve different kinds of datasets with two evaluation metrics.
Moreover, RTC method obviously make the inference speed getting faster.
We will make further investigations on the the task of domain adaptation problem by extending our proposed method.