Meta-Information Guided Meta-Learning for Few-Shot Relation Classification (2020)

Abstract: Few-shot classification requires classifiers to adapt to new classes with only a few training instances. State-of-the-art meta-learning approaches such as MAML learn how to initialize and fast adapt parameters from limited instances, which have shown promising results in few-shot classification. However, existing meta-learning models solely rely on implicit instance-based statistics, and thus suffer from instance unreliability and weak interpretability. To solve this problem, we propose a novel meta-information guided meta-learning (MIML) framework, where semantic concepts of classes provide strong guidance for meta-learning in both initialization and adaptation. In effect, our model can establish connections between instance-based information and semantic-based information, which enables more effective initialization and faster adaptation. Comprehensive experimental results on few-shot relation classification demonstrate the effectiveness of the proposed framework. Notably, MIML achieves comparable or superior performance to humans with only one shot on FewRel evaluation.

## introduction conventional machine learning algorithms, especially neural methods, require an adequate amount of data to learn model parameters. to alleviate the heavy reliance on annotated data, few-shot learning, which aims at adapting to new tasks with only a few training examples, has drawn more and more attention. few-shot classification is a typical few-shot learning task, which samples several new classes with a handful of training examples (i.e., support instances) and query instances, and requires models to classify these queries into given classes @xcite . to grasp the patterns of new classes with limited examples, meta-learning was proposed. inspired by human behaviors, meta-learning models focus on learning to learn: they learn how to better initialize parameters and fast adapt classification models from given instances. for example, maml @xcite finds the best initialization point of parameters, where it can take minimal efforts to reach the optimal points for each class. to this end, maml adapts towards each class by gradient steps using support instances, and uses the loss of the adapted model on the query instances to optimize the initialization parameters. however, meta-learning still has three challenges: (1) most meta-learning methods learn how to learn (i.e., how to initialize and adapt) solely relying on instance statistics, which inevitably suffer from data sparsity and noise in low-resource scenarios, especially in text domain. (2) the approach of learning to learn, like the learning process itself, is a black-box and thus lacks interpretability. (3) most conventional meta-learning methods are designed for few-shot classification, and cannot well handle zero-shot scenarios, where no support instances are available. in contrast, humans usually learn novel concepts with high-level descriptive definitions, instead of solely learning from several unsystematic instances. for figure 1 : diagram of meta-learning models. (a) maml learns a class-agnostic representation θ 0 that can fast adapt to new classes. (b) miml learns meta-parameters φ to fast initialize class-aware parameter θ 0 i , and to quickly adapt to new classes using informative instances, where both phases are guided by meta-information. informative instances and noisy instances are marked accordingly. inspired by the learning process of humans, we propose a novel meta-information guided meta-learning (miml) framework, as shown in figure 1 . the meta-information derives from the semantic concepts of classes, and could provide strong guidance for both parameter initialization and fast adaptation in meta-learning. specifically, miml integrates meta-information in two essential components, namely the meta-information guided fast initialization and fast adaptation. (1) in meta-information guided fast initialization, instead of using a static class-agnostic initialization point for all classes as in maml, miml uses meta-information to estimate dynamic class-aware initialization parameters for each class. this alleviates the reliance on support instances to reach optimal adapted parameters. (2) in meta-information guided fast adaptation, miml adapts the class-aware initialization parameters with gradient steps according to the support instances, where informative support instances are selected to contribute more to the adaptation gradients with a novel meta-information based attention mechanism. by integrating high-level meta-information and concrete instances, miml achieves superior performance on low-resource tasks. moreover, miml also provides better interpretability in meta-learning process. note that we are not the first attempt to use meta-information for low-resource classification tasks: in zero-shot learning, where there are no training examples for new classes at all, class names are used to produce semantic representations for classification @xcite @xcite . in few-shot scenarios, however, supporting examples can bring more direct supervision. in this paper, we argue that both signals are crucial to the learning process, and combining them could achieve the best results. in experiments, the significant improvements on few-shot relation classification tasks demonstrate the effectiveness and robustness of miml in low-resource relation classification. we show the advantage of miml in handling noisy instances, and its potential in zero-shot classification. we also conduct comprehensive ablation study and visualization to better understand our model. in summary, our main contributions are twofold: (1) we propose a principled meta-information guided meta-learning framework for few-shot classification. to the best of our knowledge, we are the first to introduce meta-information to meta-learning for few-shot relation classification. (2) we conduct comprehensive experiments to demonstrate the effectiveness of miml. notably, miml achieves human-level performance with only one shot on fewrel evaluation. we also show the robustness and interpretability of miml, as well as its potential in zero-shot classification through experiments. ## preliminary in few-shot classification, we aim to learn a model that can handle the classification task with only a few available training instances. specifically, given a set of classes c from the class distribution p(c), the algorithm 1 meta-information guided meta-learning require: p(c): distribution over classes require: β: meta learning rate 1: randomly initialize: φ = {φ e , φ n , φ a }: meta-parameters 2: while not done do 3: sample batch of classes c i ∼ p(c) 4: sample support instance set s and query instance set q 5: for all c i do 6: fast initialize parameters of c i : θ 0 @xmath0; φ n ) 7: for @xmath1, . . . , t do 8: compute gradients and learning rates for fast adaptation using support instance set s 9: compute adapted parameters with gradient descent: θ t+1 = θ ti,j α i,j ∇ θ t l(f θ t ,{φe,φn} , x j , y j ) 10: meta-optimize using query instance set q: our work is inspired by maml @xcite , an effective meta-learning approach to the few-shot classification problem. maml contains two key phases: initialization and fast adaptation. initialization aims to learn a globally shared initialization point of parameters for different classes, such that a few gradient steps of fast adaptation on the initialization parameters can produce good results on new classes. we refer readers to the paper @xcite for more details about maml. ## methodology in this section, we introduce our meta-information guided meta-learning (miml) framework. despite the effectiveness of maml, we observe that two assumptions in maml limit the model capacity: to address the aforementioned problems, we propose miml to integrate meta-information into metalearning, and provide strong guidance in both initialization and adaptation phases. the intuition behind miml is that human learn new concepts from both high-level meta-information and concrete instances. specifically, miml consists of four components: instance encoder. given a sentence and the corresponding entity pair, we employ deep neural networks (with meta-parameters φ e ) to construct the representation of the relation between the entity pair. meta-information guided fast initialization. in fast initialization phase, miml dynamically initializes the parameters for each class based on meta-information (with meta-parameters φ n ), which can be viewed as a rough but flexible estimation of class parameters from high-level semantics. ## experiments in this section, we empirically evaluate miml on few-shot relation classification. to evaluate the robustness of miml, we conduct experiments in the presence of noisy instances. we also show the potential of miml in zero-shot classification. ablation study and visualization are conducted to better understand the inner mechanism of miml.