Knowledge-Aware Meta-learning for Low-Resource Text Classification (2021)

Abstract: Meta-learning has achieved great success in leveraging the historical learned knowledge to facilitate the learning process of the new task. However, merely learning the knowledge from the historical tasks, adopted by current meta-learning algorithms, may not generalize well to testing tasks when they are not well-supported by training tasks. This paper studies a low-resource text classification problem and bridges the gap between meta-training and meta-testing tasks by leveraging the external knowledge bases. Specifically, we propose KGML to introduce additional representation for each sentence learned from the extracted sentence-specific knowledge graph. The extensive experiments on three datasets demonstrate the effectiveness of KGML under both supervised adaptation and unsupervised adaptation settings.

## introduction learning-to-learn (or meta-learning) @xcite @xcite @xcite has recently emerged as a successful technique for training models on large collections of low-resource tasks. in the natural language domain, it has been used to improve machine translation @xcite , semantic parsing @xcite ), text classification @xcite @xcite , sequence labelling @xcite ), text generation @xcite , knowledge graph reasoning @xcite , among many other applications in low-resource settings. meta-learning has been shown to dominate selfsupervised pretraining techniques such as masked language modeling @xcite when the training tasks are representative enough of the tasks encountered at test time @xcite . however, in practice, it requires access to a very large number of training tasks @xcite 2021) and, especially in the natural language domain, mitigating discrepancy between training and test tasks becomes non-trivial due to new concepts or entities that can be present at test time only. in this paper, we propose to leverage external knowledge bases (kbs) in order to bridge the gap between the training and test tasks and enable more efficient meta-learning for low-resource text classification. our key idea is based on computing additional representations for each sentence by constructing and embedding sentence-specific knowledge graphs (kgs) of entities extracted from a knowledge base shared across all tasks (e.g., fig. 1 ). these representations are computed using a graph neural network (gnn) which is meta-trained endto-end jointly with the text classification model. our approach is compatible with both supervised and unsupervised adaptation of predictive models. ## preliminaries we consider the standard meta-learning setting, where given a set of training tasks t 1 , . . . , t n , we would like to learn a good parameter initialization θ for a predictive model f θ such that it can be quickly adapted to new tasks given only a limited amount of data (i.e., few-shot regime). each task t i has a support set of labeled or unlabeled sentences in our text classification setup, we assume that parameters θ are split into two subsets: (1) bert @xcite parameters θ b shared across tasks and (2) task-specific parameters θ c that are adapted for each task. below, we discuss two adaptation strategies: supervised and unsupervised. ## approach in this section, we present the proposed kgml framework (fig. 2 ), which allows us to enhance supervised and unsupevised adaptation methods described in the previous section with external knowledge extracted from a shared kb and. in the following subsections, we elaborate the key components of kgml: (1) extraction and representation of sentence-specific knowledge graphs (kgs) and (2) knowledge fusion. ## experiments in this section, we show the effectiveness of our proposed kgml on three datasets and conduct related analytic study.