Transformer to CNN: Label-scarce distillation for efﬁcient text classiﬁcation
Abstract
Signiﬁcant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018.
The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can beneﬁt from training on both task-agnostic and task-speciﬁc unlabelled data.
However, these advantages come with signiﬁcant size and computational costs.
This workshop paper outlines how our proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300 (cid:2) inference speedup and 39 (cid:2) reduction in parameter count.
In some cases, the student model performance surpasses its teacher on the studied tasks.
Introduction
The last year has seen several major advances in NLP modelling, stemming from previous innovations in embeddings [1] [2] [3] and attention models [4] [5] [6] that allow Language Models (LMs) to be trained on very large corpuses : For instance ELMo [7], OpenAI Transformer [8] and recently
BERT [9]
In addition the power of building on LM-enhanced contextualised embeddings, using a ﬁne-tuning approach on task-speciﬁc unlabelled data [10], has shown huge beneﬁts for downstream tasks (such as text classiﬁcation) - especially in a typical industrial setting where labelled data is scarce.
In order to make use of these advances this work shows how a model distillation process [11] can be used to train a novel ‘student’ CNN structure from a much larger ‘teacher’ Language Model.
The teacher model can be ﬁne-tuned on the speciﬁc task at hand, using both unlabelled data and the (small number of) labelled training examples available.
The student network can then be trained using both labelled and unlabelled data, in a process akin to pseudo-labelling [12] [13].
Our results show it is possible to achieve similar performance to (and surpass in some cases) large attention-based models with a novel, highly efﬁcient student model with only convolutional layers.
Model distillation
In this work, we used the OpenAI Transformer [8] model as the ‘teacher’ in a model-distillation setting, with a variety of different ‘student’ networks (see Figure 1).
The OpenAI Transformer model consists of a Byte-Pair Encoded subword [14] embedding layer followed by 12-layers of “decoder-only transformer with masked self-attention heads” [4], pretrained on the standard language modelling objective on a corpus of 7000 books.
This LM’s ﬁnal layer outputs were then coupled with classiﬁcation modules and the entire model was discriminatively ﬁne-tuned with an auxiliary language modelling objective, achieving excellent performance on various NLP tasks.
E
E...
E N
T
Trm
T N T...
Trm Trm
Trm
Trm Trm
C TN
L TN
E
E...
E N
CNN CNN
CNN
Pool Pool
Pool ...
C CNN
L CNN
Concat + MLP
Distillation Loss
BlendCNN (student) OpenAI GPT + classi er (teacher)
To optimize for speed and memory constraints of industrial deployment, a variety of different models were trained (a) on the classiﬁcation task directly; and (b) via distillation [11] of the logit layer output by the pretrained OpenAI classiﬁcation model.
To combat label-scarcity and improve distillation quality, we inferred distillation logits for unlabelled samples in a pseudo-labelling manner [12] [13], while using transfer learning through pretrained GloVe embeddings [2].
Student models
A number of common network structures were tested in the student role, speciﬁcally:
(cid:15) a two-layer BiLSTM network [15] (cid:15) a wide-but shallow CNN network [16] (cid:15) a novel CNN structure, dubbed here ‘BlendCNN’
The BlendCNN architecture was inspired by the ELMo ‘something from every layer’ paradigm, and aims to be capable of leveraging hierarchical representations for text classiﬁcation [6] [17].
The BlendCNN model is illustrated in Figure 1, and comprises a number of CNN layers (with n_channels =100, kernel_width =5, activation = relu ), each of which exposes a global pooling output as a ‘branch’.
These branches are then concatenated together and “blended” through a dense network ( width =100), followed by the usual classiﬁcation logits layer.
[Table]
2-Layer BiLSTM
[Table]
[Table]
Experiments
Each of the models was trained and tested on the 3 standard datasets described in [18] : AG News,
DBpedia and Yahoo Answers.
The experiment proceeded in two phases, the ﬁrst being to evaluate the performance of two baseline methods (TFIDF+SVM [19] and fastText [20]) along with that of the student networks (without the beneﬁt of a LM teacher) [Table] and the large LM, with a classiﬁcation ‘head’ trained on the task.
The second phase used the large LM in a ‘teacher’ role, to train the other networks as students via distillation of the LM classiﬁer logits layer (with a Mean Absolute Error loss function) [Table]
Results
Referring to Table 2, the 3-Layer and 8-Layer variants of the proposed BlendCNN architecture achieve the top scores across all studied datasets.
However, the performance of the proposed architecture is lower without the ‘guidance’ of the teacher teacher logits during training, implying the marked improvement is due to distillation.
The additional results given for BlendCNN quantiﬁes the advantage of adding unlabelled data into the distillation phase of the student model training.
Notably from Table 1, the 3-Layer BlendCNN student has 39 (cid:2) fewer parameters and performs inference 300 (cid:2) faster than the OpenAI Transformer which it empirically out-scores.
Discussion
For text classiﬁcations, mastery may require both high-level concepts gleaned from language under standing and ﬁne-grained textual features such as key phrases.
Similar to the larval-adult form analogy made in [11], high-capacity models with task-agnostic pre-training may be well-suited for task mastery on small datasets (which are common in industry).
On the other hand, convolutional student architectures may be more ideal for practical applications by taking advantage of massively parallel computation and a signiﬁcantly reduced memory footprint.
Our results suggest that the proposed BlendCNN architecture can efﬁciently achieve higher scores on text classiﬁcation tasks due to the direct leveraging of hierarchical representations, which are learnable (even in a label-sparse setting) from a strong teaching model.
Further development of specialized student architectures could similarly surpass teacher performance if appropriately designed to leverage the knowledge gained from a pretrained, task-agnostic teacher model whilst optimizing for task-speciﬁc constraints.