Layer-wise Guided Training for BERT : Learning Incrementally Refined Document Representations (2020)

Abstract: Although BERT is widely used by the NLP community, little is known about its inner workings. Several attempts have been made to shed light on certain aspects of BERT, often with contradicting conclusions. A much raised concern focuses on BERT’s over-parameterization and under-utilization issues. To this end, we propose o novel approach to fine-tune BERT in a structured manner. Specifically, we focus on Large Scale Multilabel Text Classification (LMTC) where documents are assigned with one or more labels from a large predefined set of hierarchically organized labels. Our approach guides specific BERT layers to predict labels from specific hierarchy levels. Experimenting with two LMTC datasets we show that this structured fine-tuning approach not only yields better classification results but also leads to better parameter utilization.

## introduction despite bert's @xcite popularity and effectiveness, little is known about its inner workings. several attempts have been made to demystify certain aspects of bert @xcite , often leading to contradicting conclusions. for instance, @xcite argue that attention measures the importance of a particular word when computing the next level representation for this word. however, @xcite showed that most attention heads contain trivial linguistic information and follow a vertical pattern (attention to @xcite , @xcite , and punctuation tokens), which could be related to under-utilization or overparameterization issues. other studies attempted to link specific bert heads with linguistically interpretable functions @xcite @xcite @xcite , agreeing that no single head densely encodes enough relevant information but instead different linguistic features are learnt by different attention heads. we hypothesize that the aforementioned largely contributes to the lack of attention-based explainability of bert. another open topic is how the knowledge is distributed across bert layers. most studies agree that syntactic knowledge is gathered in the middle layers @xcite @xcite , while the final layers are more task-specific. most importantly, it seems that any semantic knowledge is spread across the model, explaining why non-trivial tasks are better solved at the higher layers @xcite . driven by the above discussion, we propose a novel fine-tuning approach where different parts of bert are guided to directly solve increasingly challenging classification tasks following an underlying label hierarchy. specifically, we focus on large scale multilabel text classification (lmtc) where documents are assigned with one or more labels from a large predefined set. the labels are organized in a hierarchy from general to specific concepts. our approach attempts to tie specific bert layers with specific hierarchy levels. in effect, each of these layers is responsible for predicting the labels of the corresponding level. we experiment with two lmtc datasets (eurlex57k, mimic-iii) and several variations of structured bert training. our contributions are: (a) we propose a novel structured approach to fine-tune bert where specific layers are tied to specific hierarchy levels; (b) we show that structured training yields better results than the baseline across all levels of the hierarchy, while also leading to better parameter utilization. ## structured learning with bert before we proceed with the description of our methods (figure 1 ), we introduce some notation. given a label hierarchy l of depth d, l n denotes the set of labels in the n th level of this hierarchy (n d). also, f @xmath0, where w i and b i are trainable parameters, c i is the [cls] token in the i th bert layer, foot_4 and σ is the sigmoid activation function. note that the sizes of w i and b depend on the number of labels that f i is responsible for predicting, i.e., if f i predicts the labels of l n , w @xmath1. flat: this is a simple baseline which uses f 12 to predict all labels in the hierarchy in a flat manner. in effect, w 12 ∈ r |l|×768 and b @xmath2. note ## experiments comparing [cls] representations: figure 3 shows the average angular distances between the [cls] representations of each layer on development data of eurlex57k. 8 the angular distance is calculated on unit (l 2 normalized) vectors, takes values in [0, 1], and a distance of 0.5 indicates an angle of 90 • . we observe that one-by-one leads to larger angles between the representations than last-six which in turn yields larger angles than flat. in effect, one-by-one and to a lesser extent last-six lead to a better parameter utilization than flat. to better support this claim we provide a geometric interpretation. we first l 2 normalize all [cls] representations. each normalized representation can be interpreted as a vector having its initial point at the origin and its terminal point at the surface of a 768-dimensional hyper-sphere (centered at the origin). the larger the angle between two [cls] vectors the further apart they are on the 8 for brevity, we only show the heatmaps of three methods. we include the missing ones in appendix e. ## related work our approach is similar to @xcite but they experiment with fully connected networks, which are not well suited for text classification, contrary to stacked transformers @xcite . similarly, @xcite used convolutional neural networks, albeit with shallow hierarchies (2 levels). although our approach leverages the label hierarchy it should not be confused with hierarchical classification methods (silla and freitas, 2011), which typically employ one classifier per node and cannot scale-up to large hierarchies when considering neural classifiers. a notable exception is the work of @xcite who employed one bidirectional lstm with label-wise attention @xcite per hierarchy node. however, for their method to scale-up, they use probabilistic label trees @xcite to organize the labels in their own shallow hierarchy which does not follow the abstraction level of the original hierarchy. to the best of our knowledge we are the first to apply this approach to pre-trained language models. ## conclusions and further work we proposed a novel guided approach to fine-tune bert, where specific layers are tied to specific hierarchy levels. experimenting with two lmtc datasets, we showed that structured training not only yields better results than a flat baseline, but also leads to better parameter utilization. in the future we will try to further increase the parameter utilization by guiding bert's attention heads to explicitly focus on specific hierarchy parts. we also plan to improve the explainability of our methods with respect to the utilization of their parameters.