Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting (2022)

Abstract: In document classification for, e.g., legal and biomedical text, we often deal with hundreds of classes, including very infrequent ones, as well as temporal concept drift caused by the influence of real world events, e.g., policy changes, conflicts, or pandemics. Class imbalance and drift can sometimes be mitigated by resampling the training data to simulate (or compensate for) a known target distribution, but what if the target distribution is determined by unknown future events? Instead of simply resampling uniformly to hedge our bets, we focus on the underlying optimization algorithms used to train such document classifiers and evaluate several group-robust optimization algorithms, initially proposed to mitigate group-level disparities. Reframing group-robust algorithms as adaptation algorithms under concept drift, we find that Invariant Risk Minimization and Spectral Decoupling outperform sampling-based approaches to class imbalance and concept drift, and lead to much better performance on minority classes. The effect is more pronounced the larger the label set.

## introduction large-scale multi-label document classification is the task of assigning a subset of labels from a large predefined set -of, say, hundreds or thousands of labels -to a given document. common applications include labeling scientific publications with concepts from ontologies @xcite , associating medical records with diagnostic and procedure labels @xcite , pairing legislation with relevant legal concepts @xcite , or categorizing product descriptions @xcite . the task in general presents interesting challenges due to the large label space and two-tiered skewed label distributions. class imbalance in multi-label classification, datasets often exhibit class imbalance, i.e., skewed chronological splits across the medium-sized datasets ( temporal concept drift moreover, class distributions may change over time. this is one dimension of the temporal generalization problem @xcite . recently, @xcite argued chronological data splits are necessary to estimate real-world performance, contrary to random splits @xcite , because random splits artificially removes drift. temporal concept drift, which we focus on here -instead of covariate shift @xcite , for example -is an instance of concept drift @xcite , often discussed in the domain adaptation literature, e.g., @xcite . multi-label class imbalance class imbalance in (large-scale) multi-label classification has so far been studied through the lens of network architectures, searching for the best neural architecture for handling few-and zero-shot labels in the multi-label setting. to improve the performance for underrepresented (few-shot) classes, @xcite introduced prototypical networks that average all instances in each class to form prototype label vectors (encodings), a form of inductive bias, which improved few-shot learning. in a similar direction, @xcite developed the label-wise attention network (lwan) architecture, in which label-wise document representations are learned by attending to the most informative words for each label, using trainable label encodings (representations). @xcite extended lwan and the idea of prototype label encodings. they combined label descriptors with information from a graph convolutional network @xcite that considered the relations of the label hierarchy to improve the results in few-shot and zero-shot settings. alternatives to lwan were considered by @xcite , presenting minor improvements in the few-shot setting, but harming the overall performance. ## datasets we experiment with three datasets ( eur-lex european union (eu) legislation is published in eur-lex. 4 all eu laws are annotated by eu's publications office with multiple concepts from eurovoc, a thesaurus maintained by the publications office. 5 eurovoc has been used to index documents in systems of eu institutions, e.g., in web legislative databases, such as eur-lex and cellar, the eu publications office's common repository of metadata and content. we use the english part of the dataset of @xcite , which comprises 65k eu laws (documents). 6 the dataset is chronologically split in training @xcite development (5k, 2010-2012), test (5k, 2012-2016) subsets. it supports four different label granularities. we use the 1st and 2nd level of the eurovoc taxonomy including 21 and 127 categories, respectively. bioasq the bioasq (task a: large-scale online biomedical semantic indexing) dataset @xcite comprises biomedical articles from pubmed, foot_6 annotated with concepts from the medical subject headings (mesh) taxonomy. 8 mesh is a controlled and hierarchically-organized vocabulary produced by the national library of medicine. the current version of mesh contains more than 29k concepts referring to various aspects of the biomedical research (e.g., diseases, chemicals and drugs). it is used for indexing, cataloging, and searching of biomedical and health-related information, e.g., in medline/pubmed, and the nlm databases. we use a subset of 100k documents derived from the latest version (v.2021) of the dataset. 9 we subsample documents in the period 2000-2021, and we consider chronologically split training @xcite , development (10k, 2015-2018), test (10k, 2018-2020) subsets. we use the 1st and 2nd levels of mesh, including 16 and 116 categories. ## fine-tuning algorithms in our experiments, we rely on pre-trained english language models @xcite and fine-tune these using different learning objectives. our main goal during fine-tuning is to find a hypothesis (h) for which the risk r(h) is minimal: similar to previous studies, r(h) is an expectation of the selected loss function (l). in this work, we study multi-label text classification (section 3), thus we aim to minimize the binary cross-entropy loss across l classes: erm (vapnik, 1992), which stands for empirical risk minimization, is the most standard and widely used optimization technique to train neural methods. the loss is calculated as follows: furthermore, we consider a representative selection of group-robust fine-tuning algorithms that try to mitigate performance disparities with respect to a given attribute (a), e.g., in a standard scenario that could be the gender of a document's author in sentiment analysis, or the background landscape in image classification. in our case, the attribute of interest is the labeling of the documents. the attribute is split into g groups, which in our case are the classes (@xmath0. all algorithms rely on a balanced group sampler, i.e., an equal number(n g i ) of instances (samples) per group (g i ) are included at each batch. most of the algorithms are built upon group-wise losses (l g i ), computed as follows: in our case, contrary to previous applications of group-robust algorithms, the groups (classes) are not mutually exclusive (documents are tagged with multiple labels). hence, the group sampler can only guarantee that at least n groups (labels) will be considered at each step, but most probably even more. in this work, we examine the following group-robust algorithms in a label-wise fashion: group uniform is the more naive group robust algorithm that uses the average of the group-wise (label-wise) losses -all groups (labels) are considered equally important-, instead of the standard sample-wise average, as follows: group dro @xcite , stands for group distributionally robust optimization (dro). group dro is an extension of the group uniform algorithm, where the group-wise (labelwise) losses are weighted inversely proportional to the group (label) performance. the total loss is calculated as follows: v-rex @xcite , which stands for risk extrapolation, is yet another proposed grouprobust optimization algorithm. @xcite hypothesize that variation across training groups is representative of the variation later encountered at test time, so they also consider the variance across the group-wise (label-wise) losses. in v-rex the total loss is calculated as follows: irm @xcite , which stands for invariant risk minimization, mainly aims to penalize variance across multiple training dummy estimators across groups, i.e., performance cannot vary in samples that correspond to the same group. the total loss is computed as follows: deep coral @xcite , minimizes the difference in second-order statistics (covariances) between the source and target feature activations. in practice, it introduces group-pair penalties: spectral decoupling @xcite relies on the idea of gradient starvation. pezeshki et al. state that a network could become over-confident in its predictions by capturing only one or a few dominant features. thus, adding an l2 penalty on the network's logits (ŷ i ) provably decouples the fixed points of the dynamics. the total loss is computed as follows: in our work, we consider the aforementioned algorithms in a label-wise setting, instead of a groupwise setting given a protected attribute. in our case, @xmath1, where l is the number of labels. ## experimental setup baseline models for both legal datasets (uk-lex, eur-lex), we use the small legal-bert model of @xcite , a bert @xcite ) model pre-trained on english legal corpora. for bioasq, we use the small english bert model of @xcite . following devlin et al. (2019) , we feed each document to the pre-trained model and obtain the top-level representation h [cls] of the special @xcite token as the document representation. the latter goes through a dense layer of l output units, one per label, followed by a sigmoid activation. we also experiment with the label-wise attention network (lwan) relying on a bert encoder @xcite , dubbed bert-lwan. 10 10 the original model was proposed by @xcite , with a cnn encoder. chalkidis et al. reported state-of-art results in eur-lex and amazon-13k using bert-lwan compared to several baselines. bert-lwan uses one attention head per label to generate l document representations d l : t is the document length in tokens, h t the contextaware representation of the t-th token, k, v are linear transformations of h t , and q l a trainable vector used to compute the attention scores of the l-th attention head; q l can also be viewed as a label representation. intuitively, each head focuses on possibly different tokens of the document to decide if the corresponding label should be assigned. bert-lwan employs l linear layers (o l ) with sigmoid activations, each operating on a different label-wise document representation d l , to produce the probability of the corresponding label p l : across experiments, we use bert models following a small configuration (6 transformer blocks, 512 hidden units and 8 attention heads), which allows us to increase the batch size up to 64 and consider samples with multiple labels (groups) in the group robust algorithms. in practice, this enables us to sample at least 4 samples per group (label) for all labels in the small label sets, and at least 1 sample per group (label) for 64 labels in the medium-sized label sets (69-112 labels). training details we fine-tune all models using the adamw @xcite optimizer with a learning rate of 2e-5. we use a batch size of 64 and train models for up to 20 epochs using early stopping on the development set. we run three repetitions with different random seeds and report the test scores based on the seed with the best scores on development data. we report development scores on appendix b.2. evaluation metrics given the large number and skewed distribution of labels, retrieval measures have been favored in large-scale multi-label text classification literature @xcite @xcite . following @xcite , we report mean r-precision (m-rp) @xcite , while data and code in our experiments, we extend the wilds (koh et al., 2021) library, which provides an experimental framework for experimenting with group-robust algorithms. we effectively rewrote all parts of code to consider label-wise groups and losses, while we also implemented the unsupported methods (group uniform, v-rex, and spectral decoupling). for reproducibility and further exploration with new group-robust methods, we release our code on github. 11 6 results ## alternative combined algorithm having a clear understanding of what irm and spectral decoupling offer, it seems that we could combine both to leverage all features: (a) rely on group-wise (label-wise) losses as the main driver of the optimization process (equation 6 ); (b) penalize the classifier if there is a performance disparity between samples labeled with the same classes (equation 10 ); and (c) penalize the classifier for being over-confident (equation 13 ). we name the new algorithm label-wise distributional robust optimization, lw-dro in short, as it mainly aims to mitigate label-wise disparities, and investigate two alternatives (variants): the notation used in equations 17 and 18 follows the one presented in section 4. in in figure 4 , we present the results of erm, and the 3 overall best group-robust algorithms (irm, spectral decoupling, and lw-dro (v2)) across all eur-lex settings. lw-dro (v2) has comparable performance in the first two setting (small, medium), while being slightly better than irm in the large-sized setting. while lw-dro (v2) seems to control the trade-offs between irm and spectral decoupling, we believe that future work should better seek alternative directions with respect to algorithmic advances that possibly mitigate label degeneration and tackles label-wise disparities. ## conclusions & future work we considered one of the main challenges in largescale multi-label text classification, which comes from the fact that not all labels are well represented in the training set due to the class imbalance and the effect of temporal concept drift. to mitigate label disparities, we considered several group-robust optimization algorithms initially proposed to mitigate group disparities given specific attributes. experimenting with three datasets in two different settings, we empirically find that group-robust algorithms vastly improve performance considering macroaveraged measures, while two of the group-robust algorithms (invariant risk minimization and spectral decoupling) improve performance across all measures. considering a more well-suited neural method (lwan-bert), we observe a vast performance improvement using erm, leading to comparable overall results (µ-f 1 , m-rp) with the grouprobust algorithms; although is still outperformed considering m-f 1 . lastly, based on our understanding of what irm and spectral decoupling, the two best group-robust algorithms, offer, we introduced and evaluated a new algorithm, label-wise dro, which combines features from both, and one of its variants has comparable or better performance considering larger label sets. in the future, we would like to further investigate the two-tier anomaly (class imbalance and temporal concept drift). in this direction, we would like to directly take into consideration the time dimension by utilizing this information in group sampling and algorithms (e.g., groups over period of time). we would also like to consider data augmentation techniques (e.g., paraphrasing via masked-language modeling @xcite , and teacher forcing exploiting unlabeled data @xcite ) to improve the data (feature) sampling variability, as the group sampler used in group-robust algorithms over-sample minority classes with the same limited instances. further on, we would like to investigate the use of zero-shot lwan methods @xcite , which currently harm averaged performance in favor of improved worst case performance. label encodings based on contextualized word representations generated by pre-trained language models @xcite may mitigate the effect of using non-contextualized ones (e.g., word2vec). iulia turc, ming-wei chang, kenton lee, and kristina toutanova. 2019. well-read students learn better: the impact of student initialization on knowledge distillation. corr, abs/1908.08962. v. vapnik. 1992. principles of risk minimization for learning theory. in advances in neural information processing systems, volume 4. morgan-kaufmann. ronghui you, zihan zhang, ziye wang, suyang dai, hiroshi mamitsuka, and shanfeng zhu. 2019. atten-tionxml: label tree-based attention-aware deep model for high-performance extreme multi-label text classification. in advances in neural information processing systems, pages 5812-5822. a measuring class-wise bias @xcite recently introduced two evaluation measures to estimate class-wise bias of two models in comparison to one another in a multiclass setting, and show that these metrics can be also used to measure fairness and bias with respect to protected attributes. following @xcite , in figure 5 we present the normalized combined error variance (cev) in-between algorithms. cev estimates the class-wise bias of a model a relative to another model b has increased of the change between model a and a random predictor. for a detailed analysis of the cev metric, please refer to @xcite . in our case, as different models, we consider bert trained with a different algorithm. in both uk-lex and eur-lex, swapping group uniform, irm, or spectral decoupling with erm, or group dro leads to a higher class-wise bias, which is highly expected given the aforementioned performance analysis, i.e., improved m-f 1 scores.