Learning under Label Proportions for Text Classification (2023)

Abstract: We present one of the preliminary NLP works under the challenging setup of Learning from Label Proportions (LLP), where the data is provided in an aggregate form called bags and only the proportion of samples in each class as the ground truth. This setup is inline with the desired characteristics of training models under Privacy settings and Weakly supervision. By characterizing some irregularities of the most widely used baseline technique DLLP, we propose a novel formulation that is also robust. This is accompanied with a learnability result that provides a generalization bound under LLP. Combining this formulation with a self-supervised objective, our method achieves better results as compared to the baselines in almost 87% of the experimental configurations which include large scale models for both long and short range texts across multiple metrics.

## introduction the supervised classification setting in machine learning usually requires access to data samples with ground truth labels and our goal is to then learn a classifier that infers the label for new data points. however, in many scenarios obtaining such a dataset with labels for each individual sample can be infeasible and thus calls attention to alternative training paradigms. in this work, we study the setup where we are provided with bags of data points and the proportion of samples belonging to each class (instead of the one-one correspondence between a sample and its label). the inference however, still needs to be performed at the instance level in order to extract meaningful predictions from the model. this setup, which following @xcite we refer to as learning from label proportions (llp), is attractive especially for at least two broad reasons @xcite . the first of these is privacy centric learning ( @xcite . with ever increasing de-mand of user privacy in almost all applications of digital mediums, the individual record (here the label) for each sample (say a user's document) can't be exposed and thus learning in a fully supervised manner deems infeasible. the most notable applications include medical data @xcite and e-commerce data @xcite , amongst various others. both of these contain abundant language data with multiple use cases of learning a classification model to perform disease prediction from patient's medical data and analyzing user's behavior for example, respectively. this makes llp a highly relevant learning paradigm to train models, both large and small, for data that needs k-anonymity. the second relevant property is weak supervision. since it is not always feasible to obtain clean data at a large scale, the training of the nlp models, both large and small, relies heavily on weakly supervised or self-supervised learning scrapped from the open web. llp can play a key role as obtaining data at an aggregated level (formal definition in section 2) can be a relatively easier and more feasible process. while there exist various prior works that have proposed new formulations to learn under this setting @xcite @xcite , many of them are either not readily applicable to learning deep models or very difficult to align with language data. furthermore, to our knowledge, there exists only one prior work of (ardehaly and culotta, 2016) that discusses llp for language data but is out of scope of this work as they focus on domain adaptation. we provide an elaborate discussion of the llp literature in section 5. in this work, we address some shortcomings of one of the first loss formulations proposed for learning deep neural networks under llp setup by @xcite , termed as dllp method. for a given bag, dllp optimizes the kl divergence between the ground truth bag level pro-portion and the aggregate of the predictions for the instances in the entire bag. in section 3, we highlight certain properties of dllp objective that can be highly undesirable for training deep networks. motivated by this, we propose a novel objective function that is a parametrization of the total variation distance (tvd), which itself is a lower bound to the kl via the pinsker's inequality. our formulation enjoys more functional flexibility because of the introduced parameter while retaining the outlier robustness property of the tvd. we also discuss some theoretical results for the proposed novel formulation. lastly, we combine our formulation with an auxiliary self-supervised objective that greatly aids in representation learning during the fine-tuning stage of the large scale nlp models experimented with. experimentally, we first demonstrate that the proposed formulation is indeed better and align with the theoretical motivation provided. in the main results, we demonstrate that our formulation achieves better results compared to the baselines in almost 87% of the 20 extensive configurations across 4 widely used models and 5 datasets. we observe up to 40% improvement in weighted precision metric on the bert model. in many cases, the improvements range between 2% to 9% which are highly substantial. further analysis including the bag sizes and hyperparameters also provide interesting insights into our formulation. to summarize, we have the following contributions: (i) a novel loss formulation that addresses the shortcomings of the previous work with supporting theoretical and empirical results. (ii) one of the preliminary works discussing the application of llp to natural language tasks. (iii) strong empirical results demonstrating that our method outperforms the baselines in most of the configurations. ## preliminaries we consider the standard multi-class classification setup with c classes (@xmath0, ..., c}). the input instances x are sampled from an unknown distribution over the feature space x . contrary to the availability of the data samples of the form (x, y), where @xmath1, to train the model in full supervision, we are provided a bag of input instances, @xmath2, with associated proportions of the labels ρ = (ρ 1 , ρ 2 , ..., ρ c ). the elements of ρ are defined as: it is important to note that the ground truth labels y i are not accessible and the entire information is contained in ρ. this ρ essentially provides the weak supervision for training the model. given the training bags along with the label proportions as the tuple (b i , ρ i ) , the objective is still to learn an instance level predictor that maps an instance x to the class distribution ∆ = {@xmath3. the predicted class is attained as: arg max @xmath4. we denote the classifier by f θ : @xmath5, which typically is a neural network such as bert with appropriate classification head. ## method since the supervision under the llp setup is coarse grained, given by ρ, the works in the literature (ardehaly and culotta, 2017), @xcite utilize training criteria based on the true proportions ρ i and predicted proportions ρi for the input bag b i . the predicted proportions ρi are typically some function of predicted class distributions f θ (x j ) for x @xmath6, ..., f θ (x |b i | )). the popular choice for g is the mean function and we retain the same in this work. a descriptive pipeline of the llp setup is provided in figure 1 . ## experiments to investigate the effectiveness of the novel loss formulation in eq. 6, we conduct extensive experiments across various models and datasets. these cover both short and long range texts as well as corresponding models developed to handle these texts. we begin by providing a brief overview of the setup. ## related work we discuss the literature of llp via the following taxonomy: shallow learning based: this is the category of models that doesn't leverage deep learning models. the notable works include @xcite (using svm, k-nn), @xcite (probabilistic models), @xcite (mutual contamination framework) etc. note that these methods do not scale for higher dimensional and larger datasets, thus making these unfit for language tasks. deep learning based: some of the notable works include (ardehaly and culotta, 2017) and @xcite which aim to minimize a divergence between the true proportions ρ and the predicted proportions of a given bag ρ. @xcite also used the concept of pseudo labeling @xcite , where an estimate of the individual labels within each bag is jointly optimized with the model during training. @xcite revisited pseudo labeling for llp by proposing a two-stage scheme that alternately updates the network parameters and the pseudo labels. however, both these works leveraging pseudo labeling perform experiments with computer vision datasets only. furthermore, open source codes could not be found thus making it difficult to compare against these formulations. auxiliary loss based: recent trends in learning with auxiliary losses alongside a primary objective also gained attention in the llp community. @xcite proposed llp-gan framework by incorporating generative adversarial networks (gan). @xcite proposed llp-vat that incorporated an additional consistency regularizer to learn consistent representations in the setting where the input instances are perturbed adversarially. it is noteworthy that both these methods leverage vision datasets for experiments. the tasks of generation and adversarial perturbation incur significant overhead for nlp and in various cases, this overhead can be larger than the primary objective task itself. for example, the generation of new instances via gans. @xcite proposed a contrastive self-supervised auxiliary loss to improve the representation learning of the underlying deep network in conjunction with the dllp loss. they augment this with an extra penalty term to further diversify the representations learned. lastly, to our knowledge, the only work that emphasizes the setting of llp for natural language classification tasks is the work by (ardehaly and culotta, 2016). however, it is out of the scope of this work as they aim to improve the performance of models under the llp setup when the training and testing data admit change in the underlying domain distributions, ie, the domain adaptation setting. ## conclusion we study the setup of learning under label proportions for natural language classification tasks. the llp setup admits two properties: privacy centric learning and weak supervision that render it as a highly practical modern research topic. we first discuss the shortcomings of one of the first llp loss formulations proposed for learning deep neural networks. these provide the motivation for our novel loss formulation, termed as l α t v ⋆ , that does not admit such irregularities. the formulation is also supported with theoretical justification and hardness results. empirically, we justify that the new loss function achieves better performance on diverse configurations. the main results achieved by combining this loss with a self-supervised contrastive formulation achieve significantly better performance as compared to the baselines.