Models Teaching Models: Improving Model Accuracy with Slingshot Learning (2023)

Abstract: One significant obstacle to the successful application of machine learning to real-world data is that of labeling: it is often prohibitively expensive to pay an ethical amount for the human labor required to label a dataset successfully. Human-in-the-loop techniques such as active learning can reduce the cost, but the required human time is still significant and many fixed costs remain. Another option is to employ pre-trained transformer models as labelers at scale, which can yield reasonable accuracy and significant cost savings. However, such models can still be expensive to use due to their high computational requirements, and the opaque nature of these models is not always suitable in applied social science and public use contexts. We propose a novel semi-supervised method, named Slingshot Learning, in which we iteratively and selectively augment a small human-labeled dataset with labels from a high-quality “teacher” model to slingshot the performance of a “student” model in a cost-efficient manner. This reduces the accuracy trade-off required to use these simpler algorithms without disrupting their benefits, such as lower compute requirements, better interpretability, and faster inference. We define and discuss the slingshot learning algorithm and demonstrate its effectiveness on several benchmark tasks, using ALBERT to teach a simple Naive Bayes binary classifier. We experimentally demonstrate that Slingshot learning effectively decreases the performance gap between the teacher and student models. We also analyze its performance in several scenarios and compare different variants of the algorithm.

## introduction in standard computational linguistics modeling, the typical strategy for achieving high model accuracy is to obtain a large fraction of high-quality human labels on the dataset at hand. however, it is not uncommon in applied social science settings to find that the scope of human labeling of a dataset is highly constrained @xcite . for example, it may be prohibitively expensive to pay an ethical amount for human labeling. alternatively, the project's institutional access to raw corpora may have ended, or perhaps the domain experts are no longer available. one solution is to use an active learning approach where a model is trained concurrently with the labeling process to "maximize a model's performance gain while annotating the fewest samples possible" @xcite . while this can be more efficient than unguided labeling, it still requires significant human labeling resources as the learning process is fundamentally human-guided. some issues are relatively fixed costs, such as bias, repeatability, and initial training requirements; this enforces a non-trivial lower bound on the efficiency of human-in-the-loop active learning methods. active learning also does not help if the data has already been collected, and there is no way to collect further data. alternatively, one might directly apply a large, pre-trained (and possibly fine-tuned) transformer model to obtain state-of-the-art outcomes @xcite @xcite . pre-trained models are well suited to learning from relatively small amounts of data due to the large amount of prior knowledge they already contain. but such a strategy comes with complications for applied social science, including high (and rising) costs @xcite @xcite ) a lack of model interpretability @xcite , and issues with repeatability when using models hidden behind an api. given the power of modern machine learning models, a natural question arises: can the power of modern machine learning architectures (such as transformer models) be harnessed to enhance the accuracy of standard models where human labeling is finished? here, we introduce and evaluate slingshot learning, an efficient, low-cost, widely applicable, and powerful methodology to enhance the accuracy of standard inference models. slingshot learning adopts a student-teacher framework, such that the standard model (the 'student') is iteratively taught by the transformer model (the 'teacher'). akin to the gravitational slingshot maneuver, in which a spacecraft uses the gravity of a large body to increase its own speed, the student model leverages the knowledge and performance of the teacher model, in a way that allows it to express where it expresses low certainty in its predictions, essentially asking "questions" of the teacher model. this iterative behavior can be modified, or entirely replaced with purely random behavior through the selection of hyperparameters. we demonstrate that, under slingshot learning, the accuracy of the student model can be at a relatively low cost. after the algorithm finishes iterating, the student model can be used as a compromise between the high performance of the complex teacher model, and the inherent benefits of the student model, which might include reproducibility, interpretability, computational efficiency, and compute costs that are orders of magnitude lower than the teacher model. this student model can then be used to label the rest of the corpus, and/or label future, unseen data. ## background active learning (al) seeks to minimize the human labeling cost of training a model while collecting enough data to successfully train the model @xcite . it has proven popular in recent years due to the prevalence of large, unlabelled datasets, and methods are increasingly leveraging powerful, pre-trained models such as bert @xcite , but active learning is still a fundamentally human-led labeling approach which adopts the pitfalls of experimental research involving humans. the ability to repeat experiments in the first place (let alone with any consistency), expand their scope and analyze the underlying models are all taken for granted in machine learning but, once humans are involved in the labeling process, the difficulty of such tasks can vary from non-trivial to impossible. iterative research design is also very difficult without significantly increasing the cost and time requirements of the overall experiment. large language models such as gpt-3 @xcite have achieved exceptional performances in nlp tasks under zero-shot and few-shot learner constraints, making these a compelling approach for low-resource tasks @xcite . gpts have been used in semi-supervised settings to generate pseudo-labels via training offline/student models @xcite . the study by @xcite leverages the use of gpt-3 to reduce costs in comparison to human annotators. in one of their labeling strategies, they employ gpt-3 (teacher) to generate a fully labeled dataset which is then used to train an offline model (student), a transformer model @xcite . while they show improved performance of the offline model on the fully labeled gpt-3 dataset in comparison to using human annotators, the performance on multiclass classifications was lower than that of human annotators and showed no improvement after increasing the number of gpt-3 labels. the authors also explore an active learning version, where the low-confidence labels of gpt-3 or the most challenging examples are relabelled using human annotators and the result is used to train the offline model. while this approach demonstrates better performance, it has the same pitfalls as other active learning approaches. our approach assumes a low budget and no access to human annotators beyond a small "seed" labeled dataset. in the realm of model interpretability, a similar approach is that of surrogate models (sm), which employ an interpretable simple model to explain the behavior of a more complex ml model @xcite . they include global models that explain the overall predictions and local models that provide local explanations on the predictors by varying their sampling strategies. the global models are trained on the same data set (or a similar data set) as the complex model @xcite . the output of the surrogate is then compared to the complex model to determine how well the surrogate replicates the complex model within an arbitrary threshold. the ones that employ sampling use simpler models to gain insight into interpretability through locally selected regions from the complex model's predictions @xcite . the goal or output here is interpretability and mostly requires the continued use of the underlying complex model. however, our intention is not just interpretability but to build a stable simple model that is standalone. the goal is to train the simpler model to make better predictions. further, we are limited by our training dataset size. the idea is to also leverage a largely unlabelled dataset into training. knowledge distillation (kd) is another wellknown approach that uses the teacher-student model @xcite . the teacher -a deep teacher network -distills its knowledge onto a smaller model. the smaller models are also neural nets, and draw feature learnings from the parameters and activations of the intermediate layers of the teacher model. the intention here is to deploy a high-performing, compressed, smaller model in a low-resource environment @xcite and thus they lack easy interpretability and transparency. while there have been some studies that use kd in an interpretable model @xcite , the data size limitation discussed earlier applies to this approach as well. our goal is to build a standalone model that can achieve high accuracy despite limited training data, through the use of large unlabelled data sets. ## slingshot learning here we explain slingshot learning from an intuitive view, and then present a more rigorous algorithmic definition. ## results we have included all experimental results in tabular form in the supplementary materials. source code can be found here. when performing these experiments, we ran the full slingshot learning algorithm until there is no more data left to classify. however, in practice, throughout these results, we refer to the δ ("delta") metric. intuitively, the delta metric δ s i is the ratio between the amount of knowledge that the teacher model has "taught" the student model (equal to the difference between the performance of s i and the baseline lq model), and the knowledge gap between the teacher and the student (equal to the difference in their accuracy). ## discussion these empirical results show slingshot learning to be an effective and reliable method for improving model performance in semi-supervised classification problems. while the exact values of δ vary between datasets, each dataset shows a clear correlation between the number of iterations and the performance of the resulting model, as well as (in most cases) the effect of diminishing returns as the number of iterations is increased. due to the degeneracy problem discussed below, we have elected to remove any runs for which the initial hq model did not show at least a 15% improvement in accuracy over the lq model baseline (before any iterations of the algorithm). foot_4 ## limitations our experiments were performed on english nlp datasets, collated from the internet. the results in this paper are intended and should be taken, purely in the context of evaluating the performance of the slingshot learning algorithm. we have no reason to believe slingshot learning would not be equally applicable to other languages, domains, problems, etc., and have simply chosen these datasets due to their prevalence and familiarity in the literature. we conjecture that different (perhaps more complex) language structures might make slingshot learning more useful. we do not have expertise in this area -as we discussed in the paper, we consider it necessary to run further experiments evaluating the application of slingshot learning in other domains to be necessary. we are hopeful, however, that the method can be applied to many problems besides the ones discussed in this paper. ## ethics statement this paper discusses methods that are intended to enhance the ability of researchers within domains such as social sciences to more easily apply state-of-the-art nlp algorithms to their work. this inherently comes with the risk of bias, and it is entirely possible that seed datasets based on incomplete or biased datasets will cause slingshot learning to serve as an "amplifier". we encourage caution when applying machine learning to social or other "real-world impactful" domains, and doubly so when extrapolating from smaller amounts of data using machine learning. this paper serves as an initial evaluation and high-level exploration of slingshot learning, and we would not recommend using slingshot learning in situations that could potentially have negative consequences. further exploration is certainly required, both specific to slingshot learning and regarding the more general use of machine learning in these scenarios. on the other hand, methods like slingshot learning can empower researchers and allow them to leverage tools that would otherwise be infeasible. it also has other benefits, which are discussed in the paper -for example, it facilitates taking a "snapshot" of a black box model (such as an api-based nlp model). it is our intention, and our hope, that slingshot learning is a useful tool that helps democratize machine learning within the social sciences and other applied domains.