Self-Training with Differentiable Teacher (2022)

Abstract: Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the teacher generates pseudo-labels, and the student makes predictions. The two models are updated alternatingly. However, such a straightforward alternating update rule leads to training instability. This is because a small change in the teacher may result in a significant change in the student. To address this issue, we propose DRIFT, short for differentiable self-training, that treats teacher-student as a Stackelberg game. In this game, a leader is always in a more advantageous position than a follower. In self-training, the student contributes to the prediction performance, and the teacher controls the training process by generating pseudo-labels. Therefore, we treat the student as the leader and the teacher as the follower. The leader procures its advantage by acknowledging the follower’s strategy, which involves differentiable pseudo-labels and differentiable sample weights. Consequently, the leader-follower interaction can be effectively captured via Stackelberg gradient, obtained by differentiating the follower’s strategy. Experimental results on semi- and weakly-supervised classification and named entity recognition tasks show that our model outperforms existing approaches by large margins.

## introduction self-training is a classic method that was first proposed for semi-supervised learning @xcite . it is also interpreted as a regularization method @xcite , and is extended to weakly-supervised learning and domain adaptation @xcite . the approach has gain popularity in many applications. for example, in conjunction with pre-trained language models @xcite , self-training has demonstrated superior performance on tasks such as natural language understanding @xcite , named entity recognition @xcite , and question answering @xcite . conventional self-training can be interpreted as a teacher-student framework. within this framework, a teacher model generates pseudo-labels for the unlabeled data. then, a student model updates its parameters by minimizing the discrepancy between its predictions and the pseudo-labels. the teacher subsequently refines its parameters based on the updated version of the student using predefined rules. such rules include minimizing a loss function @xcite , copying the student's parameters @xcite , and integrating models from previous iterations @xcite . the above procedures are operated iteratively. computationally, the alternating update procedure often causes training instability. such instability comes from undesired interactions between the teacher and the student. in practice, we often use stochastic gradient descent to optimize the student, and the noise of the stochastic gradient can cause oscillation during training. this means in a certain iteration, the student is optimized towards a certain direction; while in the next iteration, it may be optimized toward a drastically different direction. such a scenario renders the optimization ill-conditioned. moreover, the student model's gradient is determined by the pseudo-labels generated by the teacher. because of the training instability, a small change in the pseudo-labels may result in a substantial change in the student. to resolve this issue, we propose drift (differentiable self-training), where we formulate self-training as a stackelberg game @xcite . the concept arises from economics, where there are two players, called the leader and the follower. in a stackelberg game, the leader is always in an advantageous position by acknowledg-ing the follower's strategy. within the self-training framework, we grant the student a higher priority than the teacher. this is because the teacher serves the purpose of generating intermediate pseudolabels, such that the student can behave well on the task. the student (i.e., the leader) procures its advantage by considering what the response of the teacher (i.e., the follower) will be, i.e., how will the follower react after observing the leader's move. then, the leader makes its move, in anticipation of the predicted response of the follower. we remark that the stackelberg game formulation has also been used in other domains such as adversarial training @xcite . we highlight that in drift, the student has a higher priority than the teacher. in contrast, in conventional self-training, the two models are treated equally and have the same priority. when using conventional self-training, the student only reacts to what the teacher has generated. in differentiable self-training, the student recognizes the teacher's strategy and reacts to what the teacher is anticipated to response. in this way, we can find a better descent direction for the student, such that training can be stabilized. to facilitate the leader's advantage, our framework treats the follower's strategy (i.e., pseudolabels generated by the teacher) as a function of the leader's decision (i.e., the student's parameters). in this way, differentiable self-training can be viewed solely as a function of the student's parameters. therefore, the problem can be efficiently solved using gradient descent. besides pseudo-labels, the teacher can also generate sample weights @xcite @xcite . sample reweighting associates low-confidence samples with small weights, such that the influence of noisy labels can be effectively reduced. similar to pseudo-labels, sample weights and the student model are also updated iteratively. as such, we can further equip drift with differentiable sample weights. this can be achieved by integrating the weights as a part of the follower's strategy. we remark that our method is flexible and can incorporate even more designs to the follower's strategy. we evaluate the performance of differentiable self-training on a set of weakly-and semisupervised text classification and named entity recognition tasks. in some weakly-supervised learning tasks, our proposed method achieves com-petitive performance in comparison with fullysupervised models. for example, we obtain a 97.3% vs. 96.2% classification accuracy on yelp, and we do not use any labeled training data from the yelp dataset. we highlight that our proposed differentiable self-training approach is an efficient substitution to existing self-training methods. moreover, our method does not introduce any additional tuning parameter to the teacher-student framework. additionally, drift is flexible and can combine with various neural architectures. we summarize our contributions as the following: (1) we propose a differentiable self-training framework drift, which employs a stackelberg game formulation of the teacher-student approach. (2) we employ differentiable pseudo-labels and differentiable sample weights as the follower's strategy. our method alleviates the training instability issue. (3) extensive experiments on semi-supervised node classification, semi-and weakly-supervised text classification and named entity recognition tasks verify the efficacy of drift. ## method here n l is the number of labeled data, and n u is the number of unlabeled data. note that in weakly-supervised learning, we have unlabeled data because of the limited coverage of weak supervision sources. the difference between semiand weakly-supervised learning is that in the former case, the labels {y i } n l @xmath0, whereas in the latter case, the labels are noisy. the goal is to learn a classifier f : @xmath1, where @xmath2, @xmath3, • • • , c} is the label set, and c is the number of classes. the classifier f outputs a point in the c-dimensional probability simplex, where each dimension denotes the probability that the input belongs to a specific class. ## experiments we conduct two sets of experiments: weakly-and semi-supervised text classification. we also examine semi-supervised node classification on graphs (appendix a). all the results have passed a paired t-test with @xmath4.05. when using pre-trained language models, we employ a roberta-base @xcite model obtained from the hugging-face @xcite codebase. we implement all the methods using pytorch @xcite , and experiments are run on nvidia 2080ti gpus. all the training details are deferred to the appendix. ## discussion and conclusion in this paper, we propose a differentiable selftraining framework, drift, which formulates the teacher-student framework in self-training as a stackelberg game. the formulation treats the student as the leader, and the teacher as the follower. in drift, the student is in an advantageous position by recognizing the follower's strategy. in this way, we can find a better descent direction for the student and can stabilize training. empirical results on weakly-and semi-supervised natural language processing tasks suggest the superiority of drift to conventional self-training. conventional self-training is a heuristic and does not pose a well-defined optimization problem. in conventional methods, the teacher optimizes an im-plicit function through different components, e.g., pseudo-labels. we follow this convention and formulate self-training as a stackelberg game. our formulation is a principle that can motivate followup works. in our stackelberg game formulation (3), the student's utility function is the objective function of the minimization problem. the teacher's utility function is an implicit function, which can be written as the following: here, the first term d is some divergence between the teacher and the student (i.e., the kl-divergence in ( 8 )), and the two regularizers r 1 and r 2 are defined implicitly. that is, r 1 regularizes model confidence (realized by the soft pseudo-labels in ( 6 )), and r 2 regularizes model uncertainty (realized by the sample weights in ( 7 )). even though the utility function of the teacher is implicit, the solution of it is explicitly given, namely the teacher's strategy f (θ t ) in (3) is the solution to the teacher's implicit utility. because the strategy of the teacher is explicit (in contrast to implicitly defined by an optimization problem), the teacher's utility is maximized with such a strategy. thus, equilibrium of the stackelberg game exists, and every local optimum of the minimization problem is an equilibrium. in this work, we use off-the-shelf algorithms (adam and adamw) to find local minima of (3) (a.k.a. equilibria of the game). we remark that in a stackelberg game, the names "leader" and "follower" indicate the relative importance and priority of the two players. in our framework, we use the student model for prediction. therefore, the student model is more important than the teacher, so that we grant it a higher priority and say it is the leader. we remark that self-distillation @xcite is a special case of self-training, which is a supervised learning method. we can also differentiate the teacher model in self-distillation, such that drift can be extended to supervised learning. a semi-supervised learning on graphs datasets. we adopt three citation networks: cora, citeseer, and pubmed @xcite as benchmark datasets. their statistics are summarized in baselines. in addition to self-training, we adopt four graph neural network methods as baselines. note that self-training uses gcn as its backbone. settings. to demonstrate that differentiable selftraining can be effectively combined with different models, we adopt drift to two architectures: gcn, which is a graph convolution-based method; and grand, which is a mlp-based method that achieves state-of-the-art performance. results. experimental results are summarized in visualization of learned representations. figure 5 visualizes the learned representations of selftraining and drift. from fig. 5a , we can see that self-training mixes the representations of the red class and the blue class, as indicated in the red box. such erroneous classification is alleviated by drift (fig. 5b ). on citeseer, notice that selftraining generates a meaningless cluster (fig. 5c ), which is a sign that self-training overfits on the label noise.