Co-training for Low Resource Scientific Natural Language Inference (2024)

Abstract: Scientific Natural Language Inference (NLI) is the task of predicting the semantic relation between a pair of sentences extracted from research articles. The automatic annotation method based on distant supervision for the training set of SciNLI, the first and most popular dataset for this task, results in label noise which inevitably degenerates the performance of classifiers. In this paper, we propose a novel co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels, reflective of the manner they are used in the subsequent training epochs. That is, unlike the existing semi-supervised learning (SSL) approaches, we consider the historical behavior of the classifiers to evaluate the quality of the automatically annotated labels. Furthermore, by assigning importance weights instead of filtering out examples based on an arbitrary threshold on the predicted confidence, we maximize the usage of automatically labeled data, while ensuring that the noisy labels have a minimal impact on model training. The proposed method obtains an improvement of 1.5% in Macro F1 over the distant supervision baseline, and substantial improvements over several other strong SSL baselines. We make our code and data available on Github.

## introduction scientific natural language inference (nli) aims at predicting the semantic relation between a pair of sentences extracted from research articles. this task was recently proposed by @xcite as a natural language understanding (nlu) benchmark for scientific text along with a new dataset called scinli. the training set of scinli was constructed automatically using a distant supervision method @xcite . the labels assigned using this method contain strong signals, and achieved a macro f1 of ∼ 78% on the human annotated test set using pre-trained language models. nevertheless, despite the strong signals, the labels assigned based on distant supervision still contain noise that inevitably degenerates the performance of classifiers. the noisy labels can be particularly harmful for training large scale deep learning models because these models can achieve zero training loss even for mislabeled examples simply by memorizing them @xcite . semi-supervised learning (ssl) methods such as pseudo-labeling, i.e., self-training @xcite , consistency regularization @xcite , and cotraining @xcite have emerged as promising approaches that utilize a small amount of human annotated data, and learn from a large amount of unlabeled data. generally, these methods first train models with the limited human labeled data, and combine their most confident predictions (selected by applying a fixed high threshold, e.g., 0.9 on confidence) for unlabeled examples with the human labeled data for subsequent training. while a fixed high threshold ensures high quality pseudo-labels, it also ignores a large fraction of diverse examples that have correct pseudo-labels but with lower confidence. consequently, several dynamic thresholding mechanisms @xcite have been proposed to ensure a higher utilization of the pseudo-labels. however, a dynamic threshold still discards a large number of examples, compromises the pseudo-label quality, and is susceptible to error accumulation. in this paper, we propose a novel co-training approach for scientific nli that takes the signal from labels assigned based on distant supervision, but their quality is discerned by two classifiers simultaneously trained on different regions of the data map @xcite ) that guide each other in the learning process. unlike existing co-training approaches which also train two classifiers simultaneously in a cross-labeling manner by exchanging the most confident pseudo-labels, in our approach, the quality of each distantly supervised label is determined mutually by each classifier in the form of importance weights, which are exchanged between the two classifiers. thus, instead of exchanging the pseudo-labels, we exchange the classifiers' beliefs in the quality of the distantly supervised labels. in addition, we do not discard any examples but utilize all of them with different importance weights to ensure that noisy labels have a minimal impact on model training. the weights in our approach are estimated by monitoring the training dynamics to capture the behavior of the classifiers on each example-easy, hard, and ambiguous. specifically, we calculate the weights based on both average confidence, and variability of the probability predicted over the training epochs for each label assigned based on distant supervision. our weight calculation strategy ensures a high weight of the easy examples which are more likely to be clean, and a low weight of the hard examples which are more likely to be noisy for both classifiers; and a contrasting weight of the ambiguous examples by each classifier to encourage divergence between them. we explore our proposed approach by using a small amount of human annotated examples, and a large number of automatically annotated examples based on distant supervision from the scinli training set. our experiments show that the proposed approach improves the performance by more than 1.5% over distant supervision (i.e., models trained directly on the scinli training set); and obtains substantial improvements over co-training, co-teaching @xcite , and several other baselines designed for low-resource settings. our key contributions can be summarized as follows: semi-supervised learning self-training @xcite @xcite incorporates unlabeled data into model training by using three general steps: a) training a model using the available labeled examples; b) assigning pseudolabels to unlabeled examples based on the model's prediction; c) selecting a subset of pseudo-labeled examples based on some quality assurance measures and using them for further model training in addition to the labeled examples. consequently, the model gets exposed to more data which results in a better performance. co-training methods @xcite @xcite @xcite employ a similar approach as self-training to incorporate pseudo-labeled data into model training with a key difference. instead of training a single classifier as self-training, two classifiers are trained simultaneously which exchange their most confident pseudo-labels for further training. in contrast to the existing approaches, we do not discard any examples, and enable co-training by exchanging importance weights indicative of the distantly supervised label quality, calculated based on the classifiers' training dynamics. consistency regularization @xcite uses a consistency loss term (in addition to the supervised loss) that minimizes the distance between the predictions made by the model for different versions of the same unlabeled data. for example, fixmatch (sohn et al., 2020) first generates pseudo-labels for weak augmentations (e.g., replacing a random subset of tokens in a sentence with their synonyms) of unlabeled examples. a subset of these examples are then selected based on a fixed confidence threshold. fixmatch then uses strong augmentations (e.g., automatically translating a sentence to a different language and translating it back to the original language to get its paraphrased version) of the selected unlabeled examples as the input with these pseudo-labels as their target. recently, there have been a plethora of consistency regularization based methods that build on top of fixmatch. for example, flexmatch @xcite uses a label-wise dynamic thresholding method based on the learning status for each label. softmatch @xcite assigns soft weights to pseudo-labels based on the model confidence instead of filtering them out. although softmatch does not filter out any examples, the soft weights are assigned based only on the current training iteration. that is, similar to other ssl approaches, softmatch ignores the historical behavior of the classifier on the unlabeled examples. co-teaching the methods for learning from noisy labels generally rely on the classifier prediction during training to evaluate the quality of the labels. from this point of view, this research area is closely related to ssl. co-teaching @xcite is a method for training two classifiers simultaneously (similar to co-training) which exchange small-loss training instances (instances with low training loss) for further training. one of the key motivations behind training two classifiers is that each classifier will filter our different types of noise by learning complementary information. if the classifiers reach a consensus, they are no longer able to complement each other, reducing the learning process to single classifier training. therefore, a divergence between the classifiers throughout the training process is crucial. to this end, de-coupling @xcite and co-teaching+ @xcite use only the examples for which the classifiers disagree in their predictions. however, similar to ssl approaches, these methods also evaluate the label quality based only on the current classifiers' predictions; ignore the history based on their training dynamics; and discard a large number of examples. ## background we design our proposed approach based on our observations from the data map @xcite of scinli. in this section, we define the metrics used to characterize each example; plot them on a data map based on their characterizations; and discuss our observations from the data map that guide the development of our approach. notations consider a dataset @xmath0, y i )} @xmath1,...,n is a small human labeled training set of size n; and d @xmath2, ỹi )} @xmath3,...,m is an automatically labeled set of size m, constructed using distant supervision. dataset cartography dataset cartography @xcite is a tool for mapping and diagnosing a dataset by analyzing the training dynamics i.e., the model's behavior on each example during training. particularly, each example is characterized by three metrics-confidence, variability, and correctness, defined below. for an example (x, y), swayamdipta et al. (2020) defines the confidence as the mean of the probabilities predicted for its assigned label y over the training epochs {1, .., e}. that is, the confidence is calculated as follows: here, c θ is a function that calculates the confidence of an example (x,y) by a classifier θ. the variability is defined as the standard deviation of the predicted probability for the assigned label y over the training epochs. specifically, variability is calculated as follows: the correctness is calculated as the fraction of epochs in which the classifier predicts the assigned label y correctly. we train a roberta @xcite classifier on the full scinli training set d, and plot 55k randomly selected examples in figure 1 based on their training dynamics. note that, we plot only 55k examples (instead of all 101k) for clarity. the plot shows that the examples with high confidence and low variability (top left corner) have a high correctness. that is, the model's prediction consistently matches with the assigned label for these examples. therefore, they are easy to learn for the model, and the assigned labels are more likely to be correct. in contrast, the examples with low confidence and low variability (bottom left corner of the plot) have a low correctness. in other words, the model's predictions disagree with the assigned labels consistently. thus, the examples in this region are hard for the model to learn, and the assigned labels are more likely to be incorrect. the examples with a high variability (right side of the plot) shows a moderate correctness, and confidence. this indicates the model is uncertain about these examples, and their predictions occasionally match with the assigned labels. in our approach, we employ a weighting strategy that utilizes the information from the data maps to assign importance weights to distantly supervised examples during training. ## proposed approach we now present our co-training approach where we train two classifiers simultaneously that capture complementary information from the data. unlike the existing co-training approaches, we utilize all examples in dataset d a , and do not filter out anything. rather, our approach assigns importance weights to the automatically annotated examples. based on these weights, we decide the impact of each automatically annotated example in subsequent training of the classifiers. we devise the weights of the automatically labeled examples guided by the training dynamics of the classifiers in the form of confidence and variability of each example over the training epochs. furthermore, in contrast to the existing approaches which exchange pseudo-labels between classifiers, we exchange the weights calculated based on the historical behavior of the classifiers in evaluating the quality of each example. we ensure that the two classifiers learn complementary information by employing a weight calculation strategy that maintains their divergence. an overview of our approach can be seen in alg. 1. ## experiments this section describes the baselines, our models, experimental settings, and results. ## analysis our analysis consists of two parts. first, we perform various ablation experiments ( §6.1). next, we analyze the generalizability of our co-training approach to other nlp tasks ( §6.2). we also analyze the robustness of wct -cv by evaluating its out-of-domain performance using mscinli @xcite in appendix c. ## conclusion & future work in this paper, we propose a novel co-training approach for scientific nli. in contrast to the existing approaches, we assign importance weights to automatically annotated examples based on the historical training dynamics of the classifiers. instead of filtering out examples based on an arbitrary threshold, we decide the impact of the automatically annotated examples in subsequent training based on their assigned weights. we encourage divergence between the classifiers by assigning a contrasting weight to the ambiguous examples to train each classifier. our experiments show that the proposed approach obtains substantial improvements over the existing approaches. in the future, we will explore methods to further harness the training dynamics of the classifiers in improving the performance of ssl methods.