T ASK -A DAPTIVE P RE -T RAINING FOR B OOSTING L EARNING W ITH N OISY L ABELS : A S TUDY ON T EXT C LASSIFICATION FOR A FRICAN L ANGUAGES
For high-resource languages like English, text classification is a well-studied task.
The performance of modern NLP models easily achieves an accuracy of more than 90% in many standard datasets for text classification in English (Xie et al., 2019;
Yang et al., 2019; Zaheer et al., 2020).
However, text classification in low-resource languages is still challenging due to the lack of annotated data.
Although methods like weak supervision and crowdsourcing can help ease the annotation bottleneck, the annotations obtained by these methods contain label noise.
Models trained with label noise may not generalize well.
To this end, a variety of noise-handling techniques have been proposed to alleviate the negative impact caused by the errors in the annotations (for extensive surveys see (Hedderich et al., 2021; Algan & Ulusoy, 2021)).
In this work, we experiment with a group of standard noisy-handling methods on text classification tasks with noisy labels.
We study both simulated noise and realistic noise induced by weak supervision.
Moreover, we find task-adaptive pre-training techniques (Gururangan et al., 2020) are beneficial for learning with noisy labels.
Problem Settings We consider a k -class text classification problem with noisy labels.
In particular, instead of accessing a training set D = { ( x i , y i ) ni
=1 } sampled from the clean data generation distribution, we work with a noisy version of D , denoted by ˆ D = { ( x i , ˆ y i ) ni
=1 } .
The goal is to train a model on ˆ D that generalizes well on the clean distribution.
Noise Simulation Research in this field often constructs a noisy dataset by injecting noise into a clean dataset (Han et al., 2018;
Jindal et al., 2019; Tänzer et al., 2021).
This simulates annotation scenarios such as crowdsourcing, where some annotators answer randomly or overlook some entries in multiple-choice questions.
It also allows controlling the noise level and type.
The noise in such a simulation is feature-independent.
That is, it assumes p (ˆ y | y
= i, x ) =
p (ˆ y | y = i ) .
Weak Supervision Noise In weak supervision, the labels are annotated in an automatic manner.
For example, Hedderich et al.
(2020) assign labels to a text document by leveraging word lists and a set of simple rules.
Unlike the simulated noise, the noise caused by weak supervision is featuredependent, meaning that the independence assumption does not hold.
Task-Adaptive Pre-Training (TAPT)
The BERT model is pre-trained on a large unlabeled, general-domain corpus.
Task-adaptive pre-training further pre-trains BERT on text from the downstream task.
In particular, given a task-domain corpus, one continues to train BERT with the Masked
Language Model (MLM) task and the Next Sentence Prediction (NSP) task.
Previous studies have shown that task-adaptive pre-training is beneficial for downstream NLP tasks with clean training data (Sun et al., 2019; Gururangan et al., 2020).
However, it remains unclear whether the noisy settings can inherit this benefit.
Data We simulate the noise on two established text classification dataset: AG-News (Zhang et al., 2015) and IMDB (Maas et al., 2011).
We study two simulated noise types, uniform (van Rooyen et al., 2015) and single-flip (Reed et al., 2015) noise.
We also evaluate two datasets with weak supervision noise in two low-resource African languages:
Hausa and Yorùbá (Hedderich et al., 2020).
The weak labels for these two datasets are generated by simple rules.
For example, to identify texts for the class “Africa”, a labeling rule based on a list of African countries and their capitals is used.
Baselines We compare learning without noise-handling with four popular noise-handling methods.
Co-Teaching : Han et al.
(2018) trains two networks to pick cleaner training subsets for each [Table] other.
2) Noise Matrix :
A noise transition matrix is appended after BERT’s prediction to transform the clean label distribution to the noisy one.
A variety of methods exists for estimating the noise matrix, i.e. Sukhbaatar et al.
; Bekker & Goldberger (2016);
Patrini et al.
; Hendrycks et al.
; Yao et al.
To exclude the effects of estimation errors in the evaluation, we use the ground truth transition matrix as it is the best possible estimation.
This matrix is fixed after initialization.
3): Noise Matrix with Regularization : Similar to Noise Matrix , Jindal et al.
(2019) appends a noise matrix after BERT’s output.
During training, the matrix is learned with an l 2 regularization and is not necessarily normalized to be a probability matrix.
This method is the current state-of-the-art method for text classification with simulated noise 4) Label Smoothing : label smoothing Szegedy et al.
(2016) is a commonly used method to improve model’s generalization and calibration.
It mixes the one-hot label with a uniform vector, preventing the model from getting overconfident on the samples.
Lukasik et al.
(2020) further shows that it improves noise robustness.
Finally, for the sake of comparison, we also include a model, denoted by No Validation , which neither applies noise-handling nor early-stopping.
Results We evaluate our baselines on four datasets with different noise levels and noise types.
We find that BERT is robust to simulated noise, especially under low to mild noise levels.
For example, the performance of BERT without noise-handling drops less than 5% under 60% uniform noise in
AG-News.
In these cases, noise-handling methods rarely outperform the baseline without noisehandling.
When the noise levels go higher, noise-handling starts benefiting learning.
However, the results are mixed.
For example, Co-Teaching outperforms the baseline without noise-handling on
IMDB with 45% single-flip noise, yet it underperforms in AG-News with 70% uniform noise.
Please refer to Appendix A for detailed performance numbers in each noise setting.
Task-adaptive pre-training offers an implicit way to assist BERT in all noise settings we investigate.
Table 1 presents the performance difference before and after adding TAPT.
Compared to the mixed results we see from applying noise-handling, TAPT helps consistently.
We also notice that the performance difference between different methods is reduced.
This could be particularly beneficial if computational resources are limited and trying out all noise-handling methods is not possible.
In this case, just using TAPT without a complex noise-handling architecture should already offer a strong baseline.
In summary, we observe that the benefit from noise-handling methods is rather limited and sometimes inconsistent, especially under weak noise.
TAPT, on the other hand, is a stable method to assist text classifiers when learning with noisy labels. A CKNOWLEDGMENTS
This work has been partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 232722074 – SFB 1102 and the EU Horizon 2020 projects ROXANNE under grant number 833635 and COMPRISE under grant agreement No.
A D ETAILED ON M ODEL P ERFORMANCE
We summarize our experimental results on simulated noise and weak supervision noise in Tableand 3, respectively
[Table] [Table]