WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding
Abstract
Building machine learning models for natural language understanding (NLU) tasks relies heavily on labeled data.
Weak supervision has been proven valuable when large amount of labeled data is unavailable or expensive to obtain.
Existing works studying weak supervision for NLU either mostly focus on a specific task or simulate weak supervision signals from ground-truth labels.
It is thus hard to compare different approaches and evaluate the benefit of weak supervision without access to a unified and systematic benchmark with diverse tasks and real-world weak labeling rules.
In this paper, we propose such a benchmark, named WALNUT 1 , to advocate and facilitate research on weak supervision for NLU.
WALNUT consists of NLU tasks with different types, including document-level and token-level prediction tasks.
WALNUT is the first semi-weakly supervised learning benchmark for NLU, where each task contains weak labels generated by multiple real-world weak sources, together with a small set of clean labels.
We conduct baseline evaluations on WALNUT to systematically evaluate the effectiveness of various weak supervision methods and model architectures.
Our results demonstrate the benefit of weak supervision for low-resource NLU tasks and highlight interesting patterns across tasks.
We expect WALNUT to stimulate further research on methodologies to leverage weak supervision more effectively.
The benchmark and code for baselines are available at aka.ms/walnut_benchmark .
Introduction
To tackle natural language understanding (NLU) tasks via supervised learning, high-quality labeled examples are crucial.
Recent advances on large pre-trained language models (Peters et al., 2018;
Devlin et al., 2018; Radford et al., 2019) lead to impressive gains on NLU benchmarks, including
[Figure]
Text - ... Text  ...
Labels
Text Text
Text
WALNUT
S e
m i - w e a k l y
Sup er v
i s e d
L e a r n i n g
Few Labeled Data
[Figure]
GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), at the assumption that large amount of labeled examples are available.
For many realworld applications, however, it is expensive and time-consuming to manually obtain large-scale high-quality labels, while it is relatively easier to obtain auxiliary supervision signals, or weak supervision , as a viable source to boost model performance without expensive data annotation process.
Learning from weak supervision for NLU tasks is attracting increasing attention.
Various types of weak supervision have been considered, such as knowledge bases (Mintz et al., 2009; Xu et al., 2013), keywords (Karamanolakis et al., 2019; Ren et al., 2020), regular expression patterns (Augenstein et al., 2016), and other metadata such as user interactions in social media (Shu et al., 2017).
Also, inspired by recent advances from semi-supervised learning, semi-weakly supervised learning methods which leverage both a small set of clean labels and a larger set of weak supervision (Papandreou et al., 2015; Hendrycks et al., 2018; Shu et al., 2019; Mazzetto et al., 2021; Karamanolakis et al., 2021; Maheshwari et al., 2021; Zheng et al., 2021) are emerging to further boost task performance.
However, a unified and systematic evaluation benchmark supporting both weakly and semiweakly supervised learning for NLU tasks is rather limited.
On the one hand, many existing works only study specific NLU tasks with weak supervision, thus evaluations of proposed techniques leveraging weak supervision on a small set of tasks do not necessarily generalized onto other NLU tasks.
On the other hand, some works rely on simulated weak supervision, such as weak labels corrupted from ground-truth labels (Hendrycks et al., 2018), while real-world weak supervision signals can be far more complex than simulated ones.
Furthermore, existing weakly and semi-weakly supervised approaches are evaluated on different data with different metrics and weak supervision sources, making it difficult to understand and compare.
To better advocate and facilitate research on leveraging weak supervision for NLU, in this paper we propose WALNUT (Figure 1), a semi-weakly supervised learning benchmark of NLU tasks with real-world weak supervision signals.
Following the tradition of existing benchmarks (e.g., GLUE), we propose to cover different types of NLU tasks and domains, including document-level classification tasks (e.g., sentiment analysis on online reviews, fake news detection on news articles), and token-level classification tasks (e.g., named entity recognition in news and biomedical documents).
WALNUT provides few labeled and many weakly labeled examples (Figure 1) and encourages a consistent and robust evaluation of different techniques, as we will describe in Section 3.
In addition to the proposed benchmark, in Section 4.2 we shed light on the benefit of weak supervision for NLU tasks in a collective manner, by evaluating several representative weak and semiweak supervision methods for and several base models of various sizes (e.g., BiLSTM, BERT,
RoBERTa) , leading to more than 2,000 groups of experiments.
Our large-scale analysis demonstrates that weak supervision is valuable for low-resource
NLU tasks and that there is large room for performance improvement, thus motivating future research.
Also, by computing the average performance across tasks and model architectures, we show surprising new findings.
First, simple techniques for aggregating multiple weak labels (such as unweighted majority voting) achieve better performance than more complex weak supervision paradigms.
Second, weak supervision has smaller benefit in larger base models such as RoBERTa, because larger pre-trained models can already achieve impressively high performance using just a few clean labeled data and no weakly labeled data at all.
We identify several more challenges on leveraging weak supervision for NLU tasks and shed light on possible future work based on WALNUT.
The main contributions of this paper are: (1) We propose a new benchmark on semi-weakly supervised learning for NLU, which covers eight established annotated datasets and various text genres, dataset sizes, and degrees of task difficulty; (2)
We conduct an exploratory analysis from different perspectives to demonstrate and analyze the results for several major existing weak supervision approaches across tasks; and (3) We discuss the benefits and provide insights for potential weak supervision studies for representative NLU tasks.
Related Work 2. Weak Supervision for NLU Document-level classification
Existing works on weakly supervised learning for document-level classification attempt to correct the weak labels by incorporating a loss correction mechanism for text classification (Sukhbaatar et al., 2014; Patrini et al., 2017).
Other works further assume access to a small set of clean labeled examples (Hendrycks et al., 2018;
Ren et al., 2018; Varma and Ré, 2018;
Shu et al., 2020b).
Recent works also consider the scenario where weak signals are available from multiple sources (Ratner et al., 2017;
Meng et al., 2018;
Ren et al., 2020) to exploit the redundancy as well as the consistency in the labeling information.
Despite the recent progress on weak supervision for text classification, there is no agreed upon benchmark that can guide future directions and development of NLU tasks in semi-weakly supervised setting.
Token-level classification
Weak supervision has also been studied for token-level classification (sequence tagging) tasks, focusing on Named Entity
Recognition (NER).
One of the most common approaches is distant supervision (Mintz et al., 2009), which uses knowledge bases to heuristically annotate training data.
Besides distant supervision, several weak supervision approaches have recently addressed NER by introducing various types of labeling rules, for example based on keywords, lexicons, and regular expressions (Fries et al., 2017;
Ratner et al., 2017; Shang et al., 2018; Safranchik et al., 2020; Lison et al., 2020; Li et al., 2021).
WALNUT integrates existing weak rules into a unified representation and evaluation format.
2. NLU Benchmarks
Accompanying the emerging of large pre-trained language models, NLU benchmarks has been a focus for NLP research, including GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019).
On such benchmarks, the major focus is put on obtaining best possible performance (He et al., 2020) under the full training setting, which assumes that a large quantity of manually labeled examples are available for all tasks.
Few-shot NLU benchmarks exist (Schick and Schütze, 2021; Xu et al., 2021; Ye et al., 2021; Mukherjee et al., 2021), however these do not contain weak supervision. Though research in weak supervision in NLU has gained significant interest (Hendrycks et al., 2018; Shu et al., 2019; Zheng et al., 2021), most of these work either focus on a small set of tasks or simulate weak supervision signals from ground-truth labels, hindering its generalization ability to real-world NLU tasks.
The lack of a unified test bed covering different NLU task types and data domains motivates us to construct such a benchmark to better understand and leverage semi-weakly supervised learning for NLU in this paper.
Different from existing work based on crowdsourcing (Hovy et al., 2013; Gokhale et al., 2014) to obtain noisy labels, we focus specifically on the semi-weakly supervised learning setting, where we collect tasks with weak labels obtained from human-written labeling rules.
(Zhang et al., 2021) is concurrent work that also features weak supervision for various (not necessarily text-based) tasks and assumes a purely weakly supervised setting, i.e., no clean labeled data is available.
In contrast,
WALNUT focuses on NLU tasks under a morerealistic semi-weakly supervised setting and, as we show in Section 3, a small amount of clean labeled data plays an important role in determining the benefit of weak supervision for a target task.
WALNUT 3. Benchmark Construction Principles
We first describe the design principles guiding the benchmark construction.
Task Selection Criterion
We aim to create a testbed which covers a broad range of NLU tasks where real-world weak supervision signals are available .
To this end, WALNUT includes eight
English text understanding tasks from diverse domains, ranging from news articles, movie reviews, merchandise reviews, biomedical corpus, wikipedia documents, to tweets.
The eight tasks are categorized evenly into two types, namely document classification and token classification (sequence labeling).
It’s worth noting that we didn’t create any labeling rules ourselves to avoid bias, but rather opted with labeling rules which already exist and are extensively studied by previous research.
Therefore, WALNUT does not include other NLU tasks, such as natural language inference and question answering, as we are not aware of previous research with human labeling rules for these tasks.
Semi-weakly Supervised Learning Setting
While many previous works studied weak supervision in a purely weakly supervised setting, recent advances in few-shot and semi-supervised learning suggest that a small set of cleanly labeled examples together with unlabeled examples greatly helps boosting the task performance.
Though large scale labeled examples for a task is difficult to collect, we acknowledge that it’s rather practical to collect a small set of labeled examples.
In addition, recent methods leveraging weak supervision also demonstrate greater gains of combining a small set of labeled examples with large weakly labeled examples (Hendrycks et al., 2018; Shu et al., 2019; Zheng et al., 2021).
Therefore, WALNUT is designed to emphasize the semi-weakly supervised learning setting.
Specifically, each dataset contains both a small number of clean labeled instances and a large number of weakly-labeled instances.
Each weakly-labeled instance comes with multiple weak labels (assigned by multiple rules) and a single aggregated weak label derived from weak rules.
Note that this way WALNUT can be naturally used to support the conventional weakly supervised setting by ignoring the provided clean labels.
Consistent and Robust Evaluation
To address discrepancies in evaluation protocols from existing research on weak supervision and to better account for the small set of clean examples per task,
WALNUT is constructed to promote systematic and robust evaluations across all eight tasks.
Specif-
[Table]
ically, for each task, we first determine the number of clean examples to sample with pilot experiments (with the rest treated as weakly labeled examples by applying the corresponding weak labeling rules), such that the weakly supervised examples can be still helpful with the small clean examples present (typically 20-50 per class; see Sec. 3.3 for details); second, to consider sampling uncertainty, we repeat the sampling process for the desired number of clean examples 5 times and provide all 5 splits in WALNUT.
Methods on WALNUT are expected to be using all 5 pre-computed splits and reporting the mean and variance of its performance.
To summarize, WALNUT can facilitate research on weakly- and semi-weakly supervised learning by offering the following:
• Eight NLU tasks from diverse domains; • For each task, five pairs of clean and weakly labeled samples for robust evaluation; •
For each individual weakly labeled example, all weak labels from multiple rules and a single aggregated weak label.
. Task Categories
Here, we describe the eight tasks in WALNUT (Table 1), grouped into four document-level classification tasks (Section 3.2.1) and four token-level classification tasks (Section 3.2.2).
3.2. Document-level Classification
The goal of document-level classification tasks is to classify a sequence of tokens x 1 , . . .
, x N to a class c ∈ C , where C is a pre-defined set of classes.
We consider binary and multi-class classification problems from different application domains such as sentiment classification (Zhang et al., 2015), fake news detection (Shu et al., 2020c), and topic classification (Zhang et al., 2015).
Concretely, we include the following widely-used document-level text classification datasets: AGNews (Zhang et al., 2015), Yelp (Zhang et al., 2015), IMDB (Maas et al., 2011) and GossipCop (Shu et al., 2020a).
For Yelp, IMDB, and AGNews, the weak rules are derived from the text using keyword-based heuristics, third-party tools as detailed in (Ren et al., 2020).
For GossipCop, the weak labeling rules are derived from social context information accompanying the news articles, including related users’ social engagements on the news items (e.g., user comments in Twitter).
For example, a weak labeling rule for fake news can be “If a news piece has a standard deviation of user sentiment scores greater than a threshold, then the news is weakly labeled as fake news. ”
(Shu et al., 2020c).
3.2. Token-level Classification
The goal of token-level classification tasks is to classify a sequence of tokens x 1 , . . .
, x N to a sequence of tags y 1 , . . .
, y N ∈ C ′ , where C ′ is a pre-defined set of tag classes (e.g., person or organization).
As one of the most common tokenlevel classification tasks, Named Entity Recognition (NER) deals with recognizing categories of named entities (e.g., person, organization, location) and is important in several NLP pipelines, including information extraction and question answering.
We include in WALNUT the following four
NER datasets from different domains, for which weak rules are available:
CoNLL (Sang and
De Meulder, 2003), the NCBI Disease corpus (Do˘gan et al., 2014), WikiGold (Balasuriya et al., 2009) and the LaptopReview corpus (Pontiki et al., 2016) from the SemEval 2014 Challenge.
For the CoNLL and WikiGold dataset, we use weak rules provided by (Lison et al., 2020).
For the
NCBI and LaptopReview dataset, we use weak rules provided by (Safranchik et al., 2020). [Figure] [Figure] ) IMDB
[Figure]
c) Yelp
[Figure] [Figure] LaptopReview 3. Dataset Pre-Processing
To construct a semi-weakly supervised learning setting, we split the training dataset for each task into a small subset with clean labels ( D C ) and a large subset with weak labels ( D W ).
For robust evaluation, we create five different clean/weak train splits as we noticed that the model performances may vary with different clean train instances.
The validation/test sets are always the same across splits.
Because of different dataset characteristics (e.g., differences in number of classes, difficulty), we choose the size for D C per dataset via pilot studies.
(After having selected the instances for the
D C , we consider the remaining instances as part of the D W split.)
We defined the size of D C such that we demonstrate the benefits of weak supervision and at the same time leave substantial room for improvement in future research.
To this end, we compare the performances of the same base classification model (e.g., BiLSTM), trained using only D C (“Clean” approach)
v.s. using both D C and D W (“Clean+Weak” approach).
As shown in
Figure 2, for each dataset, we choose a small size of D C , such that the “Clean+Weak” approach has a substantially higher F1 score than the “Clean” approach and at the same time the “Clean” approach has no trivial F1 score.
The statistics of the pre-processed datasets included in WALNUT are shown in Table 1.
Baseline Evaluation in WALNUT
In this section, we describe the baselines and evaluation procedure (Section 4.1), and discuss evaluation results in WALNUT (Section 4.2).
Our results highlight the value of weak supervision, important differences across different baselines, and the potential utility of WALNUT for future research on weak supervision.
4. Baselines and Evaluation Procedure
We evaluate several baseline approaches in
WALNUT by considering different base models (text encoders) and different (semi-)weakly supervised learning methods to train the base model.
Encoder Models
To encode input text, we experiment with various text encoders, ranging from shallow LSTMs to large pre-trained transformerbased encoders (Vaswani et al., 2017).
In particular, we consider a series of models with increasing model size: Bi-directional LSTM with
Glove embeddings (Pennington et al., 2014), DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), BERT-large, and RoBERTa-large.
For each text encoder, a classification head is placed on top of the encoder to perform the task.
For more details on the base model configurations see Appendix A.1.
Learning Methods
Given the semi-weakly supervised setting in WALNUT, we evaluate eight supervision approaches in the following categories:
• Learning from clean labeled examples only .
The model is trained on only the small amount of available clean examples D C , a naive baseline method leveraging no weak supervision, which we denote as C .
• Learning from weakly labeled examples only .
The model is trained on all weakly labeled examples D W .
To produce a single weak label from the multiple labeling rules for training, we aggregate the rules via two methods: majority voting (denoted by W ) and Snorkel (Ratner et al., 2017) (denoted by Snorkel ).
• Learning from both clean and weakly labeled examples .
The model is trained with both D C and D W in a weakly-supervised setting.
The first two baselines in this category is simply concatenating D C and the aggregated weak labels (from either W and Snorkel ), and the model is trained on the combination.
We denote these two as C+W and C+Snorkel , respectively.
We also test three recent semiweakly supervised learning methods which proposed better ways to leverage both D C and D W : GLC which is a loss correction approach (Hendrycks et al., 2018), MetaWN which is a meta-learning approach to learn the importance of weakly labeled examples (Shu et al., 2019;
Ren et al., 2018) and MLC , a meta-learning approach to learn to correct the weak labels (Zheng et al., 2021).
To establish an estimate of the ceiling performance on WALNUT, for each task we also train with all clean training examples in the original dataset (denoted by Full Clean ).
Experimental Procedure
For a robust evaluation, we repeat each experiment five times on the five splits of D C and D W (clean and weak examples for each task; see Section 3.3), and report the average scores and the standard deviation across the five runs.
In WALNUT, we report the average micro-average F1 score on the test set.
2 Datasets and code for WALNUT are publicly available at aka.ms/walnut_benchmark .
4. Experimental Results and Analysis
Table 2 shows the main evaluation results on
WALNUT.
Rows correspond to supervision methods for the base model, columns correspond to tasks, and each block corresponds to a different base model.
Unless explicitly mentioned, in the rest of this section we will compare approaches based on their average performance across tasks (rightmost column in Table 2).
As expected, training with Full Clean achieves the highest F1 score, corresponding to the highresource setting where all clean labeled data are available.
Such method is not directly comparable to the rest of the methods but serves as an estimate of the ceiling performance for WALNUT.
Training with only limited clean examples achieves the lowest overall F1 score: in the low-resource setting, which is the main focus in WALNUT, using just the available clean subset ( D C ) is not effective.
Weak supervision is valuable for low-resource
NLU.
“W” and “Snorkel” achieve better F scores than “C” for many base models: even using only weakly-labeled data in D W is more effective than using just D C , thus demonstrating that simple weak supervision approaches can be useful in the low-resource setting.
Approaches such as “C+W” and “C+Snorkel” lead to further improvements compared to “C” and “Snorkel”, thus highlighting that even simple approaches for integrating clean and weak labeled data (here by concatenating
D C and D W ) are more effective than considering each separately.
There is no clear winner in WALNUT.
Our results in Table 4.2 indicate that the performance of weak supervision techniques varies substantially across tasks.
Therefore, it is important to evaluate such techniques in a diverse set of tasks to achieve a fair comparison and more complete picture of their performance.
The performance of various techniques also varies across different splits (See
Table 14 in Appendix for variances of all experiments).
Interestingly, “C+W” and “C+Snorkel” sometimes perform better than more complicated approaches, such as GLC, MetaWN and MLC.
Larger base models achieve better overall performance.
We further aggregate statistics across tasks, methods, and base models in Table 3.
The bottom row reports the average performance across methods for each base model and leads to a consistent ranking in F1 score among base models: BiLSTM ≤ DistilBERT ≤ BERT-base ≤ RoBERTabase.
Observing higher scores for larger transformer models such as RoBERTa agrees with previous observations (Brown et al., 2020).
Interestingly, switching from BERT-base to BERT-large (and from RoBERTa-base to RoBERTa-large) in base model architecture leads to marginal improvement, suggesting the need to explore more effective learning methods leveraging weak supervision.
[Table]
Clean
[Table]
Snorkel 76.
GLC 68.
MetaWN 72.
MLC 73.
RoBERTa-base (125M parameters)
Full Clean 92.
C 84.
W 66.
Snorkel 71.
C+W 70.
C+Snorkel 74.
GLC 67.
MetaWN 69.
MLC 70.
BERT-large (336M parameters) Full Clean 92. C 72. W 68.
Snorkel 73.
C+W 73.
C+Snorkel 73.
GLC 67.
MetaWN 71.
RoBERTa-large (355M parameters)
Full Clean 93.
C 86. W 74. Snorkel 75. C+W 71.
C+Snorkel 74.
GLC 67.
MetaWN 68.
Rules (no base model)
Rules 61.
[Table]
[Table] Weak supervision has smaller benefit in larger base models. Another question that we attempt to address in WALNUT is on whether weak supervision equally benefits each base model architecture. To quantify such benefit we compare the performance differences between models trained using semi-weak supervision and models trained using clean data only.
The “Weak Sup” approach in Table 4 is computed as the average F1 score across all semi-weak supervision methods (C+W,
C+Snorkel, GLC, and MetaWN).
The performance gap between “Weak Sup” and “C” (training with few clean data only) is smaller for larger models.
Additionally, the performance gap between “Full
Clean” (full clean data training) and “Weak Sup” approach is larger for larger models.
The two above observations highlight that weak supervision has smaller benefit in larger models.
An important future research direction is to develop better learning algorithms and improving the effectiveness of weak supervision in larger models.
Analysis of weak rules.
For now, we have focused on the evaluation of base models trained using weak labels generated by multiple weak labeling rules.
It is interesting also to decouple the base model performance from the rule aggregation technique (e.g., majority voting, Snorkel) that was used to generate the training labels, which is an essential modeling component for weak supervision.
The bottom row in Table 2 (“Rules”) reports the test performance of rules computed by taking the majority voting of weak labels on the test instances.
(For test instances that are not covered by any rules, a random class is predicted.)
Such majority label is available in our pre-processed datasets.
Interestingly, “Rules” sometimes outperforms base models trained using weak labels (“W”, “Snorkel”).
Note however that “Rules” assumes access to all weak labels on the test set, which might not always be available.
On the other hand, the base model learns text features beyond the heuristic-based rules and does not require access to rules during test time and thus can be applied for any test instance.
For a more in-depth analysis of the rule quality,
WALNUT also supports the analysis of individual rules and multi-source aggregation techniques, such as majority voting or Snorkel.
Figure 3 shows a precision-recall scatter plot for each rule on each of the dataset in WALNUT.
For instance, in the
CoNLL dataset rules vary in characteristics, where most rules have a relatively low recall while there are a few rules that have substantially higher recall than the rest. Across datasets, we observe that rules
[Figure]
have higher precision than recall, as most rules are sparse, i.e., apply to a small number of instances in the dataset (e.g., instances containing a specific keyword). Similar trends are observed on other datasets as well.
For detailed descriptions of all weak rules in all datasets, refer to Table 6 - 13 in the appendix.
Conclusions
Motivated by the lack of a unified evaluation platform for semi-weakly supervised learning for lowresource NLU, in this paper we propose a new benchmark WALNUT covering a broad range of data domains to advocate research on leveraging both weak supervision and few-shot clean supervision.
We evaluate a series of different semi-weakly supervised learning methods with different model architecture on both document-level and tokenlevel classification tasks, and demonstrate the utility of weak supervision in real-world NLU tasks.
We find that no single semi-weakly supervised learning method wins on WALNUT and there is still gap between semi-weakly supervised learning and fully supervised training.
We expect WALNUT to enable systematic evaluations of semi-weakly supervised learning methods and stimulate further research in directions such as more effective learning paradigms leveraging weak supervision.
Acknowledgements
We would like to thank Yichuan Li for helping adapt several weak supervision baselines for document-level classification tasks, and the reviewers for constructive feedback.
A Appendix Limitations and Broader Impact
The proposed benchmark is likely to stimulate research on weakly supervised learning for NLU, and offer the research community on weakly supervised learning a unified testbed for evaluating new methodologies developed for low-resource NLU.
For many practical NLU applications, large amount of manually labeled data is unavailable or expensive to obtain due to either cost or privacy concerns, resorting to proxy signals such as weak supervision is a viable solution to mitigate this annotation scarce problem.
We hope WALNUT would provide such an evaluation environment to advocate progress in this direction.
Limitations.
Due to the lack of existing realworld weak supervision for many NLU tasks,
WALNUT does not include NLU tasks such as
Natural Language Inference for which it is hard to construct weak supervision rules.
Also, currently
WALNUT only considers English data; a possible extension is to also include multi-lingual corpus with weak supervision available to boost the performance of multi-lingual language models with weakly supervised learning.
A. Implementation Details
We implement all baseline experiments with PyTorch and each experiment runs on a single
NVIDIA GPU.
Below are hyper-parameter specifications for all baseline methods (hyperparameters not mentioned below are given default values):
• Full Clean, C, C+W, Snorkel, C+Snorkel: Batch size is 32 for document-level classification datasets
and for the token
level classification datasets.
The code for Snorkel is adapted from: https://github.com/ snorkel-team/snorkel .
Each training experiment is conducted for the 10 epochs with the checkpoint with the best validation performance saved for evaluation on the test set.
• GLC: Code is adapted from https:// github.com/mmazeika/glc .
Batch
size is 16 for the 4 document-level classification datasets and 8 for the 4 token-level classification datasets
Each experiment trains for
10 epochs with the checkpoint with the
best validation performance saved for evaluation on test set.
• MetaWN: Code is adapted from https://github.com/xjtushujun/ meta-weight-net .
Batch size is 8 for the 4 document-level classification datasets and 4 for the 4 token-level classification datasets.
The meta-network is a three-layer feed-forward network with hidden dimension of 128.
Each experiment trains for 10 epochs with the checkpoint with the best validation performance saved for evaluation on test set.
• MLC: Code is adapted from https:// github.com/microsoft/MLC .
Batch size is 8 for the 4 document-level classification datasets and 4 for the 4 token-level classification datasets.
The meta-network is a three-layer feed-forward network with hidden dimension of 128; the label embedding dimension used in the meta-network is 64.
Each experiment trains for
10 epochs with the checkpoint with the best validation perfor-
mance saved for evaluation on test set.
To encode input text we experiment with various text encoders, ranging from shallow LSTMs to large pre-trained transformerbased encoders (Vaswani et al., 2017): • BiLSTM-based encoder: the BiLSTM implementations are all based on 50-dimensional pre-trained glove word embeddings (Pennington et al., 2014) and bi-directional LSTMs with hidden size 128.
Note that our implementation is different than other BiLSTM implementations used by previous work, which are based on 100-dimensional word embeddings and LSTM hidden size 300. This renders our BiLSTM models roughly 3 times smaller than those used by previous work, thus the numbers are not directly comparable. We chose a smaller model capacity for BiLSTMs to contrast the performance with larger models including DistilBERT and others to show the importance of model capacity on WALNUT. During training, we use a learning rate of 0.
for all BiLSTM-based models. • Transformer-based encoders: we consider pre-trained DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), BERT-large, and RoBERTa-large.
We fine-tune these models (via the huggingface library) using task-specific classification heads on top of the encoder and a learning rate of 0.00001. B Additional Benchmark Details B. Document-level classification
• AGNews: and multi-class topic classification (world vs. sports vs. business vs. sci/tech) on news articles from the AGNews dataset (Zhang et al., 2015).
• Yelp: binary sentiment classification (negative vs. positive) of Yelp restaurant reviews (Zhang et al., 2015).
• IMDB: binary sentiment classification (negative vs. positive) of IMDB movie reviews (Maas et al., 2011).
• GossipCop: binary fake news detection (fake vs. not fake) on news articles from the GossipCop 3 fact-checking websites.
The GossipCop dataset is part of the fake news detection benchmark FakeNewsNet (Shu et al., 2020a).
(We only include the results of Gossipcop to represent fake news classification task as the results for Politifact are similar.)
B. Token-level classification
According to the BIO tagging scheme, “B,” “I,” and “O,” represent the beginning, inside, and outside, of a named entity span, respectively.
(Not extracting any values corresponds to a sequence of “O”-only tags.)
Consider, for example, named entity recognition in the CoNLL dataset:
[Table]
• CoNLL: the CoNLL 2003 dataset (Sang and De Meulder, 2003) contains news articles from Reuters (split into sentences).
In total, there are 35,089 entities from 4 types: organization (ORG), person (PER), location (LOC), and miscellaneous (MISC).
Tag classes C ′ :
[’O’, ’B-PER’, ’I-PER’, ’B-ORG’, ’I-ORG’, ’B-LOC’, ’I-LOC’, ’B-MISC’, ’I-MISC’] • NCBI: the NCBI Disease corpus (Do˘gan et al., 2014) contains PubMed abstracts with 6,866 disease mentions.
Tag types: [’O’, ’B’, ’I’] • WikiGold: the WikiGold dataset (Balasuriya et al., 2009) contains English Wikipedia articles that were randomly selected and manually annotated with the same entity types as CoNLL.
Tag classes C ′ :
[’O’, ’B-PER’, ’IPER’, ’B-ORG’, ’I-ORG’, ’B-LOC’, ’I-LOC’, ’B-MISC’, ’I-MISC’] • LaptopReview: the Laptop Review corpus from the SemEval 2014 Challenge (Pontiki et al., 2016) contains 3,012 mentions to laptop features.
Tag types C ′ :
[’O’, ’B’, ’I’]
Table 5 shows detailed statistics for token-level classification datasets.
More dataset statistics are provided in Table 1.
Tables 6-13 show detailed information for all rules.
Figures 4-10 show examples of weak rules for various datasets.
[Table]
The [Table] [Table] comparisons
[Table]
Keyword-based detection of
[Table]
and
[Table]
sentiment
[Table]
Keyword-based detection of POSITIVE sentiment regarding the actors
[Table]
Keyword-based detection of POSITIVE and NEGATIVE sentiment regarding the plot
[Table] [Table]
Model-based detection of POSITIVE and NEGATIVE sentiment (TextBlob model)
[Table]
Regex-based detection of POSITIVE sentiment (recommendation expressions) 3.
[Table]
general Regex-based detection of POSITIVE and POSITIVE sentiment (general expressions) 4.
[Table]
Keyword-based detection of POSITIVE and NEGATIVE sentiment based on the user’s mood 5.
[Table]
Keyword-based detection of POSITIVE and NEGATIVE sentiment relevant to the service 6.
[Table]
Keyword-based detection of POSITIVE and NEGATIVE sentiment regarding the prices
[Table]
-based detection
[Table]
sentiment relevant
[Table]
Keyword-based detection of POSITIVE and NEGATIVE sentiment relevant to the food
[Table] [Table]
User interaction-based detection of FAKE news: If a news piece has a standard deviation of user sentiment scores greater than a threshold τ 1 , then the news is weakly labeled as FAKE news.
[Table]
User interaction-based detection of FAKE news: If the mean value of users’ absolute bias scores - sharing a piece of news – is greater than a threshold τ 2 , then the news piece is weakly-labeled as FAKE news.
credibility_score User interaction-based detection of FAKE news: If a news piece has an average credibility score less than a threshold τ 3 , then the news is weakly-labeled as FAKE news.
The Python implementation of CoNLL rules is provided in the “skweak” repo: https://github.com/NorskRegnesentral/skweak/blob/ 670fcdec680930ce3e497886d06d61e6a1f2c195/examples/ner/conll2003_ner.py
[Table]
Heuristic detection of
[Table]
names based on casing
[Table]
Heuristic detection of proper names based on casing + including at least one infrequent token
[Table]
Heuristic detection of proper names based on casing
[Table]
Heuristic detection of proper names based on casing + including at least one infrequent token
[Table]
Heuristic detection of sequences of tokens with NNP as POS-tag
[Table]
Heuristic detection of sequences of tokens with NNP as POS-tag + including at least one infrequent token (rank > 15000 in vocabulary) 10.
[Table]
Heuristic detection of proper noun phrases with compound dependency relations
[Table]
Heuristic detection of proper noun phrases with compound dependency relations + including at least one infrequent token
[Table]
Heuristic detection of entities of type NORP , LANGUAGE ,
[Table]
or EVENT
[Table]
Heuristic detection of entities of type LAW 14.
[Table]
Gazetteer using a large list of company names
[Table]
Heuristic function to detect full person names
[Table]
Probabilistic parser specialised in the recognition of dates, + times, money amounts, percents, and cardinal/ordinal values 18.
core_web_md NER model trained on Ontonotes 5. 19.
core_web_md+c NER model trained on Ontonotes 5.0 + postprocessing 20.
BTC NER model trained on the Broad Twitter Corpus 21.
BTC+c NER model trained on the Broad Twitter Corpus
+ postprocessing 22.
SEC NER model trained on SEC-filings 23.
SEC+c NER model trained on SEC-filings + postprocessing 24.
edited_core_web_md NER model trained on Ontonotes 5.0 + alternative postprocessing 25.
edited_core_web_md+c NER model trained on Ontonotes 5.0 + alternative postprocessing 26.
wiki_cased Gazetteer (case-sensitive) using Wikipedia entries 27.
wiki_uncased Gazetteer (case-insensitive) using Wikipedia entries 28.
multitoken_wiki_cased Same as above, but restricted to multitoken entities 29.
multitoken_wiki_uncased
Same as above, but restricted to multitoken entities 30.
wiki_small_cased Gazetteer (case-sensitive) using Wikipedia entries with non-empty description 31.
wiki_small_uncased Gazetteer (case-insensitive) using Wikipedia entries with non-empty description 32.
multitoken_wiki_small_cased Same as above, but restricted to multitoken entities 33.
multitoken_wiki_small_uncased Same as above, but restricted to multitoken entities 34.
geo_cased Gazetteer (case-sensitive) using the Geonames database 35.
geo_uncased Gazetteer (case-insensitive) using the Geonames database 36.
multitoken_geo_cased Same as above, but restricted to multitoken entities 37.
multitoken_geo_uncased Same as above, but restricted to multitoken entities 38.
crunchbase_cased Gazetteer (case-sensitive) using the Crunchbase Open Data Map 39.
crunchbase_uncased Gazetteer (case-insensitive) using the Crunchbase Open Data Map 40.
multitoken_crunchbase_cased
Same as above, but restricted to multitoken entities 41.
multitoken_crunchbase_uncased
Same as above, but restricted to multitoken entities 42.
product_cased Gazetteer (case-sensitive) using products extracted from DBPedia 43.
product_uncased Gazetteer (case-insensitive) using products extracted from DBPedia 44.
multitoken_product_cased
Same as above, but restricted to multitoken entities 45.
multitoken_product_uncased Same as above, but restricted to multitoken entities 46.
doclevel_voter Considers all appearances of the same entity string in the document 47.
doc_history_cased Considers already introduced entities in the document (case-sensitive) 48.
doc_history_uncased Considers already introduced entities in the document (case-insensitive) 49.
doc_majority_cased Considers all entities in the document (case-sensitive)
doc_majority_uncased Considers all majority labels in the document (case-insensitive)
[Table]
CoreDictionaryUncased AutoNER dictionary (biomedical entities) 2.
CoreDictionaryExact AutoNER dictionary (biomedical entities, exact match)
[Table]
CancerLike Heuristic detection of entities that are relevant to cancer 4.
CommonSuffixes
[Table]
common diseases
[Table]
of entities
[Table]
to deficiencies
[Table]
lesions
[Table]
entries for terms that
[Table]
relevant to body parts
[Table]
not
[Table]
to any disease
[Table]
stop words [Table] are not relevant to any disease
[Table]
Heuristic detection of punctiations that are not relevant to any disease
The Python implementation of WikiGold rules is provided in the “skweak” repo: https://github.com/NorskRegnesentral/skweak/blob/
[Table]
/examples/ner/conll2003_ner.py
[Table]
NER model trained on the Broad Twitter Corpus
[Table]
NER model trained on Ontonotes 5. 3.
[Table]
Gazetteer (case-sensitive) using the Crunchbase Open Data Map 4.
crunchbase_uncased Gazetteer (case-insensitive) using the Crunchbase Open Data Map
full_name_detector
Heuristic function to detect full person names 6.
geo_cased Gazetteer (case-sensitive) using the Geonames database 7.
geo_uncased Gazetteer (case-insensitive) using the Geonames database 8.
misc_detector Heuristic detection of entities of type NORP , LANGUAGE , FAC or EVENT 9.
wiki_cased Gazetteer (case-sensitive) using Wikipedia entries 10.
wiki_uncased Gazetteer (case-insensitive) using Wikipedia entries 11.
multitoken_crunchbase_cased
Same as above, but restricted to multitoken entities 12.
multitoken_crunchbase_uncased
Same as above, but restricted to multitoken entities 13.
multitoken_geo_cased Same as above, but restricted to multitoken entities 14.
multitoken_geo_uncased Same as above, but restricted to multitoken entities 15.
multitoken_wiki_cased Same as above, but restricted to multitoken entities 16.
multitoken_wiki_uncased
Same as above, but restricted to multitoken entities
[Table]
AutoNER dict with entries of terms that are relevant to electronics
[Table]
-defined keyword list
[Table]
on the “replace the” phrase
[Table]
on
[Table]
expression
[Table]
detection of stop words that are not relevant to electronics
[Table]
detection of punctuation that are not relevant to electronics
[Table]
Heuristic detection of pronouns that are not relevant to electronics
[Table]
Heuristic detection of terms that are not [Table] to laptop features 12.
Adv Heuristic detection of adverbs that are not relevant to electronics
C Additional Results
Table 14 shows standard deviation results for all datasets, methods, and base models.
The rightmost column responds the average standard deviation (AVG std) across tasks, which we also reported in Table 3.
Analysis of individual weak rules.
Tables 1521 show performance results for each weak rule for the datasets in WALNUT.
We evaluate two different strategies for majority voting in case of an instance that is not covered by any rules: (1) “Strict” counts the instance as misclassified and (2) “Loose” assigns a random label to the instance.
Most rules have very low F1 score while there are a few rules with a relatively high F1 score.
Figure 3 shows the precision-recall scatter plots for each weak rule individually.
(We skip the scatter plot for GossipCop as it has just 3 rules.)
Several rules have relatively high precision but most rules have very low recall.
[Table]
[Table]
rule0.
[Table] [Table]
[Table]
[Table]
rule0.
[Table] [Table]
[Table] [Table]
[Table]
rule
[Table] [Table]
[Table] [Table]
[Table] [Table]
rule
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
rule
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
rule
[Table]
[Table]
[Table]
rule
[Table]
rule
[Table]
rule
[Table]
[Table] [Table]
rule0. rule0.
Majority (strict) 0.
Majority (loose) 0.
[Table]
rule
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
rule0.
rule0. rule0.
rule0. rule0.
rule0. rule0.
Majority (strict) 0.
Majority (loose) 0.
[Table]
rule
[Table]
rule
[Table]
[Table] [Table]
majority
[Table] [Table]
Rule
Prec
Rec
F
[Table]
rule0.
rule0. rule0.
rule0.
rule0. rule0.
rule0. rule0.
rule0. rule0.
rule0. rule0.
Majority 0.
[Table]
Rule
Prec
Rec
[Table]
rule0.
rule0.
rule0. rule0.
rule0. rule0.
rule0. rule0.
rule0. rule0.
rule0. rule
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
Majority
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table]
rule
[Table] [Table]
[Table] [Table] [Table]
[Table]
rule
[Table]
[Table] [Table] [Table] [Table]
rule0. rule0.
rule1.
rule0.
rule0. rule0.
rule0. rule0.
Majority 0.