NatCat: Weakly Supervised Text Classiﬁcation with Naturally Annotated Resources
Abstract
We describe NatCat , a large-scale resource for text classiﬁcation constructed from three data sources: Wikipedia, Stack Exchange, and Reddit.
NatCat consists of documentcategory pairs derived from manual curation that occurs naturally within online communities.
To demonstrate its usefulness, we build general purpose text classiﬁers by training on
NatCat and evaluate them on a suite of 11 text classiﬁcation tasks ( CatEval ), reporting large improvements compared to prior work.
We benchmark diﬀerent modeling choices and resource combinations and show how tasks beneﬁt from particular NatCat data sources.
1. Introduction
Websites with community contributed content contain ample knowledge for natural language processing.
In this paper, we seek to improve text classiﬁcation by leveraging this knowledge in the form of texts paired with natural category annotations.
In particular, we present NatCat , a large-scale resource constructed automatically from three data sources:
Wikipedia, Stack Exchange
2 and Reddit.
3 With rich knowledge about text and categories, we show that NatCat is a useful resource for text classiﬁcation
To demonstrate the usefulness of NatCat we use it to train models that compute a score for any document-category pair.
As a result, we get weakly supervised (i e., “dataless”;
Chang et al., 2008) text classiﬁers that can be used oﬀ-the-shelf to produce interpretable and relevant topics for any document.
They can also be eﬀortlessly ported to a speciﬁc topic classiﬁcation task by computing the score for only the labels in the task.
Table 1 illustrates the use of a NatCat -trained model on a document from AGNews .
To evaluate, we propose CatEval , a standardized benchmark for evaluating weakly supervised text classiﬁcation with a choice of datasets, label descriptions for each dataset, and baseline results.
CatEval comprises a diverse choice of 11 text classiﬁcation tasks including both topic and sentiment labels, and contains both single and multi-label classiﬁcation tasks.
We show that NatCat is a valuable resource for weakly supervised text classiﬁcation and study the impact of data domain and pretrained model choice on tasks in CatEval .
We analyze the gap between weakly supervised and supervised models and show that the
Text
Israeli ambassador calls peace conference idea ‘counterproductive’.
A broad international peace conference that has reportedly been suggested by Egypt could be “counterproductive” and shouldn’t be discussed until after ...
AGNews international , sports , science technology , business
NatCat (Wiki.)
invasions , diplomats , peace , diplomacy , environmentalism , Egypt , patriotism ... mistakes of our models are reasonable and humanlike, suggesting that the construction process of NatCat is a promising approach to mining data for building text classiﬁers.
2. The NatCat Resource
In this section, we describe the creation of the NatCat resource.
NatCat is constructed from three diﬀerent data sources with natural category annotation: Wikipedia, Stack Exchange, and Reddit.
Therefore, NatCat naturally contains a wide range of world knowledge that is useful for topical text classiﬁcation.
For each data source, we describe ways of constructing document-category pairs in which the document can be labeled with the category.
Wikipedia.
Wikipedia documents are annotated with categories by the community contributors.
The categories of each Wikipedia document can be found at the bottom of the page.
We obtained Wikipedia documents from Wikimedia Downloads.
Wikipedia page-to-category mappings were generated from Wiki SQL dumps using the “categorylinks” and “page” tables.
We removed hidden categories by SQL ﬁltering, which are typically maintenance and tracking categories that are unrelated to the document content.
We also removed disambiguation categories.
After ﬁltering, there are 5.75M documents with at least one category, and a total of 1.19M unique categories.
We preprocessed the Wikipedia articles by removing irrelevant information such as the external links at the end of each article.
We then removed Wikipedia documents with fewer than 100 non-stopwords.
Some category names are lengthy and speciﬁc, e.g., “Properties of religious function on the National Register of Historic Places in the United States Virgin Islands”.
These categories are unlikely to be as useful for end users or downstream applications as shorter and more common categories.
Therefore, we consider multiple ways of augmenting the given categories with additional categories.
The ﬁrst way is to use a heuristic method of breaking long category names into shorter ones.
We ﬁrst use stopwords as separators and keep each part of the non-stopword word sequence as a category name.
For each category name of a document, we also run a named entity recognizer [Honnibal and Montani, 2017] to ﬁnd all named entities in that category
[Table]
name, and add them to the category set of the document.
This way we expand the existing category names from Wikipedia.
For the example category above, this procedure yields the following categories: “religious function”, “the national register of historic places”, “properties”, “historic places”, “the united states virgin islands”, “properties of religious function on the national register of historic places in the united states virgin islands”, “united states virgin islands”, and “national register”.
Our second method of expansion is based on the fact that Wikipedia categories can have parent categories and therefore form a hierarchical structure.
When expanding the category set by adding its ancestors, there is a trade-oﬀ between speciﬁcity/relevance and generality/utility of category names.
Using only the categories provided for the article yields a small set of high-precision, speciﬁc categories.
Adding categories that are one or two edges away in the graph increases the total number of training pairs and targets more general/common categories, but some of them will be less relevant to the article.
In NatCat , we include all categories of documents that are up to two edges away.
Stack Exchange.
Stack Exchange is a question answering platform where users post and answer questions as a community.
Questions on Stack Exchange fall into 308 subareas, each area having its own site.
We create document-category pairs by pairing question titles or descriptions with their corresponding subareas.
Question titles, descriptions, and subareas are available from Chu et al.
Many Stack Exchange subareas have their own corresponding “meta” sites.
A meta site is meant to discuss the website itself regarding its policy, community, and bugs, etc.
When creating this dataset, we merge the subareas with their corresponding meta areas.
This gives us over 2 million documents with 156 categories.
Reddit.
Inspired by Puri and Catanzaro [2019], we construct a category classiﬁcation dataset from Reddit.
In our dataset, we propose to classify Reddit post titles to their corresponding subreddit names.
We use the OpenWebText 4 toolkit to get Reddit posts with more than 3 karma and their subreddit names.
We keep only the top 3000 most frequent subreddits as they better capture the common categories that we are interested in.
This gives us over 7 million documents with 3000 categories.
[Table]
Constructed by these three data sources, NatCat covers a wide range of topics and world knowledge.
Table 2 summarizes statistics of training sets we sampled from NatCat .
Note that all documents from Stack Exchange and Reddit have only one associated category, while a document from Wikipedia may have multiple categories describing it.
3. CatEval Tasks
To evaluate NatCat , we will use it to build general purpose text classiﬁers and test them on a variety of text classiﬁcation tasks.
In this section, we introduce CatEval , which comprises a diverse choice of 11 text classiﬁcation tasks including both topic-related and sentimentrelated labels, and contains both single and multi-label classiﬁcation tasks.
For single label topic classiﬁcation, we have AGNews , 6 DBpedia [Lehmann et al., 2015], Yahoo [Zhang et al., 2015], and 20 News Groups [Lang, 1995].
For sentiment classiﬁcation, we use Emotion [Klinger et al., 2018], SST-
[Socher et al., 2013], Yelp-2 , and Amazon-
[Zhang et al., 2015].
Emotion is a ﬁne-grained sentiment classiﬁcation tasks with labels expressing various emotions, while the other three are binary sentiment classiﬁcation tasks diﬀerentiating positive and negative sentiments.
As for multi-label topical classiﬁcation, we have NYTimes [Sandhaus, 2008], Comment ,and Situation [Mayhew et al., 2019] datasets.
The NYTimes categories have hierarchical structure, but we merely use the category names from the lowest level.
We removed newspaper-speciﬁc categories that are not topical in nature.
Of the remainingcategories, we only use the 100 most frequent categories in our experiments, and randomly sample 1 million documents for the training set, 10k for a dev set, and 10k as a test set.
Table 3 summarizes the key statistics of each dataset, including the average number of sentences, average number of words, and average sentence length.
They cover a broad range of text classiﬁcation tasks and can serve as a benchmark for text classiﬁers.
Choice of label names.
For a given document classiﬁcation task, we need to specify the name of each label.
As in prior work [Chang et al., 2008, Song and Roth, 2014], we manually choose words corresponding to labels in the downstream tasks.
Our models and the baselines use the same label names which are provided in the appendix.
As NatCat is a large textual resource with ample categories, almost all labels in the CatEval datasets appear in NatCat except for some conjunction phrases, such as “written work”, “manufacturing operations and logistics”, and “home and garden”.
However, there is no guarantee that the labels in NatCat have the same deﬁnition as the labels in the downstream tasks, and in fact we ﬁnd such divergences to be causes of error, including when measuring human performance on the text classiﬁcation tasks.
Weakly supervised methods (and humans) are more susceptible to semantic imprecision in label names than supervised methods.
In recent work, we showed empirically the variance of dataless classiﬁers due to label name variations, and proposed unsupervised methods to reduce this variance for particular datasets [Chu et al., 2021].
Why is this weakly supervised?
The goal of weakly supervised text classiﬁcation is to classify documents into any task-speciﬁc categories that are not necessarily seen during training.
We build models with NatCat that are capable of scoring any candidate label for any document.
To evaluate, we take the test sets of standard document classiﬁcation tasks, use the model to score each label from the set of possible labels for that task, and return the label with the highest score.
The reason we describe our method as “weakly supervised” is because it does not require annotated training data with the same labeling schema and from the same distribution as the test set, but rather uses freely-available, naturally-annotated document/category pairs as a training resource.
Existing applications.
Even though the category distributions of NatCat do not exactly match the target distribution in a downstream task, generalization is possible if they are suﬃciently similar and cover a diverse set of knowledge.
This setting is referred to using several diﬀerent terms, including dataless classiﬁcation [Chang et al., 2008], transfer learning [Pan and Yang, 2009], distant supervision [Mintz et al., 2009], and weakly-supervised learning [Zhou, 2017].
4. Experiments 4.1 Models and Training
We ﬁne-tune BERT [Devlin et al., 2019] and RoBERTa [Liu et al., 2019] on NatCat .
In our experiments, we use BERT-base-uncased (110M parameters) and RoBERTa-base (110M parameters).
We formulate NatCat training as a binary classiﬁcation task to predict whether a category correctly describes a document.
Binary cross entropy loss is used for training.
For each document-category pair, we randomly sample 7 negative categories for training.
As documents from Wikipedia have multiple positive categories, we randomly choose one positive category.
To form the input for both BERT and RoBERTa, we concatenate the category with the document: “[CLS] category [SEP] document [SEP]”.
In our experiments, we truncate the document to ensure the category-document pair is within 128 tokens.
For BERT and RoBERTa models shown in Table 4, we train models on the whole
NatCat dataset and also separately the data from each single data source (Wikipedia,
Stack Exchange, or Reddit) for comparison.
For each single source, we train the model for one epoch on 100k instances.
When training on all three sources, we train on 300k instances.
The learning rate is set to be 0.00002, and we perform learning rate warmup for 10% of the training steps and then linearly decay the learning rate.
As BERT and RoBERTa models are known to suﬀer from randomness among diﬀerent ﬁne-tuning runs, we perform each single experiment 5 times with diﬀerent random seeds and report the median of such ﬁve runs.
We also do supervised training on Emotion , NYTimes , Comment , and Situation with the RoBERTa model.
We follow the same training procedure as we train on NatCat to solve a document-category binary classiﬁcation task.
We compare NatCat -trained models to Explicit Semantic Analysis (ESA; Gabrilovich and Markovitch, 2007) as our primary baseline.
10 ESA was shown eﬀective for dataless classiﬁcation by Chang et al.
[2008], and we mostly follow their procedure.
However, instead of setting a threshold on the number of concepts, we use all concepts as we ﬁnd this improves ESA’s performance.
In preliminary experiments, we tried other unsupervised text representation learning approaches, e.g., encode the document and category using pretrained models or pretrained word embeddings (BERT, ELMo, GloVe, etc.), then use cosine similarity as the scoring function for a document-category pair.
We also experimented with the pretrained
GPT2 model and ﬁne-tuning it on NatCat [Radford et al., 2019].
However, we found these methods do not perform as well as weakly supervised approaches such as ESA and our approach, so we do not report the results of such methods in this paper.
4.2 Evaluation
We report classiﬁcation accuracy for all single label classiﬁcation tasks, including topical and sentiment tasks.
For multi-label classiﬁcation tasks, we use label ranking average precision (LRAP), a multi-label generalization of the mean reciprocal rank.
For each example i = 1 . . .
N , let Y ( i ) ⊆
m }
=
Y denote the set of gold labels and f ( i
) ∈
R m denote the model label scores.
LRAP is deﬁned as
[Equation]
which achieves the highest value of 1 iﬀ all gold labels are ranked at the top.
To directly compare with Yin et al.
[2019] in multi label classiﬁcation tasks, we also use label-wise weighted F1 score in some cases.
4.3 Primary Results
Table 4 summarizes the experimental results of BERT and RoBERTa models trained on
NatCat and evaluated on CatEval .
RoBERTa trained on NatCat performs the best on
[Table]
compare with the dataless classiﬁer ESA (E average across tasks, but there are some diﬀerences between BERT and RoBERTa.
BERT is better on AGNews and NYTimes , both of which are in the newswire domain, as well as 20NG, which also involves some news- or technical-related material.
RoBERTa is better on
Yahoo as well as better on average in the emotion, binary sentiment, and situation tasks.
This may be due to RoBERTa’s greater diversity of training data (web text) compared to
BERT’s use of Wikipedia and books.
To provide perspective on the diﬃculty of the weakly supervised setting, we obtained annotations from 3 human annotators involved in this research project on 60 instances from AGNews , 50 from DBpedia , and 100 from Yahoo .
We showed annotators instances and the set of class labels and asked them to choose a single category using their own interpretation and judgment without the ability to look at any training examples.
Average accuracies from three annotators on these tasks are reported in Table 4.
We also compare with results from supervised methods.
The results of AGNews ,
DBpedia , Yahoo , Yelp-2 and Amazon-2 are from Zhang et al.
The SST-2 result is from Wang et al.
The 20 News Groups result is from Pappagari et al.
NYTimes , Situation , Comment and Emotion results are ﬁne-tuned RoBERTa models.
In some tasks ( AGNews and DBpedia ), supervised models outperform human annotators.
We believe this is caused by semantic drift between human interpretation and the actual meaning of the labels as determined in the dataset.
Supervised models are capable of learning such nuance from the training data, while an annotator without training is not capable of classifying documents in that way.
Weakly supervised models are like human
[Table]
+ NatCat
[Table]
Yin
[Table]
annotators in that they are only capable of classifying documents with the general knowledge they have learned (in this case from large scale naturally-annotated data).
Yin et al.
[2019] build weakly supervised text classiﬁers from Wikipedia.
Comparing with their reported F1 results in a zero-shot setting (Yahoo: 52.1; Emotion: 21.2; Situation: 27.7),
NatCat -trained BERT models yield better results in topic classiﬁcation tasks (Yahoo: 54.9;
Situation: 37.1), though not as strong on others (Emotion: 16.1), proving that NatCat is a valuable resource of topical knowledge.
4.4 Half Seen Settings
Zero-shot text classiﬁcation is often deﬁned as training models on some seen labels and testing on an expanded set of both seen and unseen labels [Yogatama et al., 2017, Yin et al., 2019].
We use NatCat as a pretraining resource in this experimental setup.
We follow the same seen and unseen label splits as Yin et al.
[2019], using their v0 and v1 splits, and same experimental setup.
We train our models both starting from the original
BERT-base-uncased model and the NatCat -pretrained BERT model.
Table 5 summarizes the results (medians over 5 random seeds).
The evaluation metrics are accuracy for Yahoo and label-weighted F1 for Emotion and Situation , in order to compare to Yin et al.
Pretraining on NatCat improves BERT’s results on Yahoo , but it does not show clear improvements on Emotion and Situation in this setting.
Our Yahoo results are not directly comparable to the results from Yin et al.
[2019] for several reasons, the most signiﬁcant being that
Yin et al.
[2019] expand label names using their deﬁnitions in WordNet, while we choose to use the plain label names for our experiments.
Also, Yin et al.
[2019] formulate the problem as an entailment task, and there are diﬀerences in training set sizes.
Yin et al.
[2019] implement a “harsh policy” to impose an advantage to unseen labels by adding an α value to the probabilities of unseen labels.
This α value is set by tuning on the development set which contains both seen and unseen labels.
However, we do not assume access to a development set with unseen labels.
[Table] The lower part of Table 5 shows the results of NatCat -trained models on the seen and unseen labels of these datasets.
On Yahoo , a topical classiﬁcation task, all models trained on the seen labels perform worse on the unseen labels (v0: 44.3%; v1: 34.9%) than
RoBERTa trained purely on NatCat (v0: 52.1%; v1: 63.5%).
Therefore, if we are primarily interested in unseen labels for topic classiﬁcation tasks, our weakly supervised approach appears stronger than the more traditional half-seen setting.
The NatCat -trained models also perform fairly well on unseen labels for Emotion and Situation , outperforming the unseen results from Yin et al.
[2019] on the v1 split.
5. Analysis 5.1 Training Resources
Table 6 shows the model performances when trained on diﬀerent resources of NatCat .
For each single resource (Wikipedia, StackEx., or Reddit), we train the model for one epoch on 100k instances.
We follow the exact same training procedure as described in Subsection 4.1.
Each data source is useful for some particular topical classiﬁcation task, most of which can be explained by domain similarities.
For example, models trained on Wikipedia are good at DBpedia , which can be explained by the fact that DBpedia is also built from Wikipedia.
Stack Exchange is especially helpful for Yahoo; both are in the domain of community question answering.
Models trained on Reddit, which contains a sizable amount of political commentary and news discussion in its most frequent categories, are particularly good at
NYTimes .
Models trained on Stack Exchange do not perform well on most sentiment related tasks.
This is likely because Stack Exchange subareas are divided by topic.
Wikipedia and Reddit are better resources for training sentiment classiﬁers, as they cover broader ranges of sentiment and emotion knowledge.
It may be surprising that Wikipedia is good for sentiment classiﬁcation.
Wikipedia contains knowledge of sentiment words/phrases and their associations with positive or negative sentiment.
For example, the Wikipedia category of “emotion” https://en.wikipedia.org/wiki/Category:Emotion provides some examples of sentiment knowledge.
5.2 Training Set Size and Model Variance
While NatCat has over 10 million documents with over a million categories, we have used a small subset of it due to computational constraints.
We compared models trained on 100k and 300k document-category pairs, following the same hyperparameter settings as in
Section 4.
We ﬁnd that increasing training size generally harms performance on CatEval tasks.
For example, averaging over all CatEval tasks, BERT trained on Wikipedia is 1.5 points lower when moving from 100k training instances to 300k instances.
For Stack
Exchange, the gap is 2.1 points.
For Reddit, it is 0.7 points.
This is likely due to overﬁtting on the NatCat binary classiﬁcation tasks.
As there is a discrepancy between training and evaluation, increasing training data or epochs may not necessarily improve results on downstream tasks.
This is a general phenomenon in weakly supervised and zero-shot classiﬁcation, as we do not have development sets to tune training parameters for such tasks.
Similar ﬁndings were reported by Puri and Catanzaro [2019], suggesting future work to ﬁgure out good ways to do model selection in zero-shot settings.
BERT and RoBERTa are known to suﬀer from instability in ﬁne-tuning, i.e., training with diﬀerent random seeds may yield models with vastly diﬀerent results.
We found both models have higher variance on sentiment tasks compared to topic classiﬁcation.
While nontrivial variances are observed (full results in Appendix E) ensembling the 5 models almost always outperforms the median of the individual models.
5.3 Error Analysis
Upon analysis of the confusion matrix of the RoBERTa ensemble predictions on AGNews ,
DBpedia , and Yahoo , we observe the following common misclassiﬁcation instances:
• In AGNews , science & technology and international are often misclassiﬁed as business .
• In DBpedia , nature is often misclassiﬁed as animal , nature as plant , written work as artist , and company as transportation .
• In Yahoo , society & culture is often misclassiﬁed as education & reference , politics & government , and business & ﬁnance .
health is often misclassiﬁed into science & mathematics , and family & relationships as society & culture .
The RoBERTa model trained on NatCat confuses closely related categories, but it rarely makes mistakes between clearly unrelated concepts.
We ﬁnd that human errors follow the same pattern: they are mostly in closely related categories.
These results suggest that models trained on NatCat are eﬀective at classifying documents into coarse-grained categories, but ﬁne-grained categorization requires annotated training data speciﬁc to the task of interest.
6. Related Work
Wikipedia is widely used for weakly-supervised text classiﬁcation.
A classical work is the dataless text classiﬁcation approach of Chang et al.
[2008], Song and Roth [2014].
This approach uses Explicit Semantic Analysis [Gabrilovich and Markovitch, 2007], a method to represent a document and a candidate category as sparse binary indicators of Wikipedia concepts and compute their relatedness by cosine similarity.
Wang et al.
[2009] learn a universal text classiﬁer based on Wikipedia by extending the dataless approach.
Yin et al.
[2019] build models by directly mapping Wikipedia documents to their annotated categories, similar to the way that NatCat Wikipedia instances are created.
There has been a great deal of prior work on weakly-supervised and zero-shot text classiﬁcation.
For example, some have focused on restricted domains such as medical text [Mullenbach et al., 2018, Rios and Kavuluru, 2018], leveraged additional information such as semantic knowledge graphs [Zhang et al., 2019], or carefully exploited weak supervision such as class keywords [Meng et al., 2019] to achieve satisfactory performance.
Some embed text and class labels into the same embedding space and use simple methods for classiﬁcation [Dauphin et al., 2013, Nam et al., 2016, Li et al., 2016b, Ma et al., 2016].
Others model the presence of a special unseen class label and design specialized training and inference procedures to handle it [Shu et al., 2017, Fei and Liu, 2016, Zhang et al., 2019].
Yogatama et al.
[2017] report zero-shot text classiﬁcation experiments with a neural model that jointly embeds words and labels.
The scale of NatCat is much larger than the small dataset-speciﬁc experiments of Yogatama et al.
[2017], and the models trained with NatCat do not require additional supervision at test time such as seed words as in topic modeling approaches [Li et al., 2016a,
Chen et al., 2015] or self-training
[Meng et al., 2020].
Recent work [Zhang et al., 2021, Meng et al., 2020, Shen et al., 2021] uses naturally annotated hierarchical tree structure of label taxonomies to improve text classiﬁcation.
Puri and Catanzaro [2019] ﬁne-tune language models on statements containing text and corresponding categories from
Reddit.
There is also a wealth of prior work in semi-supervised text classiﬁcation: using unlabeled text to improve classiﬁcation performance [Nigam et al., 2000, Howard and Ruder, 2018, Devlin et al., 2019, Liu et al., 2019, Lan et al., 2020, Peters et al., 2018].
7. Conclusion
We described NatCat , a resource of rich knowledge for text classiﬁcation constructed from online community contributed content.
We demonstrated that text classiﬁers built using
NatCat perform strongly compared to analogous previous approaches.
We have released the NatCat resource, CatEval benchmark dataset, our code for running experiments and for evaluation, and our best pretrained model at https://github.com/ZeweiChu/NatCat .
The NatCat -trained models not only handle any label set but also supply a myriad of interpretable categories for a document oﬀ-the-shelf.
We believe NatCat can be a useful resource for applications in natural language processing, information retrieval, and text mining.
Acknowledgments
We wish to thank the anonymous reviewers for their feedback.
This research was supported in part by a Bloomberg data science research grant to KS and KG.
Appendix A. Preliminary Experiments with GPT2 Models
We report preliminary results in adapting GPT2 models to perform CatEval tasks.
To do so, we construct the following descriptive text: “The document is about [category]: [document content]”, where [category] is replaced by the class label we want to score, and [document content] is the document we want to classify.
The descriptive text is tokenized by the BPE tokenizer, truncated to 256 tokens, and fed into the pretrained GPT2 model.
The class label with the lowest average loss over all tokens is picked as the predicted label.
The results are shown in the initial rows of Table 7.
We ﬁnd mixed results across tasks, with the GPT2 models performing well on sentiment tasks but struggling on the topical tasks.
Increasing GPT2 model size helps in some tasks but hurts in others.
We also ﬁne-tune GPT2 models on NatCat .
Each document in NatCat is paired with its category to construct the aforementioned descriptive text, and ﬁne-tuned as a language modeling task.
The results (upper section of Table 7) are mixed, with the topical accuracies decreasing on average and the sentiment accuracies slightly increasing for GPT small but decreasing for GPT2 medium.
A key diﬀerence between training GPT2 and BERT/RoBERTa is that with GPT2, we do not explicitly feed information about negative categories.
One way to incorporate this information is to construct descriptive text with “candidate categories” following Puri and
Catanzaro [2019]. 12 We sample 7 negative categories and 1 correct category to form the candidates.
The results, shown in the middle section of Table 7, improve greatly for some tasks (AG, DBP, and Emo), but drop for the other CatEval tasks.
Puri and Catanzaro [2019] also create training data from Reddit.
They annotate the text from each outbound weblink with the title of the Reddit post, and the subreddit that the link was posted in, while our Reddit dataset annotates each post title with the name of the subreddit it belongs to.
The GPT2 small model actually outperforms the 1/4-data training setting from Puri and Catanzaro [2019] on the sentiment tasks, though not the All-data training.
Compared to BERT and RoBERTa, it is harder to ﬁne-tune a GPT2 model that performs well across CatEval tasks.
In fact, there are many ways to convert text classiﬁcation into language modeling tasks; we explored two and found dramatically diﬀerent performance from them.
It remains an open question how to best formulate text classiﬁcation for pretrained language models, and how to ﬁne-tune such models on datasets like NatCat .
Appendix B. Hyperparameters and Model Training
The models and training hyperparametes are described in the main body of the paper.
As we run our evaluations in a zero-shot setting, we do not have a development set for parameter tuning and model selection.
All of our models (BERT, RoBERTa, GPT2) and training hyperparameters are chosen based on recommended settings from the Huggingface
Transformers project [Wolf et al., 2019].
We perform 5 experiment runs with the same hyperparameter settings, except with diﬀerent random seeds (1, 11, 21, 31, 41), and we report the median performances and standard deviations among diﬀerent runs.
The following are the sizes of the models we use: BERT-base-uncased (110M), RoBERTabase (110M), GPT-2 small (117M), GPT-2 medium (345M).
When performing supervised training on Emotion , NYTimes , Comment and Situation , all models are ﬁne-tuned for 3 epochs following the same hyperparameter settings as the main experiment section.
We train on only 1000 instances of NYTimes so we can ﬁnish training in a reasonable amount of time (less than 4 hours).
Comment is trained on the test set as it does not have a training set, so its results represent oracle performance.
We perform our experiments on single GPUs (including NVIDIA
Ti, Titan X, Titan
V, TX Pascal).
Training for a single epoch on 300k instances of NatCat takes aboutminutes, and for 100k instances it will be around 150 minutes.
For evaluation on CatEval , it varies on diﬀerent tasks.
For small tasks like SST-2, it only takes about 20 minutes.
For bigger tasks like Amazon-2, it takes about 1200 minutes on a single GPU for evaluation (we parallelize the evaluation over 40 GPUs so it only takes 30 minutes).
Appendix C. Evaluation Metrics
We use three metrics to evaluate CatEval tasks.
Accuracy.
Accuracy is the number of correct predictions over the total number of predictions.
It is used for single label classiﬁcation tasks.
[Table]
Label Ranking Average Precision.
This metric is used for evaluating multi-label text classiﬁcation tasks, and is described in the main body of the paper.
We use the scikitlearn implementation from https://scikit-learn.org/stable/modules/generated/sklearn.
metrics.label_ranking_average_precision_score.html .
Label Weighted F1 Score.
This metric is used to evaluate multi-label text classiﬁcation tasks.
We follow the implementation from Yin et al.
Table 8 compare the F1 scores of diﬀerent models on multi label topic classiﬁcation tasks.
Appendix D. Training Sizes
Table 9 shows how the model performances vary with 100k or 300k training instances, following the same hyperparameter settings as in Section 4.
We ﬁnd that increasing training size generally harms performance on CatEval tasks.
For example, averaging over all
CatEval tasks, BERT trained on Wikipedia is 1.5 points lower when moving from 100k training instances to 300k instances.
For Stack Exchange, the gap is 2.1 points.
For Reddit, it is 0.7 points.
This is likely due to overﬁtting on the NatCat binary classiﬁcation tasks.
As there is a discrepancy between training and evaluation, increasing training data or epochs may not necessarily improve results on downstream tasks.
This is a general phenomenon in weakly supervised and zero-shot classiﬁcation, as we do not have development sets to tune training
[Table] [Table]
seeds
parameters for such tasks.
Similar ﬁndings were reported by Puri and Catanzaro [2019], suggesting future work to ﬁgure out good ways to do model selection in zero-shot settings.
Appendix E. Model Variances
BERT and RoBERTa are known to suﬀer from instability in ﬁne-tuning, i.e., training with diﬀerent random seeds may yield models with vastly diﬀerent results.
To study this phenomenon in our setting, we performed training in each setting with 5 random seeds and calculate standard deviations for diﬀerent tasks.
As shown in table 10, both models have higher variance on sentiment tasks compared to topic classiﬁcation.
While nontrivial variances are observed, ensembling the 5 models almost always outperforms the median of the individual models.
Appendix F. NatCat Category Names
We list the most frequent 20 categories from Wikipedia, Stack Exchange, and Reddit, separated by semicolons.
Wikipedia: years; births by decade; 20th century births; people; people by status; living people; stub categories; works by type and year; works by year; 20th century deaths; establishments by year; 19th century births; years in music; establishments by year and country; establishments by country and year; people by nationality and occupation; 1980s; alumni by university or college in the united states by state; 1970s; 1980s events
Stack Exchange: math; gis; physics; unix; stats; tex; codereview; english; gaming; apple; sciﬁ; drupal; electronics; travel; ell; rpg; meta; mathematica; dba; magento
Reddit: AdviceAnimals; politics; worldnews; todayilearned; news; The Donald; atheism; technology; funny; conspiracy; science; trees; gaming; india; soccer; WTF; reddit.com;
Conservative; POLITIC; canada
Appendix G. CatEval Category Names
We list all the category names all tasks in this section, separated by semicolons.
AGNews : international 14 ; sports; business; science technology
DBpedia : company; educational institution; artist; athlete; politician; transportation; building; nature; village; animal; plant; album; ﬁlm; written work
Yahoo : society culture; science mathematics; health; education reference; computers internet; sports; business ﬁnance; entertainment music; family relationships; politics government 20 News Groups : atheist christian atheism god islamic; graphics image gif animation tiﬀ; windows dos microsoft ms driver drivers card printer; bus pc motherboard bios board computer dos; mac apple powerbook; window motif xterm sun windows; sale oﬀer shipping forsale sell price brand obo; car ford auto toyota honda nissan bmw; bike motorcycle yamaha; baseball ball hitter; hockey wings espn; encryption key crypto algorithm security; circuit electronics radio signal battery; doctor medical disease medicine patient; space orbit moon earth sky solar; christian god christ church bible jesus; gun fbi guns weapon compound; israel arab jews jewish muslim; gay homosexual sexual; christian morality jesus
[Table]
[Table]