Original: S ELF - SUPERVISED C ONTRASTIVE Z ERO TO F EW - SHOT L EARNING FROM S MALL , L ONG - TAILED T EXT DATA
Revised: S ELF - SUPERVISED C ONTRASTIVE Z ERO TO F EW - SHOT L EARNING FROM S MALL , L ONG - TAILED T EXT DATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For natural language processing (NLP) ‘text-to-text’ tasks, prevailing approaches heavily rely on pretraining large self-supervised models on massive external data sources.
Revised: For natural language processing (NLP) ‘text-to-text’ tasks, prevailing approaches heavily rely on pretraining large self-supervised models on massive external data sources.

Original: However, this methodology is being critiqued for: exceptional compute and pretraining data requirements; diminishing returns on both large and small datasets; and evaluation settings that overestimate performance differences.
Revised: However, this methodology is being critiqued for: exceptional compute and pretraining data requirements; diminishing returns on both large and small datasets; and evaluation settings that overestimate performance differences.

Original: The core belief behind current methodology, coined ‘the bitter lesson’ by R. Sutton, is that ‘compute scale-up beats data and compute-efﬁcient algorithms’, neglecting that progress in compute hardware scale-up is based near entirely on the miniaturisation of resource consumption.
Revised: The core belief behind current methodology, coined ‘the bitter lesson’ by R. Sutton, is that ‘compute scale-up beats data and compute-efﬁcient algorithms’, neglecting that progress in compute hardware scale-up is based near entirely on the miniaturisation of resource consumption.

Original: We thus approach pretraining from a miniaturisation perspective, such as not to require massive external data sources and models, and avoid translations from continuous input embeddings to discrete labels.
Revised: We thus approach pretraining from a miniaturisation perspective, such as not to require massive external data sources and models, and avoid translations from continuous input embeddings to discrete labels.

Original: To minimise favourable evaluation, we examine learning on a challenging long-tailed, low-resource, multi-label text classiﬁcation dataset with noisy, highly sparse labels and many rare concepts.
Revised: To minimise favourable evaluation, we examine learning on a challenging long-tailed, low-resource, multi-label text classiﬁcation dataset with noisy, highly sparse labels and many rare concepts.

Original: To this end, we propose using a ‘datasetinternal’, self-supervised contrastive autoencoding approach for pretraining that enables marked improvements in zero-shot, few-shot and supervised learning performance; even under a challenging, otherwise avoided, low-resource scenario, without defaulting to large-scale external datasets as support training signals.
Revised: To this end, we propose using a ‘datasetinternal’, self-supervised contrastive autoencoding approach for pretraining that enables marked improvements in zero-shot, few-shot and supervised learning performance; even under a challenging, otherwise avoided, low-resource scenario, without defaulting to large-scale external datasets as support training signals.

Original: Crucially, we ﬁnd evidence that zero and few-shot learning markedly beneﬁt from adding more ‘dataset-internal’, self-supervised training signals, e.g. when increasing self-supervised learning signals via large external sources is infeasible.
Revised: Crucially, we ﬁnd evidence that zero and few-shot learning markedly beneﬁt from adding more ‘dataset-internal’, self-supervised training signals, e.g. when increasing self-supervised learning signals via large external sources is infeasible.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The current prevailing approach to supervised and few-shot learning is to use self-supervised pretraining on large-scale ‘task-external’ data and then ﬁne-tune on end-task labels.
Revised: The current prevailing approach to supervised and few-shot learning is to use self-supervised pretraining on large-scale ‘task-external’ data and then ﬁne-tune on end-task labels.

Original: Recent studies have found that, thus far, this way of pretraining fails in low-resource settings (Yogatama et al., 2019;
Revised: Recent studies have found that, thus far, this way of pretraining fails in low-resource settings (Yogatama et al., 2019;

Original: S¸erbetci et al., 2020) and that reported performance improvements are caused in part by evaluation setups that are designed in line with the paradigm that “massive resources are pivotal” to improving language understanding (Linzen, 2020; Schick & Sch¨utze, 2020a; Dodge et al., 2020;
Revised: S¸erbetci et al., 2020) and that reported performance improvements are caused in part by evaluation setups that are designed in line with the paradigm that “massive resources are pivotal” to improving language understanding (Linzen, 2020; Schick & Sch¨utze, 2020a; Dodge et al., 2020;

Original: Brown et al., 2020) or computer vision (Chen et al., 2020).
Revised: Brown et al., 2020) or computer vision (Chen et al., 2020).

Original: Despite these critiques, the underlying goal of better initialisation of layer weights is a core requirement of successful learning with neural networks, where self-supervised layer-wise pretraining (Bengio et al., 2006) was replaced by better layer initialisation (Glorot & Bengio, 2010), which was in turn replaced by pretraining on growing amounts of external data (Bojanowski et al., 2017;
Revised: Despite these critiques, the underlying goal of better initialisation of layer weights is a core requirement of successful learning with neural networks, where self-supervised layer-wise pretraining (Bengio et al., 2006) was replaced by better layer initialisation (Glorot & Bengio, 2010), which was in turn replaced by pretraining on growing amounts of external data (Bojanowski et al., 2017;

Original: Devlin et al., 2019; Chen et al., 2020; Brown et al., 2020) – i.e. FastText, BERT, SIMCLR and GPT-3.
Revised: Devlin et al., 2019; Chen et al., 2020; Brown et al., 2020) – i.e. FastText, BERT, SIMCLR and GPT-3.

Original: The latter three approaches require massive compute and data resources, but enable marked learning improvements in few-shot (SIMCLR, GPT3) or zero-shot (GPT-3) scenarios compared to models that have several orders of magnitude fewer parameters.
Revised: The latter three approaches require massive compute and data resources, but enable marked learning improvements in few-shot (SIMCLR, GPT3) or zero-shot (GPT-3) scenarios compared to models that have several orders of magnitude fewer parameters.

Original: There are efforts to reduce model size requirements for few and zero-shot adaptation by orders of magnitude (Schick & Sch ¨ utze, 2020a;b; Plank & Rethmeier, 2019), with some being increasingly beneﬁcial in scenarios with low input data ( X ), label resources ( Y ), and rare events in X, Y .
Revised: There are efforts to reduce model size requirements for few and zero-shot adaptation by orders of magnitude (Schick & Sch ¨ utze, 2020a;b; Plank & Rethmeier, 2019), with some being increasingly beneﬁcial in scenarios with low input data ( X ), label resources ( Y ), and rare events in X, Y .

Original: Crucially, such approaches do not simply rely on more data, but on creating better initialised input features X .
Revised: Crucially, such approaches do not simply rely on more data, but on creating better initialised input features X .

Original: In contrast, approaches like SIMCLR or BERT (Chen et al., 2020; Devlin et al., 2019) use self-supervision via contrastive learning and input masking on large-scale datasets to create broader learning signals than supervision provides.
Revised: In contrast, approaches like SIMCLR or BERT (Chen et al., 2020; Devlin et al., 2019) use self-supervision via contrastive learning and input masking on large-scale datasets to create broader learning signals than supervision provides.

Original: SIMCLR is based on a metric learning approach called contrastive self-supervision – i.e. learning to distinguish (dis-)similar inputs using generated, but weak supervision tasks.
Revised: SIMCLR is based on a metric learning approach called contrastive self-supervision – i.e. learning to distinguish (dis-)similar inputs using generated, but weak supervision tasks.

Original: However, as Musgrave et al.
Revised: However, as Musgrave et al.

Original: ﬁnd, “when evaluating old vs. recent metric learning approaches, while controlling for data and model size , newer methods only marginally improve over the classic contrastive formulation”.
Revised: ﬁnd, “when evaluating old vs. recent metric learning approaches, while controlling for data and model size , newer methods only marginally improve over the classic contrastive formulation”.

Original: Remarkably, Bansal et al.
Revised: Remarkably, Bansal et al.

Original: (2020) recently showed that adding broader self-supervision rather than increasing data size during large-scale pretraining can substantially boost few-shot performance.
Revised: (2020) recently showed that adding broader self-supervision rather than increasing data size during large-scale pretraining can substantially boost few-shot performance.

Original: Our central question is whether increased (broader) pretraining self-supervision also boosts few and zero-shot performance using only small-scale, ‘task-internal’ data, instead of resorting to largescale pretraining on orders of magnitude more ‘task-external’ data – i.e. Do we really need large datasets for pretraining or just more (broader) self-supervised learning signals?
Revised: Our central question is whether increased (broader) pretraining self-supervision also boosts few and zero-shot performance using only small-scale, ‘task-internal’ data, instead of resorting to largescale pretraining on orders of magnitude more ‘task-external’ data – i.e. Do we really need large datasets for pretraining or just more (broader) self-supervised learning signals?

Original: To broaden small data self-supervision, we propose a contrastive self-supervised objective based on labelembedding prediction , where labels are expressed as word embeddings to learn their matching with an input text embedding.
Revised: To broaden small data self-supervision, we propose a contrastive self-supervised objective based on labelembedding prediction , where labels are expressed as word embeddings to learn their matching with an input text embedding.

Original: For contrastive learning, our method samples positive and negative word input tokens X for self-supervised pretraining, zero and few-shot learning; and positive and negative classes Y for few-shot to fully supervised ﬁne-tuning.
Revised: For contrastive learning, our method samples positive and negative word input tokens X for self-supervised pretraining, zero and few-shot learning; and positive and negative classes Y for few-shot to fully supervised ﬁne-tuning.

Original: Thus, we propose a model architecture that uniﬁes training from labels Y and inputs X .
Revised: Thus, we propose a model architecture that uniﬁes training from labels Y and inputs X .

Original: To increase evaluation robustness, we compare models of the same parameter and data sizes as suggested by Musgrave et al.
Revised: To increase evaluation robustness, we compare models of the same parameter and data sizes as suggested by Musgrave et al.

Original: (2020), and evaluate on a challenging learning problem as suggested by Linzen (2020);
Revised: (2020), and evaluate on a challenging learning problem as suggested by Linzen (2020);

Original: Hooker (2020).
Revised: Hooker (2020).

Original: Namely, we evaluate against a challenging low-resource, long-tailed, noisy multi-label data settings, where information is always limited, since the long tail grows with data size and because its modeling requires the majority of parameters (Hooker et al., 2020).
Revised: Namely, we evaluate against a challenging low-resource, long-tailed, noisy multi-label data settings, where information is always limited, since the long tail grows with data size and because its modeling requires the majority of parameters (Hooker et al., 2020b).

Original: For robust evaluation, we use a typical training, development, test setup and ﬁrst establish a solid, supervised baseline for many-class multi-label classiﬁcation that is optimised with a set of generalisation techniques proposed by Jiang et al.
Revised: For robust evaluation, we use a typical training, development, test setup and ﬁrst establish a solid, supervised baseline for many-class multi-label classiﬁcation that is optimised with a set of generalisation techniques proposed by Jiang et al.

Original: For evaluation in supervised, few and zero-shot learning scenarios, we analyse and propose evaluation metric choices which are meaningful across all scenarios for broader performance comparisons.
Revised: For evaluation in supervised, few and zero-shot learning scenarios, we analyse and propose evaluation metric choices which are meaningful across all scenarios for broader performance comparisons.

Original: Contributions: 1 We provide a straight-forward method for self-supervised contrastive labelembedding prediction and 2 evaluate it in a challenging, noisy long-tail, low-resource multi-label text prediction scenario.
Revised: Contributions: 1 We provide a straight-forward method for self-supervised contrastive labelembedding prediction and 2 evaluate it in a challenging, noisy long-tail, low-resource multi-label text prediction scenario.

Original: 3 We show that small-scale ‘data-internal’ pretraining (on 8-80MB of text) not only improves supervised performance, but also strongly boosts few and zero-shot learning by increasing self-supervision amounts for small data, rather than increasing data amounts via the standard large-scale external data pretraining approach.
Revised: 3 We show that small-scale ‘data-internal’ pretraining (on 8-80MB of text) not only improves supervised performance, but also strongly boosts few and zero-shot learning by increasing self-supervision amounts for small data, rather than increasing data amounts via the standard large-scale external data pretraining approach.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Large to Web-scale data pretraining is at the core of state-of-the-art methods in computer vision (Chen et al., 2020) and language processing (Rogers et al., 2020; Brown et al., 2020).
Revised: Large to Web-scale data pretraining is at the core of state-of-the-art methods in computer vision (Chen et al., 2020) and language processing (Rogers et al., 2020; Brown et al., 2020).

Original: However, challenges and disadvantages are increasingly being discussed.
Revised: However, challenges and disadvantages are increasingly being discussed.

Original: (i)
Revised: (i)

Original: A requirement of large-scale external text data resources (Yogatama et al., 2019; Schick & Sch ¨ utze, 2020a), (ii) an inability to pretrain recent architectures on small-scale data (Liu et al., 2020; Melis et al., 2020; S¸erbetci et al., 2020), (iii) calls for more challenging evaluation tasks (Linzen, 2020; McCoy et al., 2019) and (iv) diminishing returns of pretraining on large supervised datasets (Wang et al., 2020b).
Revised: A requirement of large-scale external text data resources (Yogatama et al., 2019; Schick & Sch ¨ utze, 2020a), (ii) an inability to pretrain recent architectures on small-scale data (Liu et al., 2020; Melis et al., 2020; S¸erbetci et al., 2020), (iii) calls for more challenging evaluation tasks (Linzen, 2020; McCoy et al., 2019) and (iv) diminishing returns of pretraining on large supervised datasets (Wang et al., 2020b).

Original: To address issue (iii) , challenging evaluations on long-tail prediction (Chang et al., 2019), few-shot (Schick & Sch¨utze, 2020a), or zero-shot (Brown et al., 2020), were recently shown to beneﬁt from self-supervised pretraining, but to-date, require massive, ‘task-external’, pretraining datasets .
Revised: To address issue (iii) , challenging evaluations on long-tail prediction (Chang et al., 2019), few-shot (Schick & Sch¨utze, 2020a), or zero-shot (Brown et al., 2020), were recently shown to beneﬁt from self-supervised pretraining, but to-date, require massive, ‘task-external’, pretraining datasets .

Original: (c) Remarkably, Bansal et al.
Revised: (c) Remarkably, Bansal et al.

Original: (2020) showed that for large ‘data-external’ pretraining, using more self-supervision, not more data, also boosts few-shot performance .
Revised: (2020) showed that for large ‘data-external’ pretraining, using more self-supervision, not more data, also boosts few-shot performance .

Original: This ﬁnding inspired us to collect evidence towards a core question: “Do we need massive data (signals) or just more (diverse) self-supervised learning signals for pretraining?”.
Revised: This ﬁnding inspired us to collect evidence towards a core question: “Do we need massive data (signals) or just more (diverse) self-supervised learning signals for pretraining?”.

Original: We collect evidence by posing three research questions and propose solutions that require designing approaches for issues (i-iii) as follows.
Revised: We collect evidence by posing three research questions and propose solutions that require designing approaches for issues (i-iii) as follows.

Original: One, to address issue (i) , “can increasing self-supervision signals during ‘data-internal’ pretraining on small data, i.e. without large-scale ‘data-external’ pretraining, boost few and zero-shot performance”?
Revised: One, to address issue (i) , “can increasing self-supervision signals during ‘data-internal’ pretraining on small data, i.e. without large-scale ‘data-external’ pretraining, boost few and zero-shot performance”?

Original: Two, to address issue (ii) , “what pretraining objectives and models do we chose that work without large training data”?
Revised: Two, to address issue (ii) , “what pretraining objectives and models do we chose that work without large training data”?

Original: Three, to address issue (iii) , “within what challenging learning scenario should we evaluate while incorporating the now standard “any NLP task as a ‘text-to-text’ problem” paradigm (Raffel et al., 2020)”?
Revised: Three, to address issue (iii) , “within what challenging learning scenario should we evaluate while incorporating the now standard “any NLP task as a ‘text-to-text’ problem” paradigm (Raffel et al., 2020)”?

Original: Fortunately, existing techniques can be extended to address these issues.
Revised: Fortunately, existing techniques can be extended to address these issues.

Original: For example, supervised label embedding prediction (pre-)training enables few and zero-shot learning of subsequent (unseen) supervised tasks.
Revised: For example, supervised label embedding prediction (pre-)training enables few and zero-shot learning of subsequent (unseen) supervised tasks.

Original: However, this requires the ﬁrst (pre-training) task to be supervised , unlike recent large scale self-supervised pretraining methods.
Revised: However, this requires the ﬁrst (pre-training) task to be supervised , unlike recent large scale self-supervised pretraining methods.

Original: Large-scale, self-supervised pretraining and label embeddings can be combined (Chang et al., 2019) to ﬁne-tune externally pretrained BERT models via label embedding prediction to boost long-tail task performance.
Revised: Large-scale, self-supervised pretraining and label embeddings can be combined (Chang et al., 2019) to ﬁne-tune externally pretrained BERT models via label embedding prediction to boost long-tail task performance.

Original: However, BERTs’ contextualized word embeddings did not work as label embeddings ELMOs’ word embeddings had to be used (3.2), further increasing resource requirements.
Revised: However, BERTs’ contextualized word embeddings did not work as label embeddings ELMOs’ word embeddings had to be used (3.2), further increasing resource requirements.

Original: Even worse, when comparing language model pretraining on small text corpora, Transformers (Wang et al., 2020a) largely underperform CNNs and LSTMs (Merity et al., 2017).
Revised: Even worse, when comparing language model pretraining on small text corpora, Transformers (Wang et al., 2020a) largely underperform CNNs and LSTMs (Merity et al., 2017).

Original: Fortunately, Liu et al.
Revised: Fortunately, Liu et al.

Original: (2017) established that label-embedding prediction CNNs boost long-tail prediction, even without modern self-supervision or using large ‘task-external’ data pretraining.
Revised: (2017) established that label-embedding prediction CNNs boost long-tail prediction, even without modern self-supervision or using large ‘task-external’ data pretraining.

Original: Further, Pappas & Henderson (2019); Zhang et al.
Revised: Further, Pappas & Henderson (2019); Zhang et al.

Original: (2018b) used supervised text label-embedding (pre-)training and investigated transfer to subsequent supervised tasks, though not under long-tail evaluation.
Revised: (2018b) used supervised text label-embedding (pre-)training and investigated transfer to subsequent supervised tasks, though not under long-tail evaluation.

Original: Here, label embeddings are average word embeddings over label description words – i.e. label descriptions are required.
Revised: Here, label embeddings are average word embeddings over label description words – i.e. label descriptions are required.

Original: The former added noise contrastive estimation (NCE) (Ma & Collins, 2018) via negative sampling of labels to zero-shot predict rare, unseen classes post supervised pretraining on seen classes.
Revised: The former added noise contrastive estimation (NCE) (Ma & Collins, 2018) via negative sampling of labels to zero-shot predict rare, unseen classes post supervised pretraining on seen classes.

Original: Later, Jiang et al.
Revised: Later, Jiang et al.

Original: (2019) adapted the same idea for zero-shot image classiﬁcation via supervised pretraining on pairs of ‘source’ images and ‘source’ text label descriptions.
Revised: (2019) adapted the same idea for zero-shot image classiﬁcation via supervised pretraining on pairs of ‘source’ images and ‘source’ text label descriptions.

Original: They reduced overﬁtting by additionally pretraining on pairs of ‘source’ image and most similar ‘zero-shot target class’ text descriptions – though this is not technically zero-shot learning because sensible target label text descriptions have to be provided, which when unknown (zero-shot), again leads to the long-tail issue.
Revised: They reduced overﬁtting by additionally pretraining on pairs of ‘source’ image and most similar ‘zero-shot target class’ text descriptions – though this is not technically zero-shot learning because sensible target label text descriptions have to be provided, which when unknown (zero-shot), again leads to the long-tail issue.

Original: All these approaches are loosely based on Matching Networks by Vinyals et al.
Revised: All these approaches are loosely based on Matching Networks by Vinyals et al.

Original: and add various training objectives.
Revised: and add various training objectives.

Original: We thus combine the advantages of self-supervised pretraining for large data with supervised label embedding prediction for smaller data to propose a contrastive self-supervised pretraining via label-embedding prediction usable for small data pretraining .
Revised: We thus combine the advantages of self-supervised pretraining for large data with supervised label embedding prediction for smaller data to propose a contrastive self-supervised pretraining via label-embedding prediction usable for small data pretraining .

Original: We extend the supervised label embedding baseline method by Zhang et al.
Revised: We extend the supervised label embedding baseline method by Zhang et al.

Original: (2018b)
Revised: (2018b)

Original: and add four important changes.
Revised: and add four important changes.

Original: First, we combine label and word embedding look-up tables into one table, as this pushes input words and label(-words) to remain in a shared vector space during training, when predicting dense label(-word) embeddings from dense input word embeddings.
Revised: First, we combine label and word embedding look-up tables into one table, as this pushes input words and label(-words) to remain in a shared vector space during training, when predicting dense label(-word) embeddings from dense input word embeddings.

Original: This ‘dense-to-dense’ prediction of words to label(-words) follows the current “any NLP task as a ‘text-to-text’ prediction” paradigm (Raffel et al., 2020), but avoids constant dense-to-sparse translation into label distributions via a compute intensive softmax.
Revised: This ‘dense-to-dense’ prediction of words to label(-words) follows the current “any NLP task as a ‘text-to-text’ prediction” paradigm (Raffel et al., 2020), but avoids constant dense-to-sparse translation into label distributions via a compute intensive softmax.

Original: Secondly, we thus use a noise contrastive estimation (NCE) objective (Ma & Collins, 2018), replacing softmax normalization with negative sampling of (supervision) labels.
Revised: Secondly, we thus use a noise contrastive estimation (NCE) objective (Ma & Collins, 2018), replacing softmax normalization with negative sampling of (supervision) labels.

Original: Combining NCE and label embeddings allows predicting arbitrarily large class set (long-tails) and unseen classes.
Revised: Combining NCE and label embeddings allows predicting arbitrarily large class set (long-tails) and unseen classes.

Original: While Pappas & Henderson (2019) used NCE for supervised label pretraining, we add self-supervised pseudo-label (word) pretraining.
Revised: While Pappas & Henderson (2019) used NCE for supervised label pretraining, we add self-supervised pseudo-label (word) pretraining.

Original: Because labels and input words occupy the same vector space, we can use pseudo-labels (words) for self-supervised pretraining by sampling positive words from a current text instance, and negative words from adjacent text instances within a mini-batch.
Revised: Because labels and input words occupy the same vector space, we can use pseudo-labels (words) for self-supervised pretraining by sampling positive words from a current text instance, and negative words from adjacent text instances within a mini-batch.

Original: Three, we chose to sample from within a batch to reduce reliance (training bias) on knowing or expecting future and past word or label distribution statistics for the whole dataset, since in a zero-shot evaluation scenario unseen label and input word statistics are unknown.
Revised: Three, we chose to sample from within a batch to reduce reliance (training bias) on knowing or expecting future and past word or label distribution statistics for the whole dataset, since in a zero-shot evaluation scenario unseen label and input word statistics are unknown.

Original: This also adds subsequent learning ﬂexibility because no statistics collection preprocessing is required.
Revised: This also adds subsequent learning ﬂexibility because no statistics collection preprocessing is required.

Original: Fourth, we add k-max pooling as in the CNN long-tail research by Liu et al.
Revised: Fourth, we add k-max pooling as in the CNN long-tail research by Liu et al.

Original: (2017), because it helps during zero-shot learning.
Revised: (2017), because it helps during zero-shot learning.

Original: Such label-embedding based self-supervised pretraining has multiple advantages.
Revised: Such label-embedding based self-supervised pretraining has multiple advantages.

Original: It does not require large or external resources as in (i).
Revised: It does not require large or external resources as in (i).

Original: Its small ‘data-internal’ self-supervised word pseudo label pretraining addresses issue (ii) and enables unsupervised zero-shot learning.
Revised: Its small ‘data-internal’ self-supervised word pseudo label pretraining addresses issue (ii) and enables unsupervised zero-shot learning.

Original: It also markedly boosts few-shot performance without requiring task external supervised annotations as in (i) or supervised embedding transfer as in Pappas & Henderson (2019); Zhang et al.
Revised: It also markedly boosts few-shot performance without requiring task external supervised annotations as in (i) or supervised embedding transfer as in Pappas & Henderson (2019); Zhang et al.

Original: (2018b)
Revised: (2018b)

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: Since label embeddings are a common long-tail prediction technique, which addresses issue (iii), it makes our approach suitable for low-resource, long-tail learning without task external labels or large-scale annotated datasets .
Revised: Since label embeddings are a common long-tail prediction technique, which addresses issue (iii), it makes our approach suitable for low-resource, long-tail learning without task external labels or large-scale annotated datasets .

Original: Finally, label embedding NCE training allows for (dense) ‘text-totext’ training, making it applicable to a variety of tasks.
Revised: Finally, label embedding NCE training allows for (dense) ‘text-totext’ training, making it applicable to a variety of tasks.

Original: We demonstrate the beneﬁts of such a selfsupervised pretraining method and model for self-supervised zero-shot learning ( input X -efﬁciency ) § 6.4 or few-shot learning ( label Y -efﬁciency )
Revised: We demonstrate the beneﬁts of such a selfsupervised pretraining method and model for self-supervised zero-shot learning ( input X -efﬁciency ) § 6.4 or few-shot learning ( label Y -efﬁciency )

Original: § 6.3. measuringan interaction text2label matchernet t.repeat text-encoding.for + ,---labels
Revised: § 6.3. measuringan interaction text2label matchernet t.repeat text-encoding.for + ,---labels

Original: 1text sequence encodernet p-value 1 label encodernet
Revised: 1text sequence encodernet p-value 1 label encodernet

Original: R no fixed number of classes/ labels interaction only one class, match yes/no (BCE) w o r d = l a b e l e m b e dd
Revised: R no fixed number of classes/ labels interaction only one class, match yes/no (BCE) w o r d = l a b e l e m b e dd

Original: i
Revised: i

Original: ng s y
Revised: ng s y

Original: + yy
Revised: + yy

Original: + yp
Revised: + yp

Original: p positive negative pos.
Revised: p positive negative pos.

Original: pseudo neg.
Revised: pseudo neg.

Original: pseudo training labels      y 

Revised: pseudo training labels      y

Original: y 2 y 3 y p
Revised: y 2 y 3 y p

Original:  S ELF - SUPERVISED , CONTRASTIVE DENSE - TO - DENSE TEXT PREDICTION
Revised:  S ELF - SUPERVISED , CONTRASTIVE DENSE - TO - DENSE TEXT PREDICTION

Original: In this section, we propose to use label-embeddings, previously used for supervised learning only (Pappas & Henderson, 2019; Zhang et al., 2018b), and exploit them for self-supervised contrastive pretraining on small-scale data.
Revised: In this section, we propose to use label-embeddings, previously used for supervised learning only (Pappas & Henderson, 2019; Zhang et al., 2018b), and exploit them for self-supervised contrastive pretraining on small-scale data.

Original: This enables contrastive self-supervised pretraining somewhat similar to methods used for large-scale models.
Revised: This enables contrastive self-supervised pretraining somewhat similar to methods used for large-scale models.

Original: However, we only use small-scale ‘task-internal’ data for pretraining, which requires orders of magnitude less data and compute than large-scale, ‘taskexternal’ pretraining approaches.
Revised: However, we only use small-scale ‘task-internal’ data for pretraining, which requires orders of magnitude less data and compute than large-scale, ‘taskexternal’ pretraining approaches.

Original: Most NLP models translate back and forth between discrete words and continuous token embeddings, often using a softmax computation that is limited to predicting classes known at training time.
Revised: Most NLP models translate back and forth between discrete words and continuous token embeddings, often using a softmax computation that is limited to predicting classes known at training time.

Original: To ease learning from small data, our ﬁrst core idea is that text input words w i ∈ x
Revised: To ease learning from small data, our ﬁrst core idea is that text input words w i ∈ x

Original: and labels w ◦ i,l should be mapped into the same word representation space, i.e. drawn from a shared embedding look-up table E , to replace dense to sparse translations with embeddingto-embedding matching.
Revised: and labels w ◦ i,l should be mapped into the same word representation space, i.e. drawn from a shared embedding look-up table E , to replace dense to sparse translations with embeddingto-embedding matching.

Original: Thus turns NLP from a discrete ‘text-to-text’ tasks, as proposed in Raffel et al.
Revised: Thus turns NLP from a discrete ‘text-to-text’ tasks, as proposed in Raffel et al.

Original: (2020), into a ‘dense(text)-to-dense(text)’ task.
Revised: (2020), into a ‘dense(text)-to-dense(text)’ task.

Original: We thus replace learning instance labels y i by their corpus-internally pretrained FastText or randomly initialised word embeddings l ◦ i ∈ L , while others (Pappas & Henderson, 2019) use text descriptions to form label embeddings as the vector average over description word embeddings.
Revised: We thus replace learning instance labels y i by their corpus-internally pretrained FastText or randomly initialised word embeddings l ◦ i ∈ L , while others (Pappas & Henderson, 2019) use text descriptions to form label embeddings as the vector average over description word embeddings.

Original: As a result, pretraining word embeddings also pretrains (favourably initialising) label embeddings .
Revised: As a result, pretraining word embeddings also pretrains (favourably initialising) label embeddings .

Original: Unknown labels ( words), in turn, can be inferred via methods like FastText subword embeddings (Bojanowski et al., 2017).
Revised: Unknown labels ( words), in turn, can be inferred via methods like FastText subword embeddings (Bojanowski et al., 2017).

Original: As outlined visually, left to right in Fig.
Revised: As outlined visually, left to right in Fig.

Original: 1, learning multi-label classiﬁcation then becomes a contrastive learning problem of matching the word-sequence embedding t i of text i 2 , with its c label (word-sequence) embeddings l ◦
Revised: 1, learning multi-label classiﬁcation then becomes a contrastive learning problem of matching the word-sequence embedding t i of text i 2 , with its c label (word-sequence) embeddings l ◦

Original: i = { l ◦ i, 1 , . . .
Revised: i = { l ◦ i, 1 , . . .

Original: l ◦ i,c } 3 , by feeding c text-vs-label combinations [[ t i , l ◦ i, 1 ] , . . .
Revised: l ◦ i,c } 3 , by feeding c text-vs-label combinations [[ t i , l ◦ i, 1 ] , . . .

Original: , [ t i , l ◦ i,c ]] 4 to a binary classiﬁer M 5 for matching.
Revised: , [ t i , l ◦ i,c ]] 4 to a binary classiﬁer M 5 for matching.

Original: This means that instead of predicting c classes at once, we predict a batch of c , single-class, binary classiﬁcations using binary cross entropy 6 , where c needs not be constant across instances i .
Revised: This means that instead of predicting c classes at once, we predict a batch of c , single-class, binary classiﬁcations using binary cross entropy 6 , where c needs not be constant across instances i .

Original: The details of steps 1 to 6 are as follows.
Revised: The details of steps 1 to 6 are as follows.

Original: To train a binary classiﬁer, we need both positive and negative labels.
Revised: To train a binary classiﬁer, we need both positive and negative labels.

Original: Thus, for each text instance w i = { w a , . . .
Revised: Thus, for each text instance w i = { w a , . . .

Original: w z } we want to classify, we need g positive labels w − i
Revised: w z } we want to classify, we need g positive labels w − i

Original: = { w + 1 , . . .
Revised: = { w + 1 , . . .

Original: w + g } ∈ R g and b negative labels w + i = { w − 1 , . . .
Revised: w + g } ∈ R g and b negative labels w + i = { w − 1 , . . .

Original: w − b } ∈ R b to form a label selection vector w ◦ i = { w + ⊕ w − } ∈ R g + b .
Revised: w − b } ∈ R b to form a label selection vector w ◦ i = { w + ⊕ w − } ∈ R g + b .

Original: To indicate positive and negative labels, we also need a g sized vector of ones 1 ∈ R g and a b sized zero vector 0 ∈ R b , to get a class indicator
Revised: To indicate positive and negative labels, we also need a g sized vector of ones 1 ∈ R g and a b sized zero vector 0 ∈ R b , to get a class indicator

Original: I i
Revised: I i

Original: = { 1 ⊕ 0 } ∈ R c
Revised: = { 1 ⊕ 0 } ∈ R c

Original: = g + b .
Revised: = g + b .

Original: Both the text (word) indices w i and the label indices w ◦ i are passed through a shared ‘word-or-label embedding’ look-up-table E 1 , after which they are passed through their respective encoder networks – T as text-sequence encoder, L as label encoder.
Revised: Both the text (word) indices w i and the label indices w ◦ i are passed through a shared ‘word-or-label embedding’ look-up-table E 1 , after which they are passed through their respective encoder networks – T as text-sequence encoder, L as label encoder.

Original: Thus, the text-encoder produces a (single) text embedding vector t i =
Revised: Thus, the text-encoder produces a (single) text embedding vector t i =

Original: T ( E ( w i )) per text instance i 2 .
Revised: T ( E ( w i )) per text instance i 2 .

Original: The label-encoder produces c = g + n label embedding vectors ( l ◦ i ) that form a label-embedding matrix L i =
Revised: The label-encoder produces c = g + n label embedding vectors ( l ◦ i ) that form a label-embedding matrix L i =

Original: [ l +1 , . . .
Revised: [ l +1 , . . .

Original: , l + g , l − 1 , . . .
Revised: , l + g , l − 1 , . . .

Original: , l − b ] ← L ( E ( w ◦ i )) 3 .
Revised: , l − b ] ← L ( E ( w ◦ i )) 3 .

Original: As text-encoder T we use a (CNN → max-k-pooling → ReLU) sub-network, while the label-encoder L is simply an (average-pool) operation, since a single label ( w ◦ i,j ), e.g. ‘multi’-‘label’, can consist of multiple words.
Revised: As text-encoder T we use a (CNN → max-k-pooling → ReLU) sub-network, while the label-encoder L is simply an (average-pool) operation, since a single label ( w ◦ i,j ), e.g. ‘multi’-‘label’, can consist of multiple words.

Original: To compare how similar the text-embedding t i is to each label-embedding l ◦
Revised: To compare how similar the text-embedding t i is to each label-embedding l ◦

Original: i,j , we repeat t i c times and combine text and label embeddings to get a text-vs-label-embedding matrix
Revised: i,j , we repeat t i c times and combine text and label embeddings to get a text-vs-label-embedding matrix

Original: M i
Revised: M i

Original: =
Revised: =

Original: [[ l + i, 1 , t i ] , . .
Revised: [[ l + i, 1 , t i ] , . .

Original: , [ l − i,c , t i ]] 4 that is passed into the matcher network M 5 to produce a batch of c probabilities p i = { σ ( M ( M i ) 1 , . . .
Revised: , [ l − i,c , t i ]] 4 that is passed into the matcher network M 5 to produce a batch of c probabilities p i = { σ ( M ( M i ) 1 , . . .

Original: , σ ( M ( M i ) c } 6 .
Revised: , σ ( M ( M i ) c } 6 .

Original: As the optimisation loss, we the use binary cross entropy (BCE) between p i and I i , i.e. 1 c (cid:80) cl =
Revised: As the optimisation loss, we the use binary cross entropy (BCE) between p i and I i , i.e. 1 c (cid:80) cl =

Original: 1 I i,l · log ( p i,l ) + (1 − I i,l ) · log (1 − p
Revised: 1 I i,l · log ( p i,l ) + (1 − I i,l ) · log (1 − p

Original: i,l ) .
Revised: i,l ) .

Original: Summing BCE over positive and negative (pseudo-)labels is referred to as noise contrastive estimation, as used in representation learning methods across ﬁelds (Ma & Collins, 2018).
Revised: Summing BCE over positive and negative (pseudo-)labels is referred to as noise contrastive estimation, as used in representation learning methods across ﬁelds (Ma & Collins, 2018).

Original: Via pseudo-label embedding pretraining, a model can predict supervised labels absent prior supervision.
Revised: Via pseudo-label embedding pretraining, a model can predict supervised labels absent prior supervision.

Original: This exploits both transfer learning from inputs and labels , using the matcher as a learned similarity function.
Revised: This exploits both transfer learning from inputs and labels , using the matcher as a learned similarity function.

Original: Positive labels w
Revised: Positive labels w

Original: +
Revised: +

Original: i can be supervision labels.
Revised: i can be supervision labels.

Original: Negative labels w − i can be sampled from the positive labels of other instances w + j in the same batch, which avoids needing to know the label set beforehand .
Revised: Negative labels w − i can be sampled from the positive labels of other instances w + j in the same batch, which avoids needing to know the label set beforehand .

Original: Since labels are words , we can sample positive words from the current and negative words from other text instances to get pseudo-labels.
Revised: Since labels are words , we can sample positive words from the current and negative words from other text instances to get pseudo-labels.

Original: Sampling pseudo-labels provides a straight-forward contrastive, partial autoencoding mechanism usable as self-supervision in pretraining or as zero-shot learner .
Revised: Sampling pseudo-labels provides a straight-forward contrastive, partial autoencoding mechanism usable as self-supervision in pretraining or as zero-shot learner .

Original: Because both real and pseudo labels are sampled words, the model does not need to distinguish between them.
Revised: Because both real and pseudo labels are sampled words, the model does not need to distinguish between them.

Original: Instead, learning is controlled by an out-of-model sampling routine for real supervision and pseudo self-supervision labels.
Revised: Instead, learning is controlled by an out-of-model sampling routine for real supervision and pseudo self-supervision labels.

Original: This leads to a second core idea: once inputs X and outputs Y are well initialised, the model Θ can also be better initialised by pretraining via self-supervision.
Revised: This leads to a second core idea: once inputs X and outputs Y are well initialised, the model Θ can also be better initialised by pretraining via self-supervision.

Original: As a result, we can learn supervised, few and zero-shot tasks in a uniﬁed manner .
Revised: As a result, we can learn supervised, few and zero-shot tasks in a uniﬁed manner .

Original:  S MALL , LONG - TAILED DENSE - TO - DENSE TEXT ( LABEL ) PREDICTION
Revised:  S MALL , LONG - TAILED DENSE - TO - DENSE TEXT ( LABEL ) PREDICTION

Original: Since it is our goal to research better zero and few-shot learning approaches for small ‘text-to-text’ pretraining models , we choose a small multi-label question tag prediction dataset as a test bed.
Revised: Since it is our goal to research better zero and few-shot learning approaches for small ‘text-to-text’ pretraining models , we choose a small multi-label question tag prediction dataset as a test bed.

Original: We use the “Questions from Cross Validated” 1 dataset, where machine learning concepts are tagged per question.
Revised: We use the “Questions from Cross Validated” 1 dataset, where machine learning concepts are tagged per question.

Original: This dataset fulﬁlls three requirements: it is small-scale, long-tailed, and entails solving a challenging, noisy ‘text-to-text’ prediction task.
Revised: This dataset fulﬁlls three requirements: it is small-scale, long-tailed, and entails solving a challenging, noisy ‘text-to-text’ prediction task.

Original: There is currently no published baseline for this task .
Revised: There is currently no published baseline for this task .

Original: The classes (tags) and input words are highly long-tailed (imbalanced).
Revised: The classes (tags) and input words are highly long-tailed (imbalanced).

Original: The ﬁrst 20% of labels occur in only 7 ‘head’ classes.
Revised: The ﬁrst 20% of labels occur in only 7 ‘head’ classes.

Original: Tags are highly sparse – at most 4 out of 1315 tags are labelled per question.
Revised: Tags are highly sparse – at most 4 out of 1315 tags are labelled per question.

Original: Word embeddings are pretrained with FastText – details in appendix App.
Revised: Word embeddings are pretrained with FastText – details in appendix App.

Original: A.3.
Revised: A.4.

Original: We use the labelled questions part of the dataset, which has 85k questions and 244k labels.
Revised: We use the labelled questions part of the dataset, which has 85k questions and 244k labels.

Original: What makes this problem particularly challenging is that 80% of the least frequent labels are distributed over 99 .
Revised: What makes this problem particularly challenging is that 80% of the least frequent labels are distributed over 99 .

Original: 5% of classes, as an extreme long tail.
Revised: 5% of classes, as an extreme long tail.

Original: The label density (% of active labels per question) is only 0 .
Revised: The label density (% of active labels per question) is only 0 .

Original: or ≈ 2 . 8 / 1305 possible classes per instance.
Revised: or ≈ 2 . 8 / 1305 possible classes per instance.

Original: For a realistic evaluation setting, we split the dataset diachronically, using the 80% earliest documents for training, the next 10% for development, and the last 10% for testing.
Revised: For a realistic evaluation setting, we split the dataset diachronically, using the 80% earliest documents for training, the next 10% for development, and the last 10% for testing.

Original: Why not large external pretraining?
Revised: Why not large external pretraining?

Original: Real-world, long-tailed datasets are always dominated by a low-learning-resource problem for most classes.
Revised: Real-world, long-tailed datasets are always dominated by a low-learning-resource problem for most classes.

Original: This makes two things obvious: (A) that model learning cannot simply be solved by using massive data sets as the long-tail problem grows as well ; (B) that studying self-supervised pretraining on challenging, but smaller , long-tailed datasets such as this one, is useful for assessing an approach’s ability to learn from complex, real-world data.
Revised: This makes two things obvious: (A) that model learning cannot simply be solved by using massive data sets as the long-tail problem grows as well ; (B) that studying self-supervised pretraining on challenging, but smaller , long-tailed datasets such as this one, is useful for assessing an approach’s ability to learn from complex, real-world data.

Original: Massive data pretraining masks and thus prevents studying these effects.
Revised: Massive data pretraining masks and thus prevents studying these effects.

Original: We thus evaluate the effects of self-supervision in a noisy low-resource setup, also as a response to recent critiques of the evaluation metrics used to assess Web-scale learning (Linzen, 2020; Yogatama et al., 2019).
Revised: We thus evaluate the effects of self-supervision in a noisy low-resource setup, also as a response to recent critiques of the evaluation metrics used to assess Web-scale learning (Linzen, 2020; Yogatama et al., 2019).

Original: As
Revised: As

Original: McCoy et al.
Revised: McCoy et al.

Original: (2019) shows, these evaluation setups are solvable by large-scale pattern overﬁtting, which, they ﬁnd, leads to a ‘Clever Hans effect‘, rather than real task progress.
Revised: (2019) shows, these evaluation setups are solvable by large-scale pattern overﬁtting, which, they ﬁnd, leads to a ‘Clever Hans effect‘, rather than real task progress.

Original:  E XPERIMENTAL SETUP AND METRICS
Revised:  E XPERIMENTAL SETUP AND METRICS

Original: We want to analyse the beneﬁts of self-supervision for (a) fully supervised, (b) few and (c) zero-shot learning in a noisy low-resource, long-tailed, multi-label classiﬁcation setting .
Revised: We want to analyse the beneﬁts of self-supervision for (a) fully supervised, (b) few and (c) zero-shot learning in a noisy low-resource, long-tailed, multi-label classiﬁcation setting .

Original: In this section, we describe suitable evaluation metrics, then discuss results in the next section.
Revised: In this section, we describe suitable evaluation metrics, then discuss results in the next section.

Original: Long-tail evaluation metrics and challenges: Long-tail, multi-label classiﬁcation is challenging to evaluate.
Revised: Long-tail evaluation metrics and challenges: Long-tail, multi-label classiﬁcation is challenging to evaluate.

Original: Many classiﬁcation metrics are unsuitable for evaluating long-tailed datasets.
Revised: Many classiﬁcation metrics are unsuitable for evaluating long-tailed datasets.

Original: They either: (i) misrepresent performance under class imbalance; (ii) do not scale to many classes; or (iii) are only meaningful if the desirable number of classes per instance is known (multi-label classiﬁcation).
Revised: They either: (i) misrepresent performance under class imbalance; (ii) do not scale to many classes; or (iii) are only meaningful if the desirable number of classes per instance is known (multi-label classiﬁcation).

Original: For problem (i) ROC AUC is known to overestimate imbalanced performance (Davis &
Revised: For problem (i) ROC AUC is known to overestimate imbalanced performance (Davis &

Original: Goadrich, 2006; Fern´andez et al., 2018), e.g. ROC AUC test scores were upwards of .98 for most of our models.
Revised: Goadrich, 2006; Fern´andez et al., 2018), e.g. ROC AUC test scores were upwards of .98 for most of our models.

Original: For problem (ii), measures such as F-score require discretisation threshold search for imbalanced prediction problems, i.e. searching for the optimal threshold per class (on a development set), which becomes computationally infeasible.
Revised: For problem (ii), measures such as F-score require discretisation threshold search for imbalanced prediction problems, i.e. searching for the optimal threshold per class (on a development set), which becomes computationally infeasible.

Original: Simply using a 0 .
Revised: Simply using a 0 .

Original: 5 probability threshold drives model selection towards balanced prediction, mismatching the long-tail problem.
Revised: 5 probability threshold drives model selection towards balanced prediction, mismatching the long-tail problem.

Original: Metrics like precision@k handle problem (i-ii), but require knowledge of k , i.e. problem (iii): these metrics can only compare a chosen number of labels k , and cannot handle cases where the correct number of labels per instance varies or is unknown (label distribution shift).
Revised: Metrics like precision@k handle problem (i-ii), but require knowledge of k , i.e. problem (iii): these metrics can only compare a chosen number of labels k , and cannot handle cases where the correct number of labels per instance varies or is unknown (label distribution shift).

Original: To more reliably measure performance under imbalance (i), to avoid unscalable class decision thresholding (ii), and to not optimise models for a set number of labels
Revised: To more reliably measure performance under imbalance (i), to avoid unscalable class decision thresholding (ii), and to not optimise models for a set number of labels

Original: k
Revised: k

Original: per instance (iii), we use the average-precision ( AP ) score.
Revised: per instance (iii), we use the average-precision ( AP ) score.

Original: It is deﬁned as AP =
Revised: It is deﬁned as AP =

Original: (cid:80) n ( R n − R n − 1 ) P n , where P n and R n are the precision and recall at the n th threshold.
Revised: (cid:80) n ( R n − R n − 1 ) P n , where P n and R n are the precision and recall at the n th threshold.

Original: AP measures classiﬁer performance over all decision thresholds , is computationally cheaper than threshold search, and allows for a dynamic number of labels per class.
Revised: AP measures classiﬁer performance over all decision thresholds , is computationally cheaper than threshold search, and allows for a dynamic number of labels per class.

Original: This latter property makes this task especially hard.
Revised: This latter property makes this task especially hard.

Original: A model has to learn when to predict a label, at what rarity, and how many such labels to predict for each instance.
Revised: A model has to learn when to predict a label, at what rarity, and how many such labels to predict for each instance.

Original: We also report the macro-averaged
Revised: We also report the macro-averaged

Original: Brier-Score ( BS ) over all classes, as a scalable, compute-efﬁcient measure of classiﬁer calibration.
Revised: Brier-Score ( BS ) over all classes, as a scalable, compute-efﬁcient measure of classiﬁer calibration.

Original: Though more accurate measures exist, computing them is more involved and they require additional evaluation labour when optimising a speciﬁc supervised dataset, which is not our goal.
Revised: Though more accurate measures exist, computing them is more involved and they require additional evaluation labour when optimising a speciﬁc supervised dataset, which is not our goal.

Original: For both measures, we use their popular scikit-learn implementations 2 .
Revised: For both measures, we use their popular scikit-learn implementations 2 .

Original: A challenging task, even for humans: On the dataset it is hard to guess how many labels per question to tag and how speciﬁc they should be, especially without domain knowledge.
Revised: A challenging task, even for humans: On the dataset it is hard to guess how many labels per question to tag and how speciﬁc they should be, especially without domain knowledge.

Original: Out of the different weighting schemes for average precision, we choose AP micro and AP macro , as they are the most pessimistic (hardest to increase) measures to reduce optimistic evaluation.
Revised: Out of the different weighting schemes for average precision, we choose AP micro and AP macro , as they are the most pessimistic (hardest to increase) measures to reduce optimistic evaluation.

Original: This choice is motivated by the goal of this work, which is to not simply to push end-task performance, but to use supervised learning scores as a proxy to evaluate the effects of pretraining on zero-shot learning as well as data-efﬁciency and speed of supervised and few-shot learning.
Revised: This choice is motivated by the goal of this work, which is to not simply to push end-task performance, but to use supervised learning scores as a proxy to evaluate the effects of pretraining on zero-shot learning as well as data-efﬁciency and speed of supervised and few-shot learning.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we ﬁrst analyse a normal and a strong supervised baseline to minimiseoverly favourable comparison againstour subsequently evaluated  self-supervision enhanced approaches.
Revised: In this section, we ﬁrst analyse a normal and a strong supervised baseline to minimise favourable comparison against subsequently evaluated label-embedding and self-supervision enhanced approaches.

Original: Finally, we analyse the beneﬁts of ‘dataset-internal’ pretraining for few-shot learning, and how the amount of pretraining learning signal and model size affect zero-shot learning.
Revised: Finally, we analyse the beneﬁts of ‘dataset-internal’ pretraining for few-shot learning, and how the amount of pretraining learning signal and model size affect zero-shot learning.

Original: Test scores are reported according to the best dev set average precision score AP micro over all classes.
Revised: Test scores are reported according to the best dev set average precision score AP micro over all classes.

Original: 6. B ASELINE MODEL RESULTS
Revised: 6. B ASELINE MODEL RESULTS

Original: In this section, we establish baseline results ( B ASE ) for a non-learning majority class baseline (ZeroR), a common (‘weak’)
Revised: In this section, we establish baseline results ( B ASE ) for a non-learning majority class baseline (ZeroR), a common (‘weak’)

Original: CNN baseline trained with binary-cross-entropy, and a solid CNN baseline optimised using a set of generalisation techniques proposed by Jiang et al.
Revised: CNN baseline trained with binary-cross-entropy, and a solid CNN baseline optimised using a set of generalisation techniques proposed by Jiang et al.

Original: The ZeroR classiﬁer is useful for establishing a baseline performance under class imbalance – e.g. if a class is present in only 10% of instances, then 90% accuracy is achieved by simply always predicting zero – i.e. the majority class.
Revised: The ZeroR classiﬁer is useful for establishing a baseline performance under class imbalance – e.g. if a class is present in only 10% of instances, then 90% accuracy is achieved by simply always predicting zero – i.e. the majority class.

Original: When doing so on our long-tailed task, where the class majority is always zero, we get an AP micro and AP macro of 0 .
Revised: When doing so on our long-tailed task, where the class majority is always zero, we get an AP micro and AP macro of 0 .

Original: 2% , since out of the 1315 classes, maximally four classes are active per instance.
Revised: 2% , since out of the 1315 classes, maximally four classes are active per instance.

Original: Importantly, this tells us that: (a) simply learning to predict zeros can not
Revised: Importantly, this tells us that: (a) simply learning to predict zeros can not

Original: [Table]
Revised: [Table]

Original: score well on under this metric and (b) that this problem setting is challenging.
Revised: score well on under this metric and (b) that this problem setting is challenging.

Original: Next, we evaluate both a weak and optimised baselines (WB, OB).
Revised: Next, we evaluate both a weak and optimised baselines (WB, OB).

Original: When using a very small CNN as baseline (WB) with max pooling over 10 ﬁlters at ﬁlter sizes 1-3 that feed into a one-layer classiﬁer, we achieved 33 .
Revised: When using a very small CNN as baseline (WB) with max pooling over 10 ﬁlters at ﬁlter sizes 1-3 that feed into a one-layer classiﬁer, we achieved 33 .

Original: 75% AP micro on the test set – after only tuning the learning rate.
Revised: 75% AP micro on the test set – after only tuning the learning rate.

Original: When tuning this baseline for parameters known to increase generalisation using a set of such methods suggested by Jiang et al.
Revised: When tuning this baseline for parameters known to increase generalisation using a set of such methods suggested by Jiang et al.

Original: (2020), we get a more solid test score of 45.01 AP micro and an of 22.81 AP macro .
Revised: (2020), we get a more solid test score of 45.01 AP micro and an of 22.81 AP macro .

Original: The macro result tells us that not all classes perform equally well.
Revised: The macro result tells us that not all classes perform equally well.

Original: Upon closer inspection, we ﬁnd that model performance worsens with increasing class rarity as expected.
Revised: Upon closer inspection, we ﬁnd that model performance worsens with increasing class rarity as expected.

Original: While establishing a solid baseline, we ﬁnd expected limitations of model width, max-k pooling and dropout scale-up, and a conﬁrmation that controlled experiment comparisons that only change one variable at a time, do not sufﬁce to ﬁnd better hyperparameter conﬁgurations.
Revised: While establishing a solid baseline, we ﬁnd expected limitations of model width, max-k pooling and dropout scale-up, and a conﬁrmation that controlled experiment comparisons that only change one variable at a time, do not sufﬁce to ﬁnd better hyperparameter conﬁgurations.

Original: For example, when widening lower layer components and observing a decrease in performance, higher layers should also be made wider to accommodate the additional feature information from lower layers – which is consistent with ﬁndings in Nakkiran et al.
Revised: For example, when widening lower layer components and observing a decrease in performance, higher layers should also be made wider to accommodate the additional feature information from lower layers – which is consistent with ﬁndings in Nakkiran et al.

Original: A more detailed breakdown of this analysis can be found in Table Tab. 2 in the appendix App.
Revised: A more detailed breakdown of this analysis can be found in Table Tab. 2 in the appendix App.

Original: A.
Revised: A.

Original: We explore a considerable amount of hyperparameter conﬁgurations in an effort to compute a solid baseline.
Revised: We explore a considerable amount of hyperparameter conﬁgurations in an effort to compute a solid baseline.

Original: This allows for more robust insights and helps to speed up optimisation of the self-supervised models.
Revised: This allows for more robust insights and helps to speed up optimisation of the self-supervised models.

Original: 6. F ULL SUPERVISION (S(+S)L S ) AS REFERENCE (*) FOR FEW AND ZERO - SHOT LEARNING
Revised: 6. 100% SUPERVISION (SL+SSL SCR ) AS REFERENCE (*) FOR FEW AND ZERO - SHOT (0%)

Original: Tab. 1 show both: models trained F ROM S CRATCH (s), and models that are ﬁrst P RETRAINED (p) using self-supervised word pseudo-labels from text inputs, and afterwards ﬁne-tuned (f) on supervision labels.
Revised: Tab. 1 show both: models trained F ROM S CRATCH (s), and models that are ﬁrst P RETRAINED (p) using self-supervised word pseudo-labels from text inputs, and afterwards ﬁne-tuned (f) on supervision labels.

Original: To ﬁt the supervised end-task (tag prediction), both ﬁne-tuning and training from scratch can either: (1) only ﬁt supervision labels (SL) or (2) jointly ﬁt supervised labels and self-supervised word pseudo-labels (S(+S)L), as described in § 3.
Revised: To ﬁt the supervised end-task (tag prediction), both ﬁne-tuning and training from scratch can either: (4) only ﬁt supervision labels (SL) or (3) jointly ﬁt supervised labels and self-supervised word pseudo-labels (S(+S)L), as described in § 3.

Original: However, before analysing results, we deﬁne a controlled experiment setup using a ﬁxed, but shared hyperparameter setting ‘(*) S(+S)Ls’ as a reference (*).
Revised: However, before analysing results, we deﬁne a controlled experiment setup using a ﬁxed, but shared hyperparameter setting ‘(*) SL+SSLscr’ as a reference (*).

Original: Since S(+S)Ls is the most basic model learning setup that uses both self-supervision and supervision, we use its optimal hyperparameters ‘(*) S(+S)Ls’ as a ﬁxed reference conﬁguration for most subsequent learning setups, as indicated by the ‘params like (*)’ marker.
Revised: Since SL+SSLscr is the most basic model learning setup that uses both self-supervision and supervision, we use its optimal hyperparameters ‘(*) SL+SSLscr’ as a ﬁxed reference conﬁguration for most subsequent learning setups, as indicated by the ‘params like (*)’ marker.

Original: This ensures a more controlled comparison of the effects of pretraining vs. training from scratch, and robust insights on how to design self-supervision during end-task ﬁtting and pretraining.
Revised: This ensures a more controlled comparison of the effects of pretraining vs. training from scratch, and robust insights on how to design selfsupervision during end-task ﬁtting and pretraining.

Original: The (*) reference will hence be used for most few and zero-shot settings.
Revised: The (*) reference will hence be used for most few and zero-shot settings.

Original: When comparing P RETRAINED models with models trained F ROM S CRATCH , we see that under comparable hyperparameters, without setting-speciﬁc parameter tuning, all four learning setups perform similarly within 1 percent point (%p) of each other.
Revised: When comparing P RETRAINED models with models trained F ROM S CRATCH , we see that under comparable hyperparameters, without setting-speciﬁc parameter tuning, all four learning setups perform similarly within 1 percent point (%p) of each other.

Original: We also see that the P RETRAINED  model which uses self-supervision during both pretraining and ﬁne-tuning performs best.
Revised: We also see that the P RETRAINED (5) model which uses self-supervision during both pretraining and ﬁnetuning performs best.

Original: Training F ROM S CRATCH using self+supervision S(+S)Ls somewhat hurts performance compared to using supervision alone in SLs.
Revised: Training F ROM S CRATCH using self+supervision SL+SSLscr somewhat hurts performance compared to using supervision alone in SLscr.

Original: Test scores are reported for the best dev set AP micro scores.
Revised: Test scores are reported for the best dev set AP micro scores.

Original: 6. F EW - SHOT : PRETRAIN FOR BETTER LONG - TAIL , LOW - RESOURCE , FEW - SHOT LEARNING
Revised: 6. F EW - SHOT : PRETRAIN FOR BETTER LONG - TAIL , LOW - RESOURCE , FEW - SHOT LEARNING

Original: In this section, we present evidence that even in a data-limited, long-tailed setting, self-supervised ‘data-internal’ pretraining: (a) increases few-shot learning performance of subsequent ﬁne-tuning, while (b) improving learning speed and stability.
Revised: In this section, we present evidence that even in a data-limited, long-tailed setting, self-supervised ‘data-internal’ pretraining: (a) increases few-shot learning performance of subsequent ﬁne-tuning, while (b) improving learning speed and stability.

Original: This demonstrates that small data pretraining has similar beneﬁts as large-scale pretraining (Brown et al., 2020; Schick & Sch ¨ utze, 2020a).
Revised: This demonstrates that small data pretraining has similar beneﬁts as large-scale pretraining (Brown et al., 2020; Schick & Sch ¨ utze, 2020a).

Original: In Fig. 2,
Revised: In [Figure]

Original: [Figure]
Revised: [Figure]

Original: AP m i c r o t e s t when using the (*) reference model from Tab. 1, we now compare training from scratch  as before (pretraining off, left), with pretraining via self-supervised word pseudo-labels, and then ﬁne-tuning on the supervised training labels  of the end-task (pretraining on).
Revised: Fig. 2, when using the (*) reference model from Tab. 1, we now compare training from scratch (4) as before (pretraining off, left), with pretraining via self-supervised word pseudo-labels, and then ﬁne-tuning on the supervised training labels (5) of the end-task (pretraining on).

Original: Note that our model architecture (Fig. 1) does not distinguish between self-supervised and supervised labels, which means that during self-supervised pretraining, we sample as many word pseudo-labels as real labels during supervised ﬁne-tuning (or when supervising from scratch).
Revised: Note that our model architecture (Fig. 1) does not distinguish between self-supervised and supervised labels, which means that during self-supervised pretraining, we sample as many word pseudo-labels as real labels during supervised ﬁne-tuning (or when supervising from scratch).

Original: When ﬁne-tuning the pretrained model on an increasingly difﬁcult F EW -S HOT portion of (100%), 75%, 50%, 25% and only 10% of the supervised training data, we see large AP micro | macro test performance improvements compared to training F ROM S CRATCH in both Tab. 1 and Fig.
Revised: When ﬁne-tuning the pretrained model on an increasingly difﬁcult F EW -S HOT portion of (100%), 75%, 50%, 25% and only 10% of the supervised training data, we see large AP micro | macro test performance improvements compared to training F ROM S CRATCH in both Tab. 1 and Fig.

Original: On the right, in Fig. 2, we see that the pretrained models start with a higher epoch-0 performance, train faster, are more stable and achieve a markedly better few-shot end performance than the lefthand ‘from scratch’ setting.
Revised: On the right, in Fig. 2, we see that the pretrained models start with a higher epoch-0 performance, train faster, are more stable and achieve a markedly better few-shot end performance than the left-hand ‘from scratch’ setting.

Original: This is conﬁrmed by detailed results for the 10% F EW - SHOT setting in Tab. 1, where pretrained models (SLpf, S(+S)Lpf) achieve ≈ .
Revised: This is conﬁrmed by detailed results for the 10% F EWSHOT setting in Tab. 1, where pretrained models (SSLpre → SLﬁn, SSLpre → SL+SSLﬁn) achieve ≈ .

Original: 18 AP micro | macro test compared to only ≈ . 30 /.
Revised: 18 AP micro | macro test compared to only ≈ . 30 /.

Original: 13 AP micro | macro test for models trained from scratch (see SLs or S(+S)Ls).
Revised: 13 AP micro | macro test for models trained from scratch (compare (7-10).

Original: This means that, when using only 10% supervised labels, pretrained models still retain 38 .
Revised: This means that, when using only 10% supervised labels, pretrained models still retain 38 .

Original: 20 , or roughly 80% , of their fully supervised performance .
Revised: 20 , or roughly 80% , of their fully supervised performance .

Original: This provides evidence to answer the underlying question: “Do we really need more data for pretraining or can we simply increase selfsupervision?”.
Revised: This provides evidence to answer the underlying question: “Do we really need more data for pretraining or can we simply increase self-supervision?”.

Original: Very recent work by Bansal et al.
Revised: Very recent work by Bansal et al.

Original: (2020) has investigated this question for largescale, self-supervised pretraining, where they showed that increasing self-supervision to create “a richer learning signal” beneﬁts few-shot performance of large models.
Revised: (2020) has investigated this question for large-scale, self-supervised pretraining, where they showed that increasing self-supervision to create “a richer learning signal” beneﬁts few-shot performance of large models.

Original: Our results demonstrate that this is also the case for small-scale, non-Transformer pretrained models, even under a much more[Figure] challenging long-tailed learning setting than Bansal et al.
Revised: Our results demonstrate that this is also the case for small-scale, non-Transformer pretrained models, even under a much more challenging long-tailed learning setting than Bansal et al.

Original: (2020) examined.
Revised: (2020) examined.

Original: However, to better understand the beneﬁts of using more self-supervised training signals  and its relation to model size, we examine the zero-shot performance of our pretraining approach in regards to label (signal) amount, network width and zero-shot X data-efﬁciency (low-resource zero-shot performance) – i.e. zero-shot performance when pretraining on fractions of inputs X to forcibly limit self-supervision.
Revised: However, to better understand the beneﬁts of using more self-supervised training signals [Figure] and its relation to model size, we examine the zero-shot performance of our pretraining approach in regards to label (signal) amount, network width and zero-shot X data-efﬁciency (low-resource zero-shot performance) – i.e. zero-shot performance when pretraining on fractions of inputs X to forcibly limit self-supervision.

Original: 6. Z ERO - SHOT : MORE IS BETTER , FOR ‘ LOW - RESOURCE ’ ZERO - SHOT PRETRAIN LONGER
Revised: 6. Z ERO - SHOT : MORE IS BETTER , FOR ‘ LOW - RESOURCE ’ ZERO - SHOT PRETRAIN LONGER

Original: In this experiment, we study how the number of self-supervised labels (signal) and the model width used for self-supervised pretraining affects zero-shot performance on the end-task test set.
Revised: In this experiment, we study how the number of self-supervised labels (signal) and the model width used for self-supervised pretraining affects zero-shot performance on the end-task test set.

Original: We show results in both Fig.
Revised: We show results in both Fig.

Original: 2  and Tab. {+
+}
Revised: 2 (11, 12) and Tab. {+
+}

Original: (Z ERO - SHOT ).
Revised: (Z ERO - SHOT ).

Original: In Fig.
Revised: In Fig.

Original: 2, we see that when using the reference hyperparameter conﬁguration ((*) in Tab. 1), pretraining gets the lowest zero-shot performance.
Revised: 2, we see that when using the reference hyperparameter conﬁguration ((*) in Tab. 1), pretraining gets the lowest zero-shot performance.

Original: When increasing the number of self-supervised word pseudo-labels from 150 to 500, the model performs better (middle curve), while not using more parameters – so increasing self-supervision signals is beneﬁcial .
Revised: When increasing the number of self-supervised word pseudo-labels from 150 to 500, the model performs better (middle curve), while not using more parameters – so increasing self-supervision signals is beneﬁcial .

Original: When additionally tripling the network’s sequence and label encoder width and doubling the label match classiﬁer size, zero-shot performance increases even more (top curve).
Revised: When additionally tripling the network’s sequence and label encoder width and doubling the label match classiﬁer size, zero-shot performance increases even more (top curve).

Original: This indicates that for zero-shot learning performance from pretraining, both the amount of training signals and model size have a signiﬁcant impact .
Revised: This indicates that for zero-shot learning performance from pretraining, both the amount of training signals and model size have a signiﬁcant impact .

Original: While increased model size has been linked to increased zeroshot performance of Web-scale pretrained models like GPT-3 (Brown et al., 2020), the inﬂuence of signal amount on zero-shot learning is much less well understood, because large-scale pretraining research often increases training data size when changing self-supervision, as outlined by Liu et al.
Revised: While increased model size has been linked to increased zero-shot performance of Web-scale pretrained models like GPT-3 (Brown et al., 2020), the inﬂuence of signal amount on zero-shot learning is much less well understood, because large-scale pretraining research often increases training data size when changing self-supervision, as outlined by Liu et al.

Original: Finally, in Fig.
Revised: Finally, in Fig.

Original: 3 we see that when pretraining our model for zero-shot prediction on only portions (100%, 75%, .50%, 25% and 10%) of the training text inputs X , i.e. an increasingly low-resource zero-shot setting, we still converge towards comparable full zero-shot performance (if we had not stopped early).
Revised: 3 we see that when pretraining our model for zero-shot prediction on only portions (100%, 75%, .50%, 25% and 10%) of the training text inputs X , i.e. an increasingly low-resource zero-shot setting, we still converge towards comparable full zero-shot performance (if we had not stopped early).

Original: However, each reduction in training size multiplies the required training time – when using the same number of self-labels.
Revised: However, each reduction in training size multiplies the required training time – when using the same number of self-labels.

Original: This provides a promising insight into selfsupervised pretraining on small datasets, which, if designed appropriately, can be used to pretrain well-initialised models for supervised ﬁne-tuning and few-shot learning from very small text sizes.
Revised: This provides a promising insight into selfsupervised pretraining on small datasets, which, if designed appropriately, can be used to pretrain well-initialised models for supervised ﬁne-tuning and few-shot learning from very small text sizes.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We showed that label-embedding prediction, modiﬁed for self-supervised pretraining on a challenging long-tail, low-resource dataset substantially improves low-resource few and zero-shot performance.
Revised: We showed that label-embedding prediction, modiﬁed for self-supervised pretraining on a challenging long-tail, low-resource dataset substantially improves low-resource few and zero-shot performance.

Original: We ﬁnd that increased self-supervision, in place of increased data size or resorting to largescale pretraining, strongly boosts few and zero-shot performance, even in challenging settings.
Revised: We ﬁnd that increased self-supervision, in place of increased data size or resorting to largescale pretraining, strongly boosts few and zero-shot performance, even in challenging settings.

Original: In future, we envision that the proposed methods could be applied in scenarios where little in-domain (pre-)training data is available, e.g. in medicine (S¸erbetci et al., 2020), and where new labels rapidly emerge at test time, e.g. for hashtag prediction (Ma et al., 2014).
Revised: In future, we envision that the proposed methods could be applied in scenarios where little in-domain (pre-)training data is available, e.g. in medicine (S¸erbetci et al., 2020), and where new labels rapidly emerge at test time, e.g. for hashtag prediction (Ma et al., 2014).

Original: The code and data splits will be published on https://github.com .
Revised: The code and data splits will be published on https://github.com .

Original: A A PPENDIX A. A BASELINE TUNED USING GENERALISATION TECHNIQUES
Revised: A A PPENDIX A. A BASELINE TUNED USING GENERALISATION TECHNIQUES

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For the baseline we found optimal hyperparameters to be: lr=0.0075, ﬁlter-sizes= { 1: 57, 2: 29, 3: 14 } , clf=one layer classiﬁer, ’conf’:
Revised: For the baseline we found optimal hyperparameters to be: lr=0.0075, ﬁlter-sizes= { 1: 57, 2: 29, 3: 14 } , clf=one layer classiﬁer, ’conf’:

Original: [ { ’do’:.2 } ] , max-k pooling=3, bs=1536, tune embedding=
Revised: [ { ’do’:.2 } ] , max-k pooling=3, bs=1536, tune embedding=

Original: True, optimizer=ADAM with pytorch defaults.
Revised: True, optimizer=ADAM with pytorch defaults.

Original: Increasing the ﬁlter size, classiﬁer size or depth or using more k decreased dev set performance due to increased overﬁtting.
Revised: Increasing the ﬁlter size, classiﬁer size or depth or using more k decreased dev set performance due to increased overﬁtting.

Original: In general the standard multilabel BCE loss overﬁt much more quickly than the contrastive methods discribed in § 3.
Revised: In general the standard multilabel BCE loss overﬁt much more quickly than the contrastive methods described in § 3.

Original: The contrastive model only differs it was able to use more ﬁlters { 1: 100, 2: 100, 3: 100 } , where using only { 1: 20, 2: 20, 3: 20 } loses 1.5 %p of performance, and that its optimal lr = 0.0005, while the batch size shrinks to 1024 due to increased memory requirements of label matching.
Revised: The contrastive model only differs it was able to use more ﬁlters { 1: 100, 2: 100, 3: 100 } , where using only { 1: 20, 2: 20, 3: 20 } loses 1.5 %p of performance, and that its optimal lr = 0.0005, while the batch size shrinks to 1024 due to increased memory requirements of label matching.

Original: This contrastive models optimal matcher classiﬁer is deeper, due to the increased task complexity – four layer classiﬁer, ’conf’:
Revised: This contrastive models optimal matcher classiﬁer is deeper, due to the increased task complexity – four layer classiﬁer, ’conf’:

Original: [ { ’do’: 0.2 } , { ’out dim’: 1024, ’do’: 0.1 } , { ’out dim’: 300, ’do’:
Revised: [ { ’do’: 0.2 } , { ’out dim’: 1024, ’do’: 0.1 } , { ’out dim’: 300, ’do’:

Original: None } , { ’out dim’: 1, ’do’:
Revised: None } , { ’out dim’: 1, ’do’:

Original: None } ] } .
Revised: None } ] } .

Original: A. F EW - SHOT : SCRATCH , PRETRAINED , ADDITIONAL SELF + SUPERVISED SCENARIOS
Revised: A. L ABEL - EMBEDDINGS , PRETRAINING EFFECTS ON THE LONG - TAIL

Revised: In this section we analyse the effects of using supervised label-embeddings and self-supervised pretraining with words as pseudo-labels. Plotting the average precision of 1305 would be unreadable. [Figure]

Revised: Instead, we sort classes from frequent to rare and assign them to one of ﬁve 20% class frequency buckets, such that each bucket has the same amount of positive labels (label occurrences) in it.

Revised: As seen in Fig.

Revised: 4, this means that the head 0 − 20% bucket (left, blue) has very few, frequent classes, while tail buckets 20 − 40% . . .

Revised: 80 − 100% have increasingly more classes (right, orange, red) that also become increasingly more rare.

Revised: We bucket classes this to balance label frequency between buckets to make them directly comparable.

Revised: AP macro test for each 20% class buckets:

Revised: [Figure]

Revised: → tail

Revised: [Figure]

Revised: head to tail class bins, the BCE multi-label objective performs worst.

Revised: Label-embedding NCE (LE, (4-6)) markedly boosts performance, especially on the long-tail.

Revised: When using label embeddings for self-supervised pretraining with the same network parameters (*) for all LE models, there is no boost.

Revised: However, when pretraining with more parameters ((5.XL) – see larger net, 3.3x labels in the zero-shot learning Fig. 3), we see substantial long-tail performance boosts (turquoise, upper most curve).

Revised: Label-embedding increase long-tail performance:

Revised: In Fig.

Revised: 5 we can see that the optimized baseline (2) from Tab. 1 performs much worse than models that use only the supervised label-embeddings (LE) and methods that also use self-supervised pretraining (PT) via noise contrastive sampling of input words as pseudo labels.

Revised: We also see that regarding end-task performance on the tag predition task, training from scratch (LE, pink × , (cid:5) ) performs only slightly worse than ﬁne-tuning after selfsupervised pretraining (purple × , (cid:5) ).

Revised: However, we also see that increasing the model size during self-supervised pretraining (5.XL) boost performance on the long-tail, especially with increasingly tailed or rare (60-100%) classes.

Revised: Previously, we saw in Fig.

Revised: 3 that the same ”larger net, 3.3x labels” model (5.XL) increased zero-shot performance over the default parameters (*), which demonstrates that improved self-supervised zero-shot performance translates into better supervised end-task ﬁnetuning performance.

Revised: We also found that increasing the size of non-pretrained (trained from scratch) models did not improve end-task performance, despite hyper-parameter tuning.

Revised: This leaves us with two insights for modeling.

Revised: First, for small-scale, ‘data-internal’, self-supervised pretraining a larger pretraining model increases long-tail performance, whereas Hooker et al.

Revised: (2020a) found that compressing larger models ﬁrst ’forgets’ long-tail performance – both experiments provide evidence that model capacity and long-tail performance are tied.

Revised: This seems to be the case even for small-scale self-supervised pretraining, i.e. it demonstrates that despite training on small data, we still need increased self-supervision signals and model size to capture long-tail information, which could explain why large-scale pretraining and models perform so well rather than simply assuming them to be overparameterized.

Revised: Second, this larger pretraining model has an end-task AP m icro score of 49% , which is only .

Revised: 8 percent points better than the pretrained model (5) at 48 .

Revised: 2% with default parameters (*), despite showing promising improvements on long-tail classes, which together with the zero and few-shot insights underlines that optimizing for learning insights and analysis other than supervised performance summary metrics can lead to a broader understanding of neural learning processes and modeling effects.

Revised: The head is learned ﬁrst (in early epochs), pretraining learns the tail much faster:

Revised: In Fig.

Revised: 6 we compare early epoch training with late (optimal) epoch test scores per class bucket.

Revised: We see that all models learn the head classes during the ﬁrst epochs (- - dashed line).

Revised: Methods (5, 5.XL) that use label-embedding (LE) and self-supervised pretraining (PT), start learning the long-tail during the ﬁrst epoch, while BCE multi-label baseline (2) does not start learning the long-tail until epoch 10, and even then at a much lower performance than the pretrained label-embedding methods.

Revised: Finally, we see that pretraining a larger model with more self-supervised pretraining signal (5.XL or “larger net, 3.3x labels” in Fig. 3) increases performance on the long-tail, even during the ﬁrst epochs.

Revised: [Figure]

Revised: Self-supervised label-embedding pretraining boosts long-tail performance: We thus conclude, that self-supervised pretraining helps us learn the long-tail better, and faster – i.e. even after a single epoch of supervised training.

Revised: This is a useful property in supervised learning scenarios where data or computation cycles are limited.

Revised: We note that: our pretraining and ﬁne-tuning do not require learning rate scheduling or normalization layers like BERT or RoBERTa (Devlin et al., 2019;

Revised: Wang et al., 2020c) .

Revised: Miscellaneous result/ an open future evaluation problem: Finally, since average precision ( AP ) has no explicit notion of over and under predictions, we plotted over and under predictions per class and over all classes.

Revised: In standard classiﬁcation, i.e. discrete, evaluation we would have over predictions as false positives and under predictions as false negatives.

Revised: Using continuous measures such as AP has computational advantages and does not limit evaluation to a single threshold like F 1 or accuracy do.

Original: Few-shot challenges: Few-shot learning increases the long-tail problem.
Revised: However, has no notion of over and under predictions, which especially regarding long-tail issues may have a signiﬁcant impact. However, plotting over and underpredictions per class (and overall) was only mildly informative. Due to the high label sparsity, we saw close to zero over predictions on average, but all-class average underpredictions were much harder to reduce (optimize). This observation is a reﬂection of the high label sparsity, i.e. at most 0 .

Original: For 10% few shot learning, we train on 6800 instances, so many classes will be unseen at training time We will publish both the parsed data splits and a cleaned code version on Github to encourage experimenting with and 
Revised: 3 of labels are active per instance, combined with a long-tail distribution – i.e. many rare events. Under this combination meaningful evaluation of prediction behaviour is hard to analyse in a meaningful, concise fashion, because per class plots become large and hard to ﬁt and interpret in a paper, and all-class averages do not reveal class dynamics.

Original: [Figure]
Revised: We include these observations because we found them instructive to outline the challenges of evaluating long-tail learning and do not include the per class plots, because they would not be readable. We are however happy to discuss them upon request. A. F EW - SHOT : SCRATCH , PRETRAINED , ADDITIONAL SELF + SUPERVISED SCENARIOS

Original: [Figure]
Revised: [Figure]

Original:  extending to other low-resource ‘dense-to-dense’ self-supervision methods, additional evaluation metrics and datasets.
Revised: Few-shot challenges: Few-shot learning increases the long-tail problem. For 10% few shot learning, we train on 6800 instances, so many classes will be unseen at training time We will publish both the parsed data splits and a cleaned code version on Github to encourage experimenting with and extending to other low-resource ‘dense-to-dense’ self-supervision methods, additional evaluation metrics and datasets.

Original: Few-shot, with and without self-supervision – as pretraining or for joint self+supervised ﬁne tuning: Fig.
Revised: Few-shot, with and without self-supervision – as pretraining or for joint self+supervised ﬁne tuning: Fig.

Original: 4 shows in more detail that thepretrained model (bottom) learns better, and that joint self+supervised end-task training (scratch or ﬁne-tuned) makes no difference.
Revised: 7 shows in more detail that the

Original: A. T EXT PREPROCESSING DETAILS
Revised: A. T EXT PREPROCESSING DETAILS

Original: We decompose tags such as ‘p-value’ as ‘p’ and ‘value’ and split latex equations into command words, as they would otherwise create many long, unique tokens.
Revised: We decompose tags such as ‘p-value’ as ‘p’ and ‘value’ and split latex equations into command words, as they would otherwise create many long, unique tokens.

Original: 10 tag words are not in the input vocabulary and thus we randomly initialise their embeddings.
Revised: 10 tag words are not in the input vocabulary and thus we randomly initialise their embeddings.

Original: Though we never used this information, we parsed the text and title and annotated them with ‘html-like’ title, paragraph and sentence delimiters.
Revised: Though we never used this information, we parsed the text and title and annotated them with ‘html-like’ title, paragraph and sentence delimiters.

Original: The dataset is ordered and annotated by time.
Revised: The dataset is ordered and annotated by time.

Original: Dev and test set are therefore future data compared to the training data, which results in a non-stationary problem, though we never determined to what extend.
Revised: Dev and test set are therefore future data compared to the training data, which results in a non-stationary problem, though we never determined to what extend.

Original: A. P OTENTIAL ETHICAL CONSIDERATIONS
Revised: A. P OTENTIAL ETHICAL CONSIDERATIONS

Original: In this section, we outline potential impacts of our work for machine learning practice, as well as its possible environmental, societal, health and privacy implications.
Revised: In this section, we outline potential impacts of our work for machine learning practice, as well as its possible environmental, societal, health and privacy implications.

Original: As with any technology there is the dilemma of dual use (Rappert & Selgelid, 2013).
Revised: As with any technology there is the dilemma of dual use (Rappert & Selgelid, 2013).

Original: Below, we brieﬂy discuss beneﬁcial and potential detrimental impacts of this work as we can foresee them Hovy & Spruit (2016);
Revised: Below, we brieﬂy discuss beneﬁcial and potential detrimental impacts of this work as we can foresee them Hovy & Spruit (2016);

Original: Brundage et al.
Revised: Brundage et al.

Original: The main goal of our research is to reduce the hardware and compute requirements of current representation pretraining methodology for language representations, especially for challenging low low-resource, long-tail problems.
Revised: The main goal of our research is to reduce the hardware and compute requirements of current representation pretraining methodology for language representations, especially for challenging low low-resource, long-tail problems.

Original: Due to the reduction in compute requirements, our methods may help reduce carbon impact and the exhaustion of precious resources like rare metal compared to large-scale pretraining.
Revised: Due to the reduction in compute requirements, our methods may help reduce carbon impact and the exhaustion of precious resources like rare metal compared to large-scale pretraining.

Original: Consuming less energy and mining less resources for hardware production has major impacts on the environment as described in detail by Tsing et al. (2017) in “Arts of Living on a Damaged Planet”.
Revised: Consuming less energy and mining less resources for hardware production has major impacts on the environment (Tsing et al., 2017).

Original: Thus, as a research community we should take action not to let AI methods become an arms race for precious metal hardware due to its devastating effects on our shared environment.
Revised: Thus, as a research community we should take action not to let AI methods become a race for precious metal hardware due to its devastating effects on our shared environment.

Original: Further, small-scale pretraining could make access to modern NLP methods easier for machine learning researchers and practitioners, who have less hardware resource privileges than are required for state-of-the-art solutions, or whose language of research does not allow for easy access to Web-scale text collections.
Revised: Further, small-scale pretraining could make access to modern NLP methods easier for machine learning researchers and practitioners, who have less hardware resource privileges than are required for state-of-the-art solutions, or whose language of research does not allow for easy access to Web-scale text collections.

Original: This may become even more important as socio-economic factors are likely to play a fundamental role in the future democratisation and fair access to AI technology (Riedl, 2020) for economics, health and other key decision making areas.
Revised: This may become even more important as socio-economic factors are likely to play a fundamental role in the future democratisation and fair access to AI technology (Riedl, 2020) for economics, health and other key decision making areas.

Original: This is especially important as large-scale hardware resources increasingly lead to research and economic inequalities as described by Hooker (2020);
Revised: This is especially important as large-scale hardware resources increasingly lead to research and economic inequalities as described by Hooker (2020);

Original: Riedl (2020).
Revised: Riedl (2020).

Original: Another important advantage of researching more dataefﬁcient methods is that using as little data as needed is a requirement of the GDPR regulations for ‘privacy by design’.
Revised: Another important advantage of researching more data-efﬁcient methods is that using as little data as needed is a requirement of the GDPR regulations for ‘privacy by design’.

Original: This principle is in direct conﬂict with the current self-supervised pretraining approaches, which parties who have both access to massive data collections and compute resources predominantly study.
Revised: This principle is in direct conﬂict with the current selfsupervised pretraining approaches, which parties who have both access to massive data collections and compute resources predominantly study.

Original: Furthermore, there may be potential implications in better learning of underrepresented and rare events from small or very limited data collections (Mitchell et al., 2020).
Revised: Furthermore, there may be potential implications in better learning of underrepresented and rare events from small or very limited data collections (Mitchell et al., 2020).

Original: When we increase selfsupervision during pretraining, i.e. when pretraining on more diverse learning signals than direct supervision can provide, we see a substantial increase increase in few-shot (low-data) performance, which, upon inspection, becomes clear is caused by a better retention of rare event performance than direct supervision could provide – see Fig. 2.
Revised: When we increase selfsupervision during pretraining, i.e. when pretraining on more diverse learning signals than direct supervision can provide, we see a substantial increase increase in few-shot (low-data) performance, which, upon inspection, becomes clear is caused by a better retention of rare event performance than direct supervision could provide – see Fig. 2.

Original: However, we did not yet study whether this pretraining reduces or increases unwanted data biases (Waseem et al., 2020), though typical analyses of gender and racial biases may be hard on the current dataset of machine learning questions.
Revised: However, we did not yet study whether this pretraining reduces or increases unwanted data biases (Waseem et al., 2020), though typical analyses of gender and racial biases may be hard on the current dataset of machine learning questions.

Original: Note that we did not chose this data set to solve a speciﬁc application task, but only as a proxy to study the effects of small-scale pretraining on challenging data
Revised: Note that we did not chose this data set to solve a speciﬁc application task, but only as a proxy to study the effects of small-scale pretraining on challenging data.

Original: Better small-scale pretraining could beneﬁt areas like medicine where large pretraining is not as effective or fails for a lack of external data resources (S¸erbetci et al., 2020).
Revised: Better small-scale pretraining could beneﬁt areas like medicine where large pretraining is not as effective or fails for a lack of external data resources (S¸erbetci et al., 2020).

Original: Due to the usage duality of research in general, research into more resource-efﬁcient learning could also cause privacy concerns, enabling easier surveillance, and improved advertisement recommendation can have unforeseen political, but also economical and even environmental impacts, as the goal of advertisement is increased resource consumption.
Revised: Due to the usage duality of research in general, research into more resource-efﬁcient learning could also cause privacy concerns, enabling easier surveillance, and improved advertisement recommendation can have unforeseen political, but also economical and even environmental impacts, as the goal of advertisement is increased resource consumption.

Original: Thus, a general approach to furthering beneﬁcial usage over detrimental applications of dual technology should regard applying ethics principles at every step of reuse of the discussed methods to support its transparent use and public veriﬁcation and auditing, to protect vulnerable groups from harmful applications.
Revised: Thus, a general approach to furthering beneﬁcial usage over detrimental applications of dual technology should regard applying ethics principles at every step of reuse of the discussed methods to support its transparent use and public veriﬁcation and auditing, to protect vulnerable groups from harmful applications.