For natural language processing (NLP) taskssuch as sentiment or topic classification, currently prevailing approaches heavily rely on pretraining large self-supervised models on massive external data resources. However, this methodology is being critiqued for: exceptional compute and pretraining data requirements ; diminishing returns on both large and small datasets; and importantly, favourable evaluation settings that overestimate performance differences. The core belief behind current methodology, coined `the bitter lesson' by R. Sutton, is that `compute scale-up beats data and compute-efficient algorithms', neglecting that progress in compute hardware scale-up is based almost entirely on the miniaturisation of resource consumption. We thus approach pretrainingfrom a miniaturisation perspective, such as not to require massive external data sources and models, or learned translations from continuous input embeddings to discrete labels. To minimise overly favourable evaluation, we examine learning on a long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts. To this end, we propose a novel `dataset-internal' contrastive autoencoding approach to self-supervised pretraining and demonstrate marked improvements in zero-shot, few-shot and solely supervised learning performance; even under an unfavorable low-resource scenario, and without defaulting to large-scale external datasets for self-supervision. We also find empirical evidence that zero and few-shot learning markedly benefit from adding more ` dataset-internal' , self-supervised training signals, which is of practical importance when retrieving or computing on large external sources of such signals is infeasible .
For natural language processing 'text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on massive external data sources, which incurs exceptional pretraining data requirements and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learning or preserving minority concept (long-tail) prediction performance along with accordingly designed evaluation scenarios remain open challenges. We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining, which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised, long-tail, few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging long-tailed, low-resource, multi-label text classification scenario with noisy, highly sparse labels and many minority concepts. We find that long-tailed zero and few-shot learning markedly benefit from increasing ' dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources .
R
(NLP) taskssuch as sentiment or topic classification, currently
'text-to-text' tasks, the
R
resources. However, this methodology is being critiqued for: exceptional compute and
sources, which incurs exceptional
R
; diminishing returns on both large and small datasets; and importantly, favourable evaluation settings that overestimate performance differences. The core belief behind current methodology, coined `the bitter lesson' by R. Sutton, is that `compute scale-up beats data and compute-efficient algorithms', neglecting that progress in compute hardware scale-up is based almost entirely on the miniaturisation of resource consumption. We thus approach pretrainingfrom a miniaturisation perspective, such as not to require massive external data sources and models, or learned translations from continuous input embeddings to discrete labels. To minimise overly favourable evaluation, we examine learning on a
and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learning or preserving minority concept (long-tail) prediction performance along with accordingly designed evaluation scenarios remain open challenges. We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining, which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised, long-tail, few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging
R
dataset
scenario
R
rare concepts. To this end, we propose a novel `dataset-internal' contrastive autoencoding approach to self-supervised pretraining and demonstrate marked improvements in zero-shot, few-shot and solely supervised learning performance; even under an unfavorable low-resource scenario, and without defaulting to large-scale external datasets for self-supervision. We also find empirical evidence that
minority concepts. We find that long-tailed
R
adding more `
increasing '
D
,
R
training signals, which is of practical importance when retrieving or computing
pretraining signals, to help reduce the reliance
D
of such signals is infeasible