  Pre-trained text-to-text transformers achieve impressive performance across a
wide range of NLP tasks, and they naturally support zero-shot learning (ZSL) by
using the task description as prompt in the input. However, this approach has
potential limitations, as it learns from input-output pairs at instance level,
instead of learning to solve tasks at task level. Alternatively, applying
existing ZSL methods to text-to-text transformers is non-trivial due to their
text generation objective and huge size. To address these issues, we introduce
Hypter, a framework that improves zero-shot transferability by training a
hypernetwork to generate task-specific adapters from task descriptions. This
formulation enables learning at task level, and greatly reduces the number of
parameters by using light-weight adapters. Experiments on two datasets
demonstrate Hypter improves upon fine-tuning baselines.
