Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020)

Abstract: A standard way to address different NLP problems is by first constructing a problem-specific dataset, then building a model to fit this dataset. To build the ultimate artificial intelligence, we desire a single machine that can handle diverse new problems, for which task-specific annotations are limited. We bring up textual entailment as a unified solver for such NLP problems. However, current research of textual entailment has not spilled much ink on the following questions: (i) How well does a pretrained textual entailment system generalize across domains with only a handful of domain-specific examples? and (ii) When is it worth transforming an NLP task into textual entailment? We argue that the transforming is unnecessary if we can obtain rich annotations for this task. Textual entailment really matters particularly when the target NLP task has insufficient annotations. Universal NLP can be probably achieved through different routines. In this work, we introduce Universal Few-shot textual Entailment (UFO-Entail). We demonstrate that this framework enables a pretrained entailment model to work well on new entailment domains in a few-shot setting, and show its effectiveness as a unified solver for several downstream NLP tasks such as question answering and coreference resolution when the end-task annotations are limited.

## introduction nowadays, the whole nlp journey has been broken down into innumerable sub-tasks. we often solve 1 "universal nlp" here means using a single machine to address diverse nlp problems. this is different from using the same machine learning algorithm such as convolution nets to solve tasks because the latter still results in task-specific models which can not solve other tasks. textual entailment (aka. natural language inference in @xcite ) is the task of studying the relation of two assertive sentences, premise (p) and hypothesis (h): whether h is true given p. textual entailment (te) was originally brought up as a unified framework for modeling diverse nlp tasks @xcite . the research on te dates back more than two decades and has made significant progress. particularly, with the advances of deep neural networks and the availability of large-scale human annotated datasets, fine-tuned systems often claim surpassing human performance on certain benchmarks. nevertheless, two open problems remain. first, the increasing performances on some benchmarks heavily rely on rich human annotations. there is rarely a trained entailment system that can work on benchmarks in other domains. current textual entailment systems are far from being deployed in new domains where no rich annotation exists. second, there is an increasing awareness in the community that lots of nlp tasks can be studied in the entailment framework. but it is unclear when it is worth transforming a target nlp tasks to textual entailment. we argue that textual entailment particularly matters when the target nlp task has insufficient annotations; in this way, some nlp tasks that share the same inference pattern and annotations are insufficient to build a task-specific model can be handled by a unified entailment system. motivated by the two issues, we build ufo-entail-the first ever generalized few-shot textual entailment system with the following setting. we first assume that we can access a largescale generic purpose te dataset, such as mnli @xcite ; this dataset enables us to build a base entailment system with acceptable performance. to get even better performance in any new domain or new task, we combine the generic purpose te dataset with a couple of domain/taskspecific examples to learn a better-performing entailment for that new domain/task. this is a reasonable assumption because in the real-world, any new domain or new task does not typically have large annotated data, but obtaining a couple of examples is usually feasible. technically, our ufo-entail is inspired by the prototypical network @xcite , a popular metric-based meta-learning paradigm, and the stilts @xcite , a framework that makes use of pretraining on indirect tasks to help the target task. ufo-entail consists of a roberta @xcite encoder and a proposed cross-task nearest neighbor block. the roberta, pretrained on mnli, provides a representation space biased to the source domain; the cross-task nearest neighbor block is in charge of mitigating the distribution difference between the source domain and the target task (given only a few examples). in experiments, we apply ufo-entail trained on mnli and k examples from the target domain/task to two out-of-domain entailment benchmarks and two nlp tasks (question answering and coreference resolution). results show the effectiveness of ufo-entail in addressing the challenges set forth in the two questions. overall, we make two contributions: our work differs in that we care more about fewshot applications of textual entailment, assuming that a new domain or an nlp task is not provided with rich annotated data. ## experiments we apply ufo-entail to entailment tasks of open domain and open nlp tasks. experimental setup. our system is implemented with pytorch on the transformers package released by huggingface foot_0 . we use "robertalarge" initialized by the pretrained language model. to mitigate the potential bias or artifacts @xcite in sampling, all numbers of k-shot are average of five runs in seeds {42, 16, 32, 64, 128}. due to gpu memory constraints, we only update the nearest neighbor block, the hidden layer and top-5 layers in roberta. for other training configurations, please refer to our released code. baselines. the following baselines are shared by experiments on open entailment tasks and open nlp tasks. ## summary in this work, we studied how to build a textual entailment system that can work in open domains given only a couple of examples, and studied the common patterns in a variety of nlp tasks in which textual entailment can be used as a unified solver. our goal is to push forward the research and practical use of textual entailment in a broader vision of natural language processing. to that end, we proposed utilizing mnli, the largest entailment dataset, and a few examples from the new domain or new task to build an entailment system via crosstask nearest neighbor. the final entailment system ufo-entail generalizes well to open domain entailment benchmarks and downstream nlp tasks including question answering and coreference resolution. our work demonstrates an example that exploring the uniform pattern behind various nlp problems, enabling us to understand the common reasoning process and create potential for machines to learn across tasks and make easy use of indirect supervision.