Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models (2020)

Abstract: While internalized “implicit knowledge” in pretrained transformers has led to fruitful progress in many natural language understanding tasks, how to most effectively elicit such knowledge remains an open question. Based on the text-to-text transfer transformer (T5) model, this work explores a template-based approach to extract implicit knowledge for commonsense reasoning on multiple-choice (MC) question answering tasks. Experiments on three representative MC datasets show the surprisingly good performance of our simple template, coupled with a logit normalization technique for disambiguation. Furthermore, we verify that our proposed template can be easily extended to other MC tasks with contexts such as supporting facts in open-book question answering settings. Starting from the MC task, this work initiates further research to find generic natural language templates that can effectively leverage stored knowledge in pretrained models.

## introduction scaling and unifying deep neural models via building task-agnostic architectures and pretraining objectives have brought about significant progress in performing various downstream natural language understanding tasks @xcite @xcite @xcite . according to recent findings, impressive performance can be attributed to a sort of internalized implicit "knowledge base" contained in pretrained models @xcite . since such knowledge is drawn from unstructured and unlabeled text via the design of neural architectures and pretraining objectives, it is non-trivial to identify an effective way to explicitly elicit such stored knowledge for each nlp task. challenging nlp tasks that require commonsense reasoning abilities (or background knowledge) could benefit from implicit knowledge captured in pretrained models. various research lines have explored techniques that effectively leverage reasoning abilities based on bert @xcite . for example, @xcite use a classification head capped on pretrained bert variants to tackle commonsense reasoning. @xcite propose to perturb token embeddings to improve model performance and robustness. alternatively, many researchers recast the multiple-choice (mc) problem setting, which is frequently used in verbal reasoning benchmarks, as a ranking problem regarding statement possibilities @xcite . @xcite further present a hybrid method that combines full-text inputs, pretrained token representations, and learning-to-rank objectives. finally, @xcite propose to unify and fine-tune pretrained models on many qa tasks jointly. as an alternative to designing task-specific architectures on top of pretrained transformers, we propose a new approach to downstream nlp tasks based on task-specific templates exploiting existing pretrained sequence-to-sequence models. we aim to identify a text-to-text approach that effectively elicits implicit knowledge embedded in the pretrained text-to-text transfer transformer source target hypothesis: home is smaller. premise: he never comes to my home, but i always go to his house because the entailment hypothesis: house is smaller. premise: he never comes to my home, but i always go to his house because the contradiction ## approach an approach based on natural language templates enables various options in formulating a verbal reasoning task as a text-to-text problem with t5. our proposed novel formulation builds on t5's mnli template, composed of a hypothesis and a premise. foot_0 consider a concrete example in winogrande: he never comes to my home, but i always go to his house because the _ is smaller. option1: home; option2: house in this case, the correct replacement for _ is option1. we decompose the above problem into two source-target training examples, where _ is replaced with each option and annotated with the correct answer as the target token, as shown in ideally, t5 would produce contrastive tokens for each input pair, as in case (1), which allows us to unambiguously select the final answer. however, the model might produce the same tokens for each input, or even tokens not in the predefined set, as in cases (2) to ( 4 ). to deal with these cases, we apply a softmax over the logits of the pair of predefined target tokens, similar to @xcite . from this, we compute the probabilities of the predefined target tokens (in the case of ## experiments in our experiments, we fine-tune t5-3b on google colab's tpu v2 with a batch size of 16, a learning rate of 2 × 10 -4 , save model checkpoints every 5000 steps and choose the checkpoints with the best performance on the development set. at inference, we use greedy decoding and select for evaluation the model checkpoint that achieves the highest score on the development set. note that we do not experiment with t5-11b due to limited computational resources. we report accuracy as our scoring metric for evaluating model performance. ## conclusions collectively, the success of large pretrained neural models, both encoder-only bert-like architectures as well as encoder-decoder architectures such as t5, raises interesting questions for the pursuit of commonsense reasoning abilities. researchers have discovered that previous models perform well on benchmark datasets because they detect incidental biases in the dataset that have nothing to do with the task; in contrast, the winogrande dataset has devoted considerable effort to reducing such biases, which may allow models to (inadvertently) "cheat" (for example, using simple statistical associations). while it is certainly true that datasets over-estimate the reasoning capabilities of modern models @xcite , there are alternative and complementary explanations as well. it has been a fundamental assumption of the research community that commonsense reasoning is difficult because it comprises tacit rather than explicit knowledge @xcite . that is, commonsense knowledge-like water is wet and that a tuba is usually too big to fit in a backpack-is not written down anywhere (unlike, say, factual knowledge, which can be modeled in a knowledge graph). as a result-the reasoning goes-data-driven techniques (even neural models) will be of limited use due to the paucity of relevant corpora. yet, previous encoder-only architectures @xcite that exploit a language modeling objective (that is, relying only on explicit textual knowledge) can clearly make headway in language reasoning tasks, and we can further improve upon these approaches with a encoder-decoder model. this leaves us with two possible explanations: despite careful controls, the winogrande challenge and other datasets still contain incidental biases that these more sophisticated pretrained models can exploit, or that we are genuinely making at least some progress in commonsense reasoning. the latter, in particular, challenges the notion that commonsense knowledge is (mostly) tacit. perhaps it is the case that in a humongous corpus of natural language text, someone really has written about trying to stuff a tuba in a backpack?