Precise Task Formalization Matters in W inograd Schema Evaluations (2020)

Abstract: Performance on the Winograd Schema Challenge (WSC), a respected English commonsense reasoning benchmark, recently rocketed from chance accuracy to 89% on the SuperGLUE leaderboard, with relatively little corroborating evidence of a correspondingly large improvement in reasoning ability. We hypothesize that much of this improvement comes from recent changes in task formalization—the combination of input specification, loss function, and reuse of pretrained parameters—by users of the dataset, rather than improvements in the pretrained model’s reasoning ability. We perform an ablation on two Winograd Schema datasets that interpolates between the formalizations used before and after this surge, and find (i) framing the task as multiple choice improves performance dramatically and (ii)several additional techniques, including the reuse of a pretrained language modeling head, can mitigate the model’s extreme sensitivity to hyperparameters. We urge future benchmark creators to impose additional structure to minimize the impact of formalization decisions on reported results.

## introduction over the last couple of years, large pretrained models have achieved human performance on a large share of established natural language understanding benchmark datasets @xcite . recent results report a surge in performance to nearhuman levels on the winograd schema challenge (wsc; @xcite in particular @xcite . however, variations in task formulation across papers and evaluations makes it hard to understand the true degree of recent progress. the wsc is an english commonsense reasoning evaluation that requires a model to resolve * equal contribution. carefully-constructed ambiguous pronouns. for example, in the sentence "jim yelled at kevin because he was so upset." the reader will likely have to consider the motivation of the query noun phrase (np) to recognize whether the pronoun he refers to jim. the accuracy of wsc has seen an abrupt increase from 64% to 89% on the superglue @xcite leaderboard upon the release of roberta @xcite . while many works @xcite @xcite attribute such improvements to improved pretraining and the use of auxiliary training datasets, the impact of the task formalization-the combination of input specification, task specific layer design, and loss function-has not yet been seriously studied. the superglue wsc baseline with bert (64%) resolves pronoun references for individual examples by concatenating the pronoun and query np embeddings and making a binary prediction for the np span. meanwhile, roberta (89%) uses a pretrained masked language modeling (mlm) head as part of the output layer and treats the task as a multiple-choice (mc) decision between candidate nps. we refer to these two task formalizations as pointwise span (p-span) and multiple choice masked language modeling (mc-mlm). in our work, we interpolate between p-span and mc-mlm using both bert and roberta to understand tasks' sensitivity to formalization and the components contributing to mc-mlm's improvement. we find mc-mlm outperforms p-span and reduces sensitivity to hyperparameters and random restarts. we also see large variances of scores spanning random guessing to state-of-the-art (sota) performance. the biggest gain comes from including mc inference. further, paired training with query and candidate nps, using a softmax over candidates, and using a pretrained mlm head all lead to reductions in variance. we show that these formalization choices impact performance differences between the bert and roberta approaches on superglue wsc, with validation accuracy increasing between p-span and mc-mlm by 21.1% using roberta and 10.5% using bert. the effect of task formalization may incentivize gains from supplemental mc options and aggressive hyperparameter tuning. to avoid this, we suggest future benchmarks impose more structure, such as in this case either explicitly distributing gold candidate nps or enforcing rules against their use. for system developers, this result highlights the value of fine-tuning pretrained language modeling heads to target tasks in low-resource settings @xcite , at least where the task format makes this an option. approaches trinh and le (2018) generate inputs for their recurrent neural network language model by replacing the pron with either the query or candidate np and compare the probability of the two sentences, yielding 64% accuracy on wsc273. @xcite use this method with a transformer language model, boosting accuracy to 71%. ruan et al. (2019) for superglue wsc, the official baseline uses bert and a linear classifier on bert's output embeddings for the [cls] token, pronoun token, and query np span representations but fail to exceed the majority-class baseline, only matching it at 64%. liu et al. (2019 ) use the newer roberta and adapt the kocijan et al. (2019) approach with cross entropy loss to raise this accuracy to 89%. t5 (raffel et al., 2020) marks the pronoun in the input and fine-tune a transformer encoder-decoder model to generate the target np, achieving the current state of the art at 94%. looking to winogrande, sakaguchi et al. (2020) adapt ruan et al. (2019)'s method with roberta as the baseline model, achieving 68% accuracy on winogrande-medium and 79% accuracy on the full test set. ## methods under study we evaluate six formalizations-three existing ones and three that we introduce-to interpolate between p-span and mc-mlm. these all use an output layer on top of an mlm pretrained transformer model, but differ in the input specification, loss function, prediction method, contextual embeddings used by the output layer, and label type. mc-mlm this approach follows that of @xcite in the introduction of roberta. here, the pronoun in the input is replaced by @xcite . the model then uses its pretrained mlm head to evaluate the probability np i should replace [mask] and uses a softmax over the log probabilities. for multi-token nps the model compares the geometric mean of these probabilities. mc-sent this approach follows the wino-grande baselines. here, we specify the inputs by replacing the pronoun with an np candidate and marking it with an additional [sep] token. the output head feeds each option's [cls] embedding into a linear layer and applies a softmax over the outputs. mc-sent trains a linear layer from scratch, while mc-mlm may take advantage of the embedding model's mlm pretraining. mc-sent-nosoftmax mc-sent-nosoftmax only differs from mc-sent by replacing the final softmax with a sigmoid and computes the probabilities of whether each input sequence is correct. without softmax, mc-sent-nosoftmax is unable to provide larger gradients for examples with smaller margins between candidates. we refer to this as softmax scaling. mc-sent-nopairloss mc-sent-nopairloss and mc-sent-nosoftmax differ by loss function, where mc-sent-nopairloss only considers the query input. mc-sent-nopairloss is unable to use gradients from multiple candidates to neutralize signals from shared words and focus on np options. we refer to this as paired training. p-sent in p-sent, we further remove mc evaluation by restricting the model to a single binary classification question. this forces p-sent to resolve pronoun references without implicitly learning to detect and eliminate nps. p-span instead of replacing pron with nps to determine the validity of the input sentence, p-span follows the superglue baseline to determine whether the np reference is correct. it first averages over the representations from the pron and np spans to create span representations. the span representations are then concatenated with the [cls] token embedding and used by a logistic regression classifier. ## experiments implementation our code foot_0 builds on huggingface transformers @xcite and fairseq @xcite . all our experiments use either pretrained roberta-large or bert-large-cased models. we evaluate on the validation set every epoch with early stopping. we conduct a random hyperparameter search of 60 trials over the space of learning rate {1e-5, 2e-5, 3e-5}, epoch limit {10, 20, 40}, batch size {8, 16, 32, 64}, and random seed. ## conclusion by only varying task formalization, we observe a wide range of results among reasonable task formalizations on wsc and winogrande evaluations. having access to candidate nps during inference alone improves the performance on superglue wsc. however, models with mc evaluation are highly sensitive to hyperparameters and fail to perform better on winogrande. we find training with paired inputs, using a softmax over candidates, and reusing a pretrained mlm head all help to learn commonsense reasoning and reduce this sensitivity. while we find evidence that these formalization choices can largely influence wsc performance, we do not see obvious evidence of similar occurrences on other task comparisons with roberta. for mc formalizations, we follow liu et al. for wsc and use spacy to mine candidate nps. this extrinsic preprocessing step yields dramatic gains without significantly changing the reasoning ability of the model. we view such gains as orthogonal to the intent of the task and urge benchmark creators to minimize the opportunity for these insubstantial improvements by imposing as much structure as is possible in the released data, for example, by providing candidate nps explicitly. we also encourage future reports of system performances to use the same task formalization whenever possible. at a minimum, greater emphasis should be given to task formalization decisions when they deviate from the prevailing standard. we believe this will help disentangle gains due to models' reasoning abilities, especially in situations where these decisions significantly impact performance, such as in wsc. finally, we find that differences between reasonable formalizations can have big impacts on performance with our case study using wsc. for example, using a pretrained mlm task head as the basis for a downstream task classifier yields strong results with very little hyperparameter sensitivity. this echoes the strong results seen with t5 and offers further motivation to explore these kinds of design decisions in other tasks.