Precise Task Formalization Matters in Winograd Schema Evaluations
Abstract
Performance on the Winograd Schema Challenge (WSC), a respected English commonsense reasoning benchmark, recently rocketed from chance accuracy to 89% on the SuperGLUE leaderboard, with relatively little corroborating evidence of a correspondingly large improvement in reasoning ability.
We hypothesize that much of this improvement comes from recent changes in task formalization— the combination of input speciﬁcation, loss function, and reuse of pretrained parameters— by users of the dataset, rather than improvements in the pretrained model’s reasoning ability.
We perform an ablation on two Winograd Schema datasets that interpolates between the formalizations used before and after this surge, and ﬁnd (i) framing the task as multiple choice improves performance by 2-6 points and (ii) several additional techniques, including the reuse of a pretrained language modeling head, can mitigate the model’s extreme sensitivity to hyperparameters.
We urge future benchmark creators to impose additional structure to minimize the impact of formalization decisions on reported results.
Introduction
Over the last couple of years, large pretrained models have achieved human performance on a large share of established natural language understanding benchmark datasets (Devlin et al., 2019).
Recent results report a surge in performance to nearhuman levels on the Winograd Schema Challenge (WSC; Levesque, 2011) in particular(Liu et al., 2019).
However, variations in task formulation across papers and evaluations makes it hard to understand the true degree of recent progress.
The WSC is an English commonsense reasoning evaluation that requires a model to resolve carefully-constructed ambiguous pronouns.
For example, in the sentence “ Jim yelled at Kevin because he was so upset. ”
the reader will likely have to consider the motivation of the query noun phrase (NP) to recognize whether the pronoun he refers to Jim .
The accuracy of WSC has seen an abrupt increase from 64% to 89% on the SuperGLUE (Wang et al., 2019)
leaderboard upon the release of RoBERTa (Liu et al., 2019).
While many works (Kocijan et al., 2019; Liu et al., 2019; Raffel et al., 2020) attribute such improvements to improved pretraining and the use of auxiliary training datasets, the impact of the task formalization—the combination of input speciﬁcation, task speciﬁc layer design, and loss function—has not yet been seriously studied.
The SuperGLUE WSC baseline with BERT (64%) resolves pronoun references for individual examples by concatenating the pronoun and query NP embeddings and making a binary prediction for the NP span.
Meanwhile, RoBERTa (89%) uses a pretrained masked language modeling (MLM) head as part of the output layer and treats the task as a multiple-choice (MC) decision between candidate NPs.
We refer to these two task formalizations as pointwise span ( P-Span ) and multiple choice masked language modeling ( MC-MLM ).
In our work, we interpolate between P-Span and
MC-MLM using both BERT and RoBERTa to understand tasks’ sensitivity to formalization and the components contributing to MC-MLM’s improvement.
We ﬁnd MC-MLM outperforms P-Span and reduces sensitivity to hyperparameters and random restarts.
We also see large variances of scores spanning random guessing to state-of-the-art (SotA) performance.
The biggest gain comes from including MC inference.
Further, paired training with query and candidate NPs, using a softmax over candidates, and using a pretrained MLM head all
Ablation
Formalization
[Table]
lead to reductions in variance. We show that these formalization choices impact performance differences between the BERT and RoBERTa approaches on SuperGLUE WSC, with validation accuracy increasing between P-Span and MC-MLM by 21.1% using RoBERTa and 10.5% using BERT. The effect of task formalization may incentivize gains from supplemental MC options and aggressive hyperparameter tuning.
To avoid this, we suggest future benchmarks impose more structure, such as in this case either explicitly distributing gold candidate NPs or enforcing rules against their use.
For system developers, this result highlights the value of ﬁne-tuning pretrained language modeling heads to target tasks in low-resource settings (Raffel et al., 2020), at least where the task format makes this an option.
Related Work
WSC Datasets
Levesque et al.
(2012) launch the
WSC with 108 handbuilt question-answer pairs, which has since grown to 273 examples, often called WSC273.
Since then, several similar or derived datasets have emerged (Kocijan et al., 2020).
The SuperGLUE version of the task recasts examples from WSC273 and the Pronoun Disambiguation Problems dataset (Morgenstern et al., 2016) into 554 training and 104 validation binary classiﬁcation problems.
146 test examples are derived from ﬁction books and handcrafted by the original
WSC authors.
Sakaguchi et al.
(2020) collect a larger dataset of ﬁll-in-the-blank-format problems, called WinoGrande, via crowdsourcing with adversarial ﬁltering.
The dataset has ﬁve training sets, ranging from 160 to 41k examples, and shared validation and test sets with 1.3k and 1.8k examples, respectively.
Approaches
Trinh and Le (2018) generate inputs for their recurrent neural network language model by replacing the P RON with either the query or candidate NP and compare the probability of the two sentences, yielding 64% accuracy on WSC273.
Radford et al.
(2019) use this method with a transformer language model, boosting accuracy to 71%.
Ruan et al.
(2019) ﬁne-tune BERT in a similar way, reaching 71% as well.
Kocijan et al.
(2019) also use BERT, but include additional Winograd-like training data and use the model’s pretrained MLM head to achieve 74% accuracy.
In another style of approach, Klein and Nabi (2019) experiment with BERT pretrained attention weights without ﬁne-tuning, and achieve an accuracy of 60%.
For SuperGLUE WSC, the ofﬁcial baseline uses
BERT and a linear classiﬁer on BERT’s output embeddings for the [CLS] token, pronoun token, and query NP span representations but fail to exceed the majority-class baseline, only matching it at 64%.
Liu et al.
(2019) use the newer RoBERTa and adapt the Kocijan et al.
(2019) approach with cross entropy loss to raise this accuracy to 89%.
T5 (Raffel et al., 2020) marks the pronoun in the input and ﬁne-tune a transformer encoder-decoder model to generate the target NP, achieving the current state of the art at 94%.
Looking to WinoGrande, Sakaguchi et al.
(2020) adapt Ruan et al.
(2019)’s method with RoBERTa as the baseline model, achieving 68% accuracy on
WinoGrande-Medium and 79% accuracy on the full test set.
Methods under Study
We evaluate six formalizations—three existing ones and three that we introduce—to interpolate between P-Span and MC-MLM.
These all use an output layer on top of an MLM pretrained transformer model, but differ in the input speciﬁcation, loss function, prediction method, contextual embeddings used by the output layer, and label type.
Table 1 presents an overview.
MC-MLM
This approach follows that of Liu et al.
(2019) in the introduction of RoBERTa.
Here, the pronoun in the input is replaced by [MASK] .
The model then uses its pretrained MLM head to evaluate the probability NP i should replace [MASK] and uses a softmax over the log probabilities.
For multi-token NPs the model compares the geometric mean of these probabilities.
MC-Sent
This approach follows the WinoGrande baselines.
Here, we specify the inputs by replacing the pronoun with an NP candidate and marking it with an additional [SEP] token.
The output head feeds each option’s
[CLS] embedding into a linear layer and applies a softmax over the outputs.
MC-Sent trains a linear layer from scratch, while MC-MLM may take advantage of the embedding model’s MLM pretraining.
MC-Sent-NoSoftmax
MC-Sent-NoSoftmax only differs from MC-Sent by replacing the ﬁnal softmax with a sigmoid and computes the probabilities of whether each input sequence is correct.
Without softmax, MC-Sent-NoSoftmax is unable to provide larger gradients for examples with smaller margins between candidates.
We refer to this as softmax scaling.
MC-Sent-NoPairLoss
MC-Sent-NoPairLoss and MC-Sent-NoSoftmax differ by loss function, where MC-Sent-NoPairLoss only considers the query input.
MC-Sent-NoPairLoss is unable to use gradients from multiple candidates to neutralize signals from shared words and focus on NP options.
We refer to this as paired training.
P-Sent
In P-Sent, we further remove MC evaluation by restricting the model to a single binary classiﬁcation question.
This forces P-Sent to resolve pronoun references without implicitly learning to detect and eliminate NPs.
P-Span
Instead of replacing P RON with NPs to determine the validity of the input sentence, PSpan follows the SuperGLUE baseline to determine whether the NP reference is correct.
It ﬁrst averages over the representations from the P RON and NP spans to create span representations.
The span representations are then concatenated with the [CLS] token embedding and used by a logistic regression classiﬁer.
Experiments Implementation
Our code 1 builds on Huggingface Transformers (Wolf et al., 2019) and fairseq (Ott et al., 2019).
All our experiments use either pretrained RoBERTa-large or BERT-large-cased models.
We evaluate on the validation set every epoch with early stopping.
We conduct a random hyperparameter search of 60 trials over the space of learning rate { 1e-5, 2e-5, 3e-5 } , epoch limit { 10, 20, 40 } , batch size { 8, 16, 32, 64 } , and random seed.
Datasets
We run experiments on SuperGLUE
WSC and WinoGrande-Medium.
We do not cover larger WinoGrande sizes due to computation constraints.
Each WSC example includes a sentence, a P RON span, and a single marked NP span.
Following Liu et al., for MC-based formalizations, we mine candidate NPs with spaCy 2 and only keep one example from the group of examples that only differ by query NP to avoid redundancy. Majority P-Span
P-Sent MC-Sent-NoPairLoss MC-Sent-NoSoftmax MC-Sent MC-MLM
Human
[Figure] A cc u r a c y ( % ) (a) WSC (RoBERTa) 63.
Majority P-Span
P-Sent MC-Sent-NoPairLoss MC-Sent-NoSoftmax MC-Sent MC-MLM
Human
A cc u r a c y ( % ) (b) WSC (BERT) 63. Majority P-
[Figure] P- [Figure] MC-Sent-
[Figure] MC- [Figure] [Figure]
MC-Sent MC
[Figure]
Human Formalization
[Figure]
( % ) (c) WinoGrande-Medium (RoBERTa) 50. WSC WinoGrande
Formalization
Test Std Kurt Std Kurt
MC-MLM
MC-SentMC-Sent-NoSoftmaxMC-Sent-NoPairLoss
P-Sent
P-Span    - and excess kurtosis ( Kurt ) for WSC and WinoGrande and test accuracy for WSC using RoBERTa.
Test results are from an ensemble of the top ﬁve models.
For WinoGrande, each example provides a sentence with two marked NP spans and a ﬁll-in-theblank to represent the P RON .
When using asymmetric formalizations like P-Sent, we duplicate each example, making one option the query and the other the candidate.
For P-Span, we use the ﬁrst appearance of query or candidate NP in the sentence as the NP span and use the blank as P RON span.
Figures 1a and 1c and Table 2 show the distribution over validation accuracies from 60 runs with each formalization using RoBERTa. We do not report WinoGrande test results since submissions require test set predictions from all ﬁve training sets and we only train using WinoGrande-Medium.
We also include the majority-class baseline and human performance.
From the WSC test results, we ﬁnd MC-MLM outperforms P-Span.
The 6% gain between P-Sent and MC-Sent-NoPairLoss indicates MC evaluation alone may improve accuracy.
However, we also ﬁnd most formalizations are sensitive to hyperparameter choices and random seeds.
Given the small size of the SuperGLUE WSC test set at 146 examples, we ﬁnd it more informative to focus on the distribution of validation results.
In both datasets, we see three main changes.
First, including paired training with MC-SentNoSoftmax increases performance variance by adding more weight to the tail of higher scores.
Second, we see the weight of higher performances increase even more with softmax scaling in MC-Sent.
In WSC, the higher scores become the body of the distribution with smaller variance.
In WinoGrande, the distribution of MC-Sent has an increased excess kurtosis indicating the tail of higher scores occur more frequently.
Finally, the model achieves higher scores with signiﬁcantly lower variance using MLM in MC-MLM. This may be a result of ﬁne-tuning the pretrained MLM head rather than a new initialization.
We see two main differences between WSC and
WinoGrande results.
First, P-Sent performs significantly worse than P-Span on WinoGrande.
We suspect this is due to WinoGrande’s adversarial ﬁltering removing examples that are easy to classify from sentence representations.
Second, MC-SentNoPairLoss does not beneﬁt WinoGrande and may indicate the beneﬁt from MC evaluation may not extend to other Winograd like corpora. BERT Results Figure 1b shows ablation results using BERT and WSC.
We ﬁnd that RoBERTa outperforms BERT with both models using the same MC-MLM formalization, which is in line with leaderboard performances. We also ﬁnd similar trends across task formalizations in Figure 1a, further highlighting the impact of formalization decisions on performance gains. Most formalizations are still sensitive to hyperparameter choices and random seed, MC evaluation alone provides a beneﬁt over P-Span at the 75 th percentile of roughly 6%, and incorporating MLM provides additional beneﬁts in performance. However, we also ﬁnd that using BERT’s pretrained MLM head does not provide the lower variance displayed with RoBERTa.
Comparing the performances of intermediate formalizations, we see that BERT generally performs worse than
RoBERTa.
This is consistent with the ﬁndings from
Tenney et al.
(2019) that show BERT embeddings encode information less suited for coreference resolution during pretraining.
Consequently, BERT’s pretrained MLM head would be less optimized for a coreference resolution task like WSC than
RoBERTa’s and may not provide the same stability beneﬁts.
By only varying task formalization, we observe a wide range of results among reasonable task formalizations on WSC and WinoGrande evaluations.
Having access to candidate NPs during inference alone improves the performance on SuperGLUE
WSC.
However, models with MC evaluation are highly sensitive to hyperparameters and fail to perform better on WinoGrande.
We ﬁnd training with paired inputs, using a softmax over candidates, and reusing a pretrained MLM head all help to learn commonsense reasoning and reduce this sensitivity.
While we ﬁnd evidence that these formalization choices can largely inﬂuence WSC performance, we do not see obvious evidence of similar occurrences on other task comparisons with RoBERTa.
For MC formalizations, we follow Liu et al.
for
WSC and use spaCy to mine candidate NPs.
This extrinsic preprocessing step yields dramatic gains without signiﬁcantly changing the reasoning ability of the model.
We view such gains as orthogonal to the intent of the task and urge benchmark creators to minimize the opportunity for these insubstantial improvements by imposing as much structure as is possible in the released data, for example, by providing candidate NPs explicitly.
We also encourage future reports of system performances to use the same task formalization whenever possible.
At a minimum, greater emphasis should be given to task formalization decisions when they deviate from the prevailing standard.
We believe this will help disentangle gains due to models’ reasoning abilities, especially in situations where these decisions signiﬁcantly impact performance, such as in WSC.
Finally, we ﬁnd that differences between reasonable formalizations can have big impacts on performance with our case study using WSC. For example, using a pretrained MLM task head as the basis for a downstream task classiﬁer yields strong results with very little hyperparameter sensitivity.
This echoes the strong results seen with T5 and offers further motivation to explore these kinds of design decisions in other tasks. Acknowledgements
We thank Vid Kocijan, Ethan Perez, Jason Phang,
Aishwarya Kamath, Iacer Calixto, Clara Vania,
Rasika Bhalerao, Alex Warstadt, Richard Pang, He
He, and Ernie Davis for their helpful feedback.
This project has beneﬁted from ﬁnancial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), by Samsung Research (under the project
Improving Deep Learning using Latent Structure ), by Intuit, Inc., and in-kind support by the NYU HighPerformance Computing Center and by NVIDIA
Corporation (with the donation of a Titan V GPU).
This material is based upon work supported by the National Science Foundation under Grant No.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the National Science Foundation.