Does the O bjective M atter? C omparing T raining O bjectives for P ronoun R esolution (2020)

Abstract: Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pre-trained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seed-wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.

## introduction hard cases of pronoun resolution have been a longstanding problem in natural language processing, which has served as a performance benchmark for the research community @xcite @xcite . for example, the wino-grande dataset @xcite consists of pronoun resolution schemas that are constructed so that resolving them requires background knowledge and commonsense reasoning. in winogrande, the pronoun is obscured by " " to remove gender and number cues. the task is to find the correct candidate for " " out of two given candidates. for example: john moved the couch from the garage to the backyard to create space. the is small. candidates: garage, backyard. recently, supervised learning on top of pretrained language models has been established as the main approach for pronoun resolution @xcite . under this type of approach, we identify four categories of objectives commonly used for pronoun resolution: we list one representative model from each category. for 1, @xcite use the bert masked language model @xcite to produce the probabilities of the pronoun to be replaced with each of the two candidates. for 2, the unsupervised deep structured semantic model (udssm-i) @xcite uses contextualized word embeddings produced by a bidirectional recurrent neural network (birnn), and then compares the word embedding of each candidate with the word embedding of the pronoun. for 3, roberta-winogrande @xcite encodes a pair of sentences (one for each candidate substituted in the input) by using roberta @xcite to determine which substitution is the correct one. finally, the zero-shot maximum attention score (mas) model @xcite selects a candidate based on how much the pronoun attends to each candidate internally in bert. the problem with all these objectives is that they have not been introduced under the same circumstances. they use different language models and word embeddings (e.g., bert, roberta, or birnn), and have been trained on different data (e.g., dpr @xcite , winogrande, or no additional data). therefore, it is unclear whether the choice of the objective function is essential for pronoun resolution tasks. moreover, the seed-wise stability and the expected performance of these models have usually not been reported. however, seed-wise instability and performance variation are well-known problems when fine-tuning transformer-based models @xcite . in this work, we compare the performance and seed-wise stability of the four categories of training objectives for pronoun resolution on equal grounds. to do this, for category 4, we adapt to training the zero-shot mas model. for category 2, we also introduce coreference semantic similarity (css), which is a simplification and modification of udssm-i for transformer encoders. we select winogrande as our training and development dataset due to its large size (40,938 examples) and generalizability to other pronoun resolution tasks @xcite . we also use for testing the following well-established datasets: the winograd schema challenge dataset (wsc) @xcite and the definite pronoun resolution dataset (dpr) @xcite . we choose as language model roberta @xcite , as it significantly outperforms bert on winogrande, wsc, and dpr @xcite . finally, our evaluations are done under an unprecedentedly large number of seeds (20). ## models this section presents the four training objectives and the models foot_0 that represent each of them. all four models share the roberta foot_1 contextualized word embeddings. roberta has an identical transformer architecture to bert @xcite , with the only difference being the training procedure. hence, roberta is a masked language model that outputs the probability distribution for filling a gap in the text (denoted by a "<@xmath0. additionally, roberta is a text encoder, with one output for each token of the input sentence. three of the models (2.1, 2.3, and 2.4) use a multi-layer perceptron (mlp) classification "head", which takes some part of the encoder as input. all four models use binary cross-entropy loss with a pair of probabilities as input, and the following target labels: sentence correctness for 2.1 and candidate correctness for 2.2, 2.3, and 2.4. ## experiments for all four models, we select the best hyperparameters via grid search using 3 seeds, and then train the models with the best hyperparameters on 20 additional seeds. for winogrande, we use wg-dev (1,267 examples) for selecting the hyperparameters, and wg-train-xl as our training dataset. for all experiments, we use linear learning rate decay with warm-up over 10% of the training data, and the adamw optimizer @xcite , for which we only alter the learning rate. ## results in order to verify the statistical significance of our main results, we used the t-test for similar variances and different sample sizes to compare the distributions of accuracy on the converging seeds. comparing the accuracies of css and wg-sr on wg-dev, wsc, and dpr, respectively, we get the following two-tailed p-values: 0.008249, 0.003026, and 0.017441. all results are significant with @xmath1.05. we also observe that, even with the best hyperparameter combination, wg-sr exhibits seed-wise instability, as it fails to converge on 2 out of 20 seeds. this does not happen to the other three models. after considering 10 additional seeds, we obtained that wg-sr fails to converge on 10% of the seeds (3 out of 30). moreover, during the hyperparameter search, we observed that all models were prone to not converge for certain combinations of hyperparameters. the convergence threshold that we used was selected as @xmath2, and its value was selected based on the performance distribution of all models. we observed that all models either perform around 50% accuracy or 70% accuracy or more on wg-dev. 60% in this context is a good middle ground threshold. wg-sr likely performs better in-domain than css, mas, and bwp, since those three use existing properties of roberta (such as the possibility to compare contextualized embeddings, the attention structure of the model, and its pre-trained lm prediction head, respectively) for a task that they were not originally designed for (pronoun resolution). wg-sr, on the other hand, only uses the output of roberta at the 0-th token, which is not pre-trained. we identify two possible reasons why wg-sr performs worse than css on out-of-domain examples. the first reason is the one mentioned above, namely, not explicitly exploiting the listed properties of the pre-trained model would lead to a better fit on a specific dataset, but worse "general knowledge". this reason is not completely warranted, since wg-sr has similar out-of-domain performance to bwp and mas. the second possible reason is that css uses an explicit candidate localization and candidate-pronoun matching (by comparing the embedding of the candidate and the pronoun), whereas in wg-sr these are achieved implicitly by feeding a pair of sentences to the model, one with the correct and one with the incorrect substitution. again, this reason is not completely warranted, since mas also uses explicit candidate localization and candidate-pronoun matching, but has a similar out-of-domain performance to wg-sr. further investigation on the reasons why css outperforms wg-sr on the out-of-domain examples is left for future work. ## summary and outlook in this work, we categorized four existing objectives for pronoun resolution, and compared their performance and seed-wise stability on equal grounds. our experiments showed that, on indomain testing, the objective of sequence ranking based on the first token in roberta outperforms the other three objectives, but can exhibit convergence problems. on out-of-domain testing, the objective of semantic similarity between the pronoun and each candidate outperforms the other three objectives. future work may investigate whether these results translate to other language models besides roberta as well as other training datasets besides winogrande. also, one could analyze the strengths and weaknesses of each objective, and evaluate other variations of these objectives.