Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction (2021)

Abstract: In this work, we propose Masked Noun-Phrase Prediction (MNPP), a pre-training strategy to tackle pronoun resolution in a fully unsupervised setting. Firstly, We evaluate our pre-trained model on various pronoun resolution datasets without any finetuning. Our method outperforms all previous unsupervised methods on all datasets by large margins. Secondly, we proceed to a few-shot setting where we finetune our pre-trained model on WinoGrande-S and XS separately. Our method outperforms RoBERTa-large baseline with large margins, meanwhile, achieving a higher AUC score after further finetuning on the remaining three official splits of WinoGrande.

## introduction co-reference resolution is an important nlp task that aims to find all expressions that refer to the same entity in a text. the resolution of an ambiguous pronoun, known as pronoun resolution, is a longstanding challenge for the nlu community and an essential step for various high-level nlp tasks such as natural language inference @xcite ), question answering @xcite , and relation extraction @xcite . the most successful approach to pronoun resolution is first fine-tuning a large pre-trained language model such as bert @xcite or roberta @xcite on a human-labeled pronoun resolution dataset such as definite pronoun resolution dataset (dpr) @xcite or winogrande (wg) @xcite , and then either directly transferring to a smaller dataset such as winograd schema challenge (wsc) @xcite or pronoun disambiguation problems (pdp) @xcite the trophy doesn't fit in the suitcase because it is too small. a. the trophy b. the suitcase the trophy doesn't fit in the suitcase because it is too big. a. the trophy b. the suitcase we then proceed to the few-shot setting where we finetune our best zero-shot model on winogrande-s and xs respectively. mnpp gives a large margin of improvements over strong baselines including css @xcite , robertalarge @xcite , and unifiedqa-bart-large @xcite . we further finetune on the remaining three data splits and achieve a higher auc score on all five splits of winogrande over roberta-large baseline. in summary, our main contributions in this work are threefold. ## related works in this work, we mainly compare with unsupervised methods. 2 on wsc, @xcite propose the first unsupervised model where they modify skip-gram @xcite objective to predict semantic dependencies then use this additional information during testing. @xcite propose unsupervised deep structured semantic models (udssm), which utilizes bilstm @xcite to compute contextual word embedding and uses models ensemble. klein and nabi (2019) directly explore the inner attention layers of bert. ye et al. (2019) adapt a masking and predicting strategy, called align, mask, and select (ams), where entities that are connected with conceptnet (speer and havasi, 2012) are masked and the model is required to select from a given list of candidate entities. an ensemble of large pre-trained models is first utilized by trinh and le (2018). gpt-2 is directly evaluated on wsc in radford et al. (2019). prakash et al. (2019) extend a language model with a knowledge hunting strategy. kocijan et al. (2019b) and kocijan et al. (2019a) are the most similar works to us and we will discuss the details in section 3.1. most recently, klein and nabi (2020) study a contrastive self-supervised learning approach (css) for wsc and dpr and also establish the first unsupervised baseline for knowref @xcite . on winogrande, knowledge hunting @xcite and language models ensemble @xcite have been studied. ## masked noun-phrase prediction we treat mnpp as a binary classification task. given the sentence: "she put the cup on the chair, but he knocked over the chair, and the cup fell.", the underlined "the chair" will be masked and a pair of replacement phrases for this masked position is given as {"the cup", "the chair"}. one of the candidates is the masked phrase,"the chair", and the other candidate is a different phrase in the sentence, "the cup" extracted from "she put the cup on the chair". the constraint we impose is that both the ground-truth noun-phrase and the alternative candidate need to appear before the masked phrase location, which mimics the pronoun resolution task. we sample sentences following the above constraint to create our synthetic datasets for pre-training. we convert the sentence into the format of {[cls] first half option second half [sep]} where first half refers to "she put the cup on the chair but he knocked over " and second half refers to ", and the cup fell.". the option is replaced by candidates, "the cup" or "the chair". we compute p(the chair|sentence, θ) and p(the cup|sentence, θ) and optimize θ, the parameters of the model, using cross-entropy loss. we use the final layer @xcite vector from transformer-based language models and pass it through a single layer feed-forward network to calculate the logits. ## conclusion in this work, we propose mnpp pre-training to tackle unsupervised pronoun resolution and study how different properties of the synthetic pretraining dataset impact zero-shot performance on downstream datasets. without finetuning on any pronoun resolution signal, mnpp outperforms all previous fully unsupervised methods on all tasks we study and even several strong supervised baselines. in the few-shot case where we finetune the zero-shot transfer model on winogrande-s and xs respectively, our model outperforms baselines by large margins, and further achieves a higher auc score. this work shows the effectiveness of unsupervised task definitions on text-based pronounresolution and common sense reasoning tasks. it would be interesting to design such tasks for multimodal common sense reasoning @xcite . a related work on supervised methods wsc & dpr. @xcite is the first work to propose transfer learning from another pronoun resolution dataset such as dpr to wsc. he et al. (2019) use a hybrid model of wang et al. (2019b) and kocijan et al. (2019b). ruan et al. (2019) explore bert's next sentence prediction with finetuning on dpr. ye et al. (2020) finetune a new language representation model called corefbert, which is trained with a novel task to strengthen the co-referential reasoning ability of bert, on dpr and then test on dpr and wsc. the sota supervised performance is provided by sakaguchi et al. (2020) where they finetune a roberta-large model on winogrande or dpr and evaluate on wsc and dpr without and with further finetuning. a detailed review of wsc and wsc related dataset can be found at kocijan et al. (2020). knowref. in emami et al. (2019), an end-toend neural system (lee et al., 2018) is trained on conll2012 shared task (pradhan et al., 2012) and then tested under three settings: directly applying to knowref test set, retraining on knowref, and retraining on knowref plus conll2012. sakaguchi et al. (2020) transfer a winogrande-finetuned roberta-large model and dpr-finetuned roberta-large model to knowref test set respectively. winogrande. the authors of winogrande finetune a roberta-large on winogrande training set and evaluate on the test set in standard supervised setting, and lin et al. (2020) finetune a t5-3b model instead. sakaguchi et al. (2020) also study finetuning bert and roberta with only local context (only tokens near the pronoun location are available instead of the whole sentence). ye et al. (2020) finetune winogrande using corefbert. klein and nabi (2020) finetune their unsupervised css model. finally, unifiedqa (khashabi et al., 2020), which is pre-trained on eight seed qa datasets spanning four different formats in a unified way, is finetuned on winogrande.