Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions (2023)

Abstract: We explore the connection between presupposition, discourse and sarcasm and propose to leverage that connection in a transfer learning scenario with the goal of improving the performance of NLI models on cases involving presupposition. We exploit advances in training transformer-based models that show that pre-finetuning—–i.e., finetuning the model on an additional task or dataset before the actual finetuning phase—–can help these models, in some cases, achieve a higher performance on a given downstream task. Building on those advances and that aforementioned connection, we propose pre-finetuning NLI models on carefully chosen tasks in an attempt to improve their performance on NLI cases involving presupposition. We notice that, indeed, pre-finetuning on those tasks leads to performance improvements. Furthermore, we run several diagnostic tests to understand whether these gains are merely a byproduct of additional training data. The results show that, while additional training data seems to be helping on its own in some cases, the choice of the tasks plays a role in the performance improvements.

## introduction in linguistics and the philosophy of language, presuppositions are beliefs that are mutually shared between discourse participants. they are assumptions that are taken for granted and that play a crucial role in understanding the proper meaning being conveyed in a certain context. for example, in the statement "roger federer won the match", the fact that roger federer played a match (which he ended up winning) is not explicitly stated and assumed, hence, we say it is presupposed. the study of presupposition could be traced to more than a century ago with @xcite . given its importance in understanding meaning and prevalence in everyday speech, scholars studied how presupposition fits into different contexts and aspects of language. specifically, they investigated how presupposition is intricately related to discourse @xcite and essential for understanding not only the meaning of an utterance but also essential for the coherence of the semantic relations between sentences forming a discourse @xcite . similarly, numerous studies underline how presupposition is linked to sarcasm, dating back to two millennia ago with the work of the roman rhetorician quintilian and more recent investigations @xcite . in this work, we set to exploit this connection between presupposition on one hand and discourse and coherence models and sarcasm on the other hand. we choose a popular nlu task, natural language inference (nli) @xcite @xcite as a testbed for exploring this connection and ask the following question: could we leverage learning tasks that center around discourse coherence and sarcasm to improve the ability of nli models at making inferences involving presupposition? to this end, we propose exploring how such learning tasks can be leveraged in a transfer learning scenario that could benefit the training of nli models for the final goal of improving their performances on nli cases involving presupposition. to do so, we utilize pre-finetuning, a training strategy that has received attention in the context of transformer @xcite models such as bert @xcite , roberta @xcite , bart @xcite and gpt-3 @xcite . generally, prefinetuning a pretrained model is to finetune it on a task/dataset before the actual finetuning stage on the downstream task of interest. the effect of pre-finetuning varies depending on different factors, such as whether (and to what extent) a pre-finetuning task is relevant to the fine-tuning task. pre-finetuning has been explored in the context of question answering @xcite , named entities @xcite ) and broader multi-task learning scenarios @xcite . we believe this to be an interesting study for two reasons. first, previous work @xcite suggested that mnli @xcite , a popular nli benchmark, includes some instances of presupposition triggers, thus raising the question of whether transformerbased models trained on mnli can generalize to other pragmatic inferences. however, previous work @xcite found that bert does not perform well on many types of presupposition. this underlines how hard it is to design computational models of pragmatic reasoning. in our case, it is thus interesting to explore ways to improve the performance of nli models on those hard cases of presupposition. second, given the aforementioned intricate connection between the notions of presupposition, discourse and sarcasm, it would be interesting to investigate whether such tasks-expected to be beneficial in principle-would be useful, in practice, for nli involving presuppositions. beyond the motivating reasons for this specific study, finetuning is still a very popular learning paradigm despite the recent rise of other paradigms like prompt-based methods @xcite and recent works continue to investigate ways to improve and apply this paradigm @xcite @xcite . accordingly, we see our effort as complementing the body of work attempting to explore the efficacy and applicability of this learning paradigm. our investigation pursues these questions in the context of three pre-finetuning tasks: discourse relation classification, coherence modeling and sarcasm detection. for each of the tasks, we run experiments on two different datasets. we find that, indeed, pre-finetuning on the selected tasks leads to an improvement in the performance of these models on nli cases involving presupposition given in the imppres dataset @xcite . furthermore, we run several diagnostic tests to understand whether these performance improvements are merely a byproduct of additional training data. the results show that, while additional training data seems to be helping on its own in some cases, the choice of the tasks seems to be playing a role in the performance improvement. our contribution is three-fold. first, we draw this connection between presupposition, discourse and sarcasm, showing its effects in neural models which could be beneficial for future investigations in the computational pragmatics area. second, we present a new case study pointing to a promising direction to train transformer-based models to better handle hard nli cases involving pragmatic phenomena. third, through an extensive array of experiments and diagnostic tests, we show that the type of learning tasks used in pre-finetuning, namely discourse-based and pragmatics-based tasks, could play a crucial role in improving the performance of nli models on cases involving presupposition and show that the performance improvements are not solely due to additional training data, but rather influenced by the learning signal coming from the pre-finetuning task. ## proposed method the learning framework consists of using a pretrained transformer-based model to be finetuned on the mnli dataset. this last stage is preceded by a pre-finetuning stage in which the model is trained on another task different than nli. the model is finally evaluated on the imppres presupposition dataset whose goal is to test nli models' ability to recognize different classes of presupposition. given that the final evaluation is concerned with the model's ability to make pragmatic inferences, we believe an interesting case study would be to use pre-finetuning tasks that deal with aspects of discourse and pragmatics. that way, we could attempt to understand whether relevant knowledge learned from the pre-finetuning stage is passed down to the latter stages. ## experiments in this work, we attempt to answer whether prefinetuning an nli model on learning tasks that deal with aspects of discourse and pragmatics would help that model, when finetuned on mnli, perform better on pragmatic nli cases (the imppres dataset). to do so, we carry out three different experiments. finally, we also run an experiment (multi) where we pre-finetune the model on all three tasks, one after the other. due to limited computational resources, we do not run all possible combinations of datasets. we chose one order at random to be presented as a proof of concept. the chosen sequence of tasks/datasets is: drc/pdtb then cm/wsj then sd/reviews. ## results pretrain: on the other hand, when we use the training data from the pre-finetuning stage for further pre-training (the pretrain experiment) instead of the corresponding learning tasks, we notice that the average performance drops in 4 of the 6 scenarios (pre-finetuning task/dataset) along with a drop in most individual cases (pre-finetuning task/dataset × imppres sub-dataset). this suggests that, in certain cases, the use of extra pre-training data could help in boosting the performance. however, the performance gains due to further pretraining is far from consistent compared to what we saw as a result of using a pre-finetuning stage (i.e. the main experiment). multi: ## sample analysis to understand better the effect of the pre-finetuning stage, we analyze the outputs of the model with and without pre-finetuning. specifically, we look at the models that are pre-finetuned on pdtb (discourse relation classification), wiki (coherence modeling) and the reviews dataset (sarcasm detection). as a case study, we choose the possessed definites existence sub-dataset of imppres as the models performance is the highest on this sub-dataset. we investigate the samples that were incorrectly predicted by the vanilla model (no pre-finetuning) but correctly predicted by the pre-finetuned model for each of these 3 cases. in all 3 cases, there were less than 60 such samples. ## limitations evidently, one limitation of this work is that results are shown for the english language only. in principle, we expect that cross-task knowledge transfer from tasks involving notions of discourse and pragmatics to help in the performance of learned models on downstream tasks that also involve notions of pragmatics (as is the case in this work). however, to what other languages this generalizes to (and to what extent)-especially languages that are morphologically richer than english-remains a question for further exploration. another limitation is that the imppres dataset which is the focus of evaluation in our work is a dataset of semi-automatically generated sentences. the generation process involves pre-specified templates that are to be filled with constituents sampled from a limited vocabulary (3000 lexical items). thus a large-scale quality check process would be required to validate the quality of the entirety of the dataset (e.g., it is mentioned in @xcite ) that the generated sentences often describe highly unlikely scenarios or might include combinations of lexical items that make the sentence sound unnatural).