A smashed glass cannot be full: Generation of Commonsense Explanations through Prompt-based Few-shot Learning (2023)

Abstract: We assume that providing explanations is a process to elicit implicit knowledge in human communication, and propose a general methodology to generate commonsense explanations from pairs of semantically related sentences. We take advantage of both prompting applied to large, encoder-decoder pre-trained language models, and few-shot learning techniques, such as pattern-exploiting training. Experiments run on the e-SNLI dataset show that the proposed method achieves state-of-the-art results on the explanation generation task, with a substantial reduction of labelled data. The obtained results open new perspective on a number of tasks involving the elicitation of implicit knowledge.

## introduction when exchanging information, it is typical to exclude details that appear self-evident or insignificant, so that only part of the message is articulated verbally while other details are implied @xcite . this is particularly true for information involving commonsense knowledge, which represents the backbone of everyday communication and reasoning. for example, consider the following sentences: we can intuitively assert that these two sentences contradict each other, and if we were asked to explain why, our answer would most probably appeal to some implicit knowledge about the world ("a glass broken into pieces cannot contain any liquid" and "a glass cannot be broken and full of a liquid at the same time", etc.) to various degrees of depth. this implicit, shared knowledge can easily be inferred by people, but represents a challenge for computational systems. crucially, a request for explanation is often a request to make explicit something that is only implied or omitted in conversation. while this is evident in everyday communication, it is also true for more specialised domains, for example a doctor-patient scenario where the patient is given a diagnosis ("your clinical case and tests indicate that you have type 2 diabetes") and asks for explanations to the doctor ("why do you believe that?"). therefore, the underlying assumption of this paper is that commonsense explanations are to some extent based upon the notion of implicitness, and that the information they rely on can not be fully derived from their textual input alone. being able to elicit commonsense implicit knowledge is a relevant step forward not only towards providing explanations, but also towards better natural language understanding. in this work, we state the "implicit knowledge problem" as the capacity to automatically generate explanations regarding the semantic relations between two sentences. we define a general methodology to elicit implicit knowledge from pre-trained large language models, motivated by the intuition that they contain most of the knowledge needed, and are able to generalize over unseen instances. in fact, a fully-supervised fine-tuning would require a large training set where each input-sentence pair should be labeled with one or more explanations, a setting which is unrealistic in an open-domain scenario. to achieve this, we propose a combination of prompting and few-shot learning techniques, which are well-suited to exploit the generative capabilities of language models and, at the same time, make use of limited supervision. similar methods have been applied to some popular nlp tasks such as text classification, inference, summarization @xcite and, more recently, to teach language models to leverage external tools via apis @xcite . however, the generation of implicit knowledge poses a further challenge to these techniques as it requires to generate text based on information or reasoning structures that are outside the input texts 1 . for these reasons, in our experiments, we select the e-snli dataset @xcite , where sentence pairs traditionally employed for a textual inference task (pairs are labelled with entail, contradict or neutral tags) were augmented with explanations for the relation tag, collected through crowd-sourcing. we compare three generation methods: unsupervised, fine-tuning and ensembling, showing that the ensembling method achieves the best results, en-par with state-of-theart while making use of limited supervision. moreover, we find that this method mitigates the potential negative effects of "bad" prompts, which is a desirable feature whenever prompt optimization is not possible. however, we also underline that evaluation and comparison with sota results is still critical, as evaluation metrics are not usually directly comparable on the explanation generation task. the innovative contributions of this paper are the following: the paper is structured as follows: section 2 provides relevant background on language modelling and elicitation of implicit knowledge. section 3 introduces the general approach to implicit knowledge generation. sections 4 and 5 report, respectively, the experimental setting and the results we have obtained on the e-snli dataset. finally, section 6 provides relevant context of recent approaches to implicit knowledge elicitation. 1 the code is available at github.com/ andreazaninello/explanationgeneration 2 background ## methodology our proposed methods belong to the "fixed-prompt language model tuning" types @xcite : the lm's parameters are updated through few-shot fine-tuning, after both training and test examples have been modified by textual templates, meaning that the prompts' parameters are fixed while the model's parameters are updated. while some recent studies aim at discovering the best prompts or performing prompt optimization @xcite , our methods aim to provide a general framework that may work well even without prompting optimization, minimizing the negative impact of "bad" prompts. finding the best patterns and prefixes would clearly contribute to the task but falls outside the scope of this study, and would be task dependent, while we aim to provide a general framework potentially applicable to different generation tasks. our method is closely related to @xcite 's, who apply a similar procedure to classification and regression tasks, and is inspired by @xcite 's genpet, who apply it to a set of generation tasks close to summarization, not involving the elicitation of implicit external knowledge. ## results and discussion in interestingly, both models have very low scores in both zero-shot configurations, indicating that the "fixed-prompt fine-tune" strategies may be particularly suitable for them, even without prompt optimization, as indicated by the low figures of the null prompt. on the other hand, bart displays a stronger baseline and is more sensitive to prompt design, as displayed by the decreasing values especially relevant on the p 0 -fine-tuning method. this indicates that when using bart, which was not specifically trained to accept prompts as inputs, prompt optimization may be a better strategy. on the other hand, for all three models the en-sembling method is able to mitigate the negative effects of shallow prompt design. while a clear benchmark for the explanation generation task does not yet exist, in we also manually analyzed several generations and compared the different models and methods. we notice that in the unsupervised settings, models tend to hallucinate, while with zero-shots bart and pegasus tend to copy (part of) the input, while t5 often returns single words. in ## related work being a relatively recent area of interest, generation of free text explanations is not a well consolidated task. particularly, evaluations metrics are still being discussed @xcite ) @xcite , attempting both to capture the explanation informativeness and to improve the correlation toward human judgements. here we report related works which are most focused on the e-snli dataset, being more comparable with our work (see generating explanation with implicit knowledge has traditionally been addressed either by constraining generations with general knowledge paths @xcite , by fine-tuning on specific or general knowledge datasets @xcite , or with a combination of both methods @xcite . @xcite train four different models with the aim to generate an explanation given only the hypothesis, generate an explanation without knowing the label, jointly predict a label and generate an explanation for the predicted label, and generate an explanation and then predict the label. their work uses straightforward recurrent neural network architectures so it is does not achieve state-of-the-art results, but it establishes a strong baseline. @xcite generate implicit knowledge connecting sentences in text, similarly to @xcite . they perform fine-tuning on corpora enriched with implicit information, by constraining them with relevant concepts and connecting commonsense knowledge paths, combining data augmentation and graph-to-text methods. @xcite both present feb, a standardized collection of four existing englishlanguage datasets and associated metrics, and results based on template-based prompting. in our work we show that specific prompting design for the e-snli task results in significant improvements with respect to more general purposes prompts. @xcite , based on the intuition that explanation generated through single-pass prompting often lacks sufficiency and conciseness, propose a two-step approach where the first-pass output from the pretrained language model is polished, and then regenerated retaining the information that supports the contents being explained. @xcite show that both the computation trace (the way the explanation is decomposed) and the natural language of the prompt, contribute to the effectiveness of explanations. according to this finding they propose automatic prompt selection that focus on prompt diversity, rather than complementarity only. ## conclusion in this work, we argued that providing explanations is often a process of eliciting implicit knowledge. we proposed a general methodology to generate commonsense explanations from pairs of semantically related sentences, taking advantage of both prompting applied to large pre-trained language models and few-shot learning techniques. experiments run on the e-snli dataset show that the proposed methods achieve sota results on the explanation generation task, with a substantial reduction of labelled data.