Reconstructing Implicit Knowledge with Language Models (2021)

Abstract: In this work we propose an approach for generating statements that explicate implicit knowledge connecting sentences in text. We make use of pre-trained language models which we refine by fine-tuning them on specifically prepared corpora that we enriched with implicit information, and by constraining them with relevant concepts and connecting commonsense knowledge paths. Manual and automatic evaluation of the generations shows that by refining language models as proposed, we can generate coherent and grammatically sound sentences that explicate implicit knowledge which connects sentence pairs in texts – on both in-domain and out-of-domain test data.

## introduction in everyday communication and in texts people usually omit information that seems clear and evident, such that only part of the message needs to be expressed in words. in the following sentence: in order to understand the connection between (i) and (ii) we must know that computers are used for sending emails, or that lectures are given by teachers. such implicit knowledge can easily be inferred by humans, since it is part of their background knowledge. by contrast, for computational systems implicitness in texts represents a challenge. in this work we propose an approach for generating implicit knowledge sentences in-between contiguous sentences, which explicate their logical connection, utilizing pre-trained language models (lms) that we refine as follows: i) we inject 'explanatory' knowledge by fine-tuning lms on specifically prepared corpora, and (ii) condition text generation through constraints in form of relevant concepts and knowledge paths. our work is inspired by the recent success of pre-trained lms @xcite @xcite in various downstream nlp tasks, including text generation and nl inference @xcite . however, for the task of reconstructing implicit knowledge, such lms need to be carefully guided, not only to yield coherent statements, but to also ensure that they convey the missing, implicit information that connects given sentences in a text. to this end we create corpora with sentence pairs enriched with implicit information based on on generics-kb @xcite ) and e-snli @xcite , which we use for lm fine-tuning. for improved performance we explore methods of constrained language generation, guiding the model by way of relevant concepts and connecting commonsense knowledge paths. we aim to build a system that is not limited to specific text genres or knowledge domains, and thus evaluate our models in-domain -on testsets from our fine-tuning corpora; and out-of-domainusing ikat @xcite , an argumentative corpus which offers sentence pairs annotated with implicit knowledge that connects them. a central contribution of this work is an indepth evaluation of the quality of generations delivered by different model variants, and their ability of expressing implicitly conveyed knowledge. we propose a manual evaluation setup covering four dimensions -grammaticality, coherence, content, and comparison to gold references -, and compare these to various automatic evaluation metrics. our experiments show that with our proposed approach we can generate coherent sentences that explicate implicit knowledge that connects given sentence pairs; and that current text generation metrics are not sufficient to evaluate this challenging task. our contributions are: (i) we empirically compare different types of lms, exploring which model is best suited for the task of generating sentences that express implicit information between sen-tences. (ii) we create datasets that include implicit information holding between sentence pairs, which we use for fine-tuning our lms, and which can be used for general commonsense reasoning tasks. (iii) we propose a method for constrained generation by injecting concepts or commonsense knowledge paths as language modeling constraints, and show that key concepts, and even more, knowledge paths improve the quality of generations. (iv) we carefully evaluate the quality of the generated implicit knowledge sentences, both manually and automatically, and discuss strengths and limitations of automatic similarity metrics. foot_0 ## related work recent progress in pretraining lms on large text corpora led to improvements for various downstream nlp tasks. it has also been shown that knowledge acquired during pre-training can be leveraged by fine-tuning these models to advanced semantic inference or nl generation tasks @xcite . recently, pre-trained lms have been augmented with external knowledge from commonsense knowledge bases such as conceptnet, which provides more explicit knowledge grounding and improves their performance on downstream tasks that require reasoning abilities. @xcite , for example, retrieve multi-hop knowledge paths from conceptnet for fine-tuning lms for multiple choice question answering. chang et al. (2020) and @xcite incorporate knowledge paths from conceptnet into pre-trained lms for solving the socialiqa task @xcite . however, all these approaches evaluate the effectiveness of integrating commonsense knowledge indirectly on downstream tasks, and do not explicitly evaluate the impact and relevance of knowledge for a specific system prediction. we address this shortcoming by generating and carefully evaluating statements that connect pairs of sentences as explanations of their underlying, implicit knowledge link. closest to this aim is the task of explanation generation, which has received attention very recently. @xcite propose the semeval-2020 task 4 (subtask c), which is to generate an explanation for why a statement does not make sense, by way of a natural language statement. a comparison of the participating systems (cf. peru-mal et al. @xcite shows that pre-trained lms play a central role in the success of the topperforming systems, demonstrating that they contain commonsense information to a good extent. the success of models enriched with knowledge from external sources such as conceptnet furthermore shows that additional knowledge supports the generation of commonsense explanations. however, there is still a large gap between systems and human performance. pre-trained lms enhanced with commonsense knowledge have also been the models of choice for other text generation tasks, e.g. dialogue generation @xcite , story ending generation @xcite , or abductive nli @xcite . while these models aim at generating explanations for a single statement, or completing a given sequence of sentences, we investigate how to make use of lms to generate a sentence that fills in implicit knowledge between two sentences. constraining lms. recent work addresses how to control content in lm text generation, while maintaining fluency, coherence and plausibility of the generated text. @xcite explore how to generate a coherent and plausible situation description given an unordered set of concepts as input, and find that even pre-trained lms (bart, t5) fine-tuned to this task cannot solve it: the generated sentences are grammatical, but highly implausible, lacking commonsense. this suggests that either the underlying lms, or input constraints for generation need to incorporate commonsense knowledge. @xcite attempt to control the content when generating longer stories by specifying facts the story needs to include. they propose a plan-and-cloze model that first creates a cloze template, placing input facts at fixed positions in the output. in the cloze step, the system expands the fact tokens into complex sentences that complete the story. while uni-directional lms such as gpt-2 or bart generate fluent text but do not well adhere to the desired content, the fine-tuned multi-directional xlnet outputs coherent text and adheres to the facts. while none of the above works incorporate external knowledge to guide generation, @xcite perform explanation generation for single statements, using conceptnet background knowledge. the model selects concepts from the statement, retrieves connecting paths from conceptnet, and selects bridge concepts from a subgraph. a pre-trained decoder generates the explanation, using as input the statement and top-ranked concepts from the subgraph. in our work we also select concepts from texts, but dynamically generate commonsense knowledge paths as constraints. importantly, we aim to generate coherent explanations in-between sentences -a challenge for uni-directional lms. ## knowledge-constrained text generation 3.1 task definition and approach the task we tackle in this work is: given two contiguous sentences (source sentences s 1 , s 2 ), generate an explanatory sentence (target sentence t ) that explains the underlying, implicit information that connects them. we explore different types of lms and their aptness for solving this task. we fine-tune them on existing or adapted datasets to inject relevant knowledge, and add key concepts or connecting knowledge-paths as constraints to achieve coherent and informative explanations. ## data and experimental setup datasets. we use the data from genericskb and e-slni for fine-tuning and testing models (in-2 details about the models appear in the appendix. input sequences. we build the input sequences by concatenating the source sentences s 1 and s 2 , separated by a sep token. when including key concepts c 1,2 or knowledge paths p as constraints, we append them to the input sequence right after s 1 and s 2 , separated by a sep token. thus, the concepts and paths we use as constraints are encoded by the tokenizer of each language model together with the rest of the input sequence. accordingly, our input sequences are structured as follows: fine-tuning lms. for lm fine-tuning, we append the target sentence to the input sequence, separated from the rest of the input by an eot tag. gpt-2 and xlnet are trained to reconstruct the target sentence t . during inference, the models only see the source sentences, and constraints if given, and they complete the input sequence by generating t . in contrast, bart encodes s 1 and s 2 , and its decoder is trained to predict t based on the encoded source sentences. we use the pre-trained models from hugging-face transformers @xcite and adapt them for fine-tuning on our customized training data. in order to generate compact sentences capturing the relevant implicit knowledge (instead of long explanations), we set a length limitation of 20 tokens for each generation. more details about our models are listed in the appendix. ## evaluation and results this section presents an in-depth evaluation of the quality of generations from different model variants, and their ability of expressing implicitly conveyed knowledge. we design a manual evaluation setup covering various dimensions, and compare the results to several automatic evaluation metrics. we conduct evaluation in-domain on our customized test data; and out-of-domain on ikat. ## discussion limitations of automatic evaluation metrics for text generations. concluding, we pinpoint two important limitations of automatic text generations metrics -especially reference-based ones: besides well-known issues regarding the reliability, interpretability and biases of such metrics @xcite , scores are mostly obtained by comparing generations against a single reference, which is -here, as in other generation tasks -often only one among several valid options. for the task of reconstructing implicit information, @xcite show that annotators often propose different valid sentences for filling knowledge gaps in argumentative texts. for our setting this means that a generated sentence may be a relevant explicitation of implicit information, even if not similar to the reference. such cases are poorly or not at all captured by automatic similarity metrics. an exception we found is sentencebert, which is based on sentence representations, and which aligned reasonably well with insights from our manual evaluation. still, automatic evaluation of text generations needs to be considered with caution, and should always be accompanied by manual evaluation. our implicitness assumption. our experiments are based on the underlying assumption that usually some information between pairs of sentences stays implicit, which has been confirmed empirically for our datasets: our annotators stated for 100% (first round) and 98% (second round) of all sentence pairs that they are implicitly connected by some unexpressed piece of knowledge. however, we did not specifically address the cases of sentence pairs between which no implicit information is missing (even though these cases are rare), nor did we investigate how our models would perform when provided with sentence pairs that are not related (arbitrary pairs). for a real-world application, both aspects would be considerable. ## conclusion in this work we propose an approach for generating statements that explicate implicit knowledge connecting sentences in text, using pre-trained lms. we show that despite their great success in many nlp downstream tasks, lms need to be well equipped and carefully guided for the challenging task of reconstructing implicit knowledge, to ensure that they convey the missing, implicit information that connects sentences in text. we refine different pre-trained lms by fine-tuning on specifically prepared corpora that we enrich with implicit information, filled in between sentences, and explore methods of constrained language generation, guiding the models by way of relevant concepts and connecting commonsense knowledge paths. while most current automatic nlg metrics are not sufficient to evaluate this challenging task, our in-depth evaluation of the quality of generations from different model variants shows that the bart model, which attends over its full input when generating text, yields most informative and relevant explanations. we also establish that e-snli, being focused on the nli task, is best suited for conditioning lms for our task, especially for out-of domain settings. finally, by providing the lms with relevant connecting key concepts as constraints, and further by connecting commonsense knowledge paths, we achieve generation of coherent and grammatically sound sentences that -according to manual evaluation -can explicate the implicit knowledge that connects sentence pairs in textsfor in-domain and out-of-domain test data. genericskb datasets, respectively, while bart requires 8 hours, and xlnet around 20 hours (due to its permutation procedure) for the same data. limiting length of generations. in order to generate compact sentences capturing the relevant implicit knowledge (instead of long explanations), we set a length limitation of 20 tokens for each generation. in the left-to-right decoding procedure of gpt-2 and bart, the generation can be stopped earlier than 20 tokens, when the model predicts an eot token. thus, both gpt-2 and bart models can predict complete sentences of up to 20 tokens due to the autoregressive decoder. in contrast, xl-net has a permutation language modeling mechanism and predicts the next tokens based on the previous and next tokens. its generations usually don't contain a significant eot token. predicted target sequence of tokens in a post-processing step by cutting it after a generated comma (,).