Improving Abstractive Summarization with Commonsense Knowledge (2021)

Abstract: Large scale pretrained models have demonstrated strong performances on several natural language generation and understanding benchmarks. However, introducing commonsense into them to generate more realistic text remains a challenge. Inspired from previous work on commonsense knowledge generation and generative commonsense reasoning, we introduce two methods to add commonsense reasoning skills and knowledge into abstractive summarization models. Both methods beat the baseline on ROUGE scores, demonstrating the superiority of our models over the baseline. Human evaluation results suggest that summaries generated by our methods are more realistic and have fewer commonsensical errors.

## introduction commonsense knowledge is crucial for understanding natural language. several benchmarks have been developed to test commonsense reasoning skills in natural language processing systems. most of these benchmarks such as helaswag @xcite , winogrande @xcite and commonsenseqa @xcite formulate commonsense reasoning in the form of discriminative tasks. while significant progress has been made on such discriminative tasks, adding commonsense knowledge and reasoning skills into natural language generation still remains a challenge. recent work has addressed this challenge to some extent. tasks such as abductive commonsense generation @xcite test commonsense reasoning ability of large pretrained models and demonstrated that the performance on such tasks is far behind human performance. @xcite added commonsense knowledge into sequence to sequence models via external knowledge graphs. the recently proposed commongen @xcite benchmark also tests commonsense reasoning skills of models via language generation. inspired from recent work on commonsense knowledge generation tasks we propose two training techniques to add commonsense knowledge to abstractive summarization models. while conducting experiments on the baseline models we observe that the generated summaries lack commonsense reasoning in them, i.e the models are unable to capture basic commonsense knowledge while generating summaries. for example, the model generated the phrase 'researchers, known as the excess uric acid'. this phrase does not comply with our general commonsensical knowledge. the model is unable to derive the correct relation between a 'researcher' and 'uric acid'. our approaches try to rectify the above problem in abstractive summarization models. our first approach is similar to @xcite wherein we use external commonsense knowledge to supply additional reasoning power to the model. by doing so we are able to add basic commonsense knowledge about the concepts into our model. our second approach is a training-finetuning based approach wherein we finetune the model to generate summaries conditioned only on the concepts in the document and not the entire document. in this way we are able to relate different parts of the document since the model has to implicitly extract relations between these concepts to generate a summary. we assume that the concepts extracted from the document in some way represent every part of the document. both our approaches solve the aforementioned problem to some extend and also improve the performance of the summarization models in terms of evaluation metric such as rouge. we also try a few other techniques which we mention in later sections. ## related work abstractive summarization aims at capturing the entire essence of a document within a small number of words. there has been great progress made in the recent past where better and better architectures have been proposed to solve the problem. @xcite used pointer generator networks to copy words from the input document. @xcite augmented the transformer architecture with a contrastive attention mechanism to ignore the irrelevant parts of the document. @xcite pretrained a transformer model for summarization. our work unlike previous work tries to add commonsense knowledge into abstractive summarization models with minimum change to the training objective and model architecture. addition of commonsense knowledge and commonsense reasoning abilities into machine learning models has been a topic of interest in recent years. several benchmarks and tasks have been established to evaluate commonsense reasoning skills. these tasks are based on temporal commonsense reasoning @xcite , commonsense knowledge about physical entities @xcite , world knowledge based commonsense reasoning @xcite , abductive commonsense inference @xcite , human emotion @xcite , coreference resolution (sakaguchi et al., 2020), sentence completion @xcite etc. another set of commonsense reasoning oriented tasks are based on natural language generation. one such task is abductive commonsense generation @xcite wherein given a set of observations the model has to generate a hypothesis which explains the second observation based on the first observation. another commonsense language generation benchmark is language generation conditioned on concepts @xcite wherein the model has to generate a plausible natural language sentence given a set of concepts. recent work has also been focused on adding commonsense knowledge into sequence to sequence models. @xcite added external commonsense knowledge extracted from knowledge graphs into topic-to-essay generation models. they use conceptnet @xcite to extract concepts and relations. @xcite added commonsense knowledge to pretrained language models, their approach is inspired by the common-gen benchmark @xcite . for abstractive summarization amplayo et al. (2018) developed an entity linking system to extract linked entities knowledge graphs. these linked entities carry commonsense knowledge since they are linked to a knowledge graph and are used to generate a representation for the topic of the summary. this representation is used to guide the decoder. @xcite developed a dialogue heterogeneous graph network using utterances and commonsense knowledge for abstractive dialogue summarization. our approach draws ideas from these works and applies them to abstractive summarization. firstly we explore methods to add external commonsense knowledge into models. we use numberbatch embeddings pretrained on conceptnet to add commonsense into our model similar to @xcite . our next approach is based on generating summaries only from document concepts. the problem setting is similar to that of @xcite where the model needs to implicitly reason between the concepts to generate the summary. we also try a fusion of these approaches. lastly we explore pretraining-finetuning based approaches wherein we pretrain our model on other commonsense reasoning tasks and then use it train our summarization model. ## method in this section we present our two methods to add commonsense knowledge and reasoning skills into abstractive summarization models. all our models are based on transformers @xcite . let the document be represented by @xmath0, x 2 , ..., x n ) where n is the length of the document and let the summary be represented by @xmath1, y 2 , ..., y m ) where m is the length of the summary. let the transformer model encoder be represented by enc and let the transformer decoder be represented dec, at the t th time step the model generates the t th word of the summary as: ## experiments we use the cnn/daily mail dataset for all our experiments. all our models are transformer based, we use the transformer base model with 6 encoder layers, 6 decoder layers, 8 attention heads, 512 is the hidden and embedding dimension and 2048 dimensional feed forward layer. we use the linear warm-up followed by square root decay schedule proposed by @xcite . we train our baseline for 200k iterations with a batch size of 32 and we employ early stopping based on validation loss. we use beam search decoding in all our experiments with a beam size of 4, we also employ a length penalty of 1.0. we use label smoothing with α = 0.1. while generating the summaries we do not let the model repeat trigrams @xcite . we use sentencepiece tokenizer in all our experiments. all the experiments are performed using pytorch on a single nvidia tesla v100 gpu. ## conclusion in this work we proposed two techniques to add commonsense knowledge to abstractive summarization models. we observe that the baseline is sometimes unable to generate summaries that comply with the general notion of commonsense. both our approaches solve this issue to a certain extend. our first method is based on adding external commonsense knowledge into the model, and our second method is based on training the model with only concepts present in the document rather than using the entire document. we show that our models outperform the baseline on the evaluation metric. we also experimented with several variants of our proposed approaches. our human evaluation results suggest that the summaries generated by our models are indeed more realistic. the application of our methods on state-of-the-art summarization systems should further improve the results. one limitation to be addressed is to formalize the notion of 'commonsensical soundness' and come up with an automatic evaluation metric to measure it. in future we would work on this limitation. we would also experiment with other techniques and devise better methods for adding commonsense knowledge into abstractive summarization models and extend our methods to other natural language generation tasks such as machine translation.