C o C o LM : Complex Commonsense Enhanced Language Model with Discourse Relations (2022)

Abstract: Large-scale pre-trained language models have demonstrated strong knowledge representation ability. However, recent studies suggest that even though these giant models contain rich simple commonsense knowledge (e.g., bird can fly and fish can swim.), they often struggle with complex commonsense knowledge that involves multiple eventualities (verb-centric phrases, e.g., identifying the relationship between “Jim yells at Bob” and “Bob is upset”). To address this issue, in this paper, we propose to help pre-trained language models better incorporate complex commonsense knowledge. Unlike direct fine-tuning approaches, we do not focus on a specific task and instead propose a general language model named CoCoLM. Through the careful training over a large-scale eventuality knowledge graph ASER, we successfully teach pre-trained language models (i.e., BERT and RoBERTa) rich multi-hop commonsense knowledge among eventualities. Experiments on multiple commonsense tasks that require the correct understanding of eventualities demonstrate the effectiveness of CoCoLM.

## introduction recently, large-scale pre-trained language representation models (lms) @xcite have demonstrated the strong ability to discover useful linguistic properties of syntax and remember an impressive amount of knowledge with self-supervised training over a large unlabeled corpus @xcite . on top of that, with the help of the fine-tuning step, lms can learn how to use the memorized knowledge for different tasks, and thus achieve outstanding performance on many downstream natural language processing (nlp) tasks. ## methods the overall framework of cocolm is presented in figure 1 . given a pre-trained language model, we inject complex commonsense knowledge about eventualities by adding one adaptive pre-training stage @xcite . specifically, we first generate eventuality sequences based on carefully controlled walks over existing eventuality knowledge graphs and then use the sequences as the context to help lms handle eventualities. besides the original token-level mlm objective, we also introduce the eventuality-level masking strategy and several auxiliary tasks to assist the training. as the training is not task-specific, the resulting lm can be easily applied to any downstream tasks via another task-specific fine-tuning stage. ## experimental results the results are presented in besides that, we can make the following observations. first, the improvement of our model is more significant on rocstories than copa and matres, which is mainly because multiple relation combinations in the eventuality sequences bring high-order information and thus help complex reasoning. second, cocolm achieves significant improvement on lower-capacity lms trained on small corpora. for example, cocolm brings up to 59.2% improvement over bert-base on the rocstories dataset. third, compared with the original supervised setting, the debiased setting is more challenging for all models, which helps verify our assumptions that previous models might benefit from the bias. here the debiased setting should be more fair for comparison. when we dig into the matres dataset, event pairs (typically verb pairs) are associated with the context, which some of could be easily inferred from the local context with obvious clues @xcite while the others may need external commonsense knowledge that can be memorized by the language models. as a comparison, both the rocstories and copa do not have any extra context, and thus require the pre-trained lms to know the essential knowledge to solve those problems. in the rest of this section, we conduct extensive experiments and case studies to demonstrate the contribution of different components. in all following analysis experiments, we use bert-large as the base language model and rocstories as the evaluation dataset. ## conclusion and future work in this work, we aim at helping pre-trained language models understand complex commonsense about eventualities. specifically, we first conduct the random walk over a large-scale eventualitybased knowledge graph to collect multi-hop event knowledge and then inject the knowledge into the pre-trained lms with an eventuality-based mask strategy as well as two auxiliary tasks. experiments on three downstream tasks as well as extensive analysis demonstrate the effectiveness of the proposed model. as our approach is a general solution, we believe that it can also be helpful for other tasks that require complex commonsense about events. for future work, we would sample sub-graph structures to explore more meaningful event-centric commonsense knowledge @xcite . moreover, we will equip our models with generative abilities by finetuning powerful t5 @xcite or bart @xcite models to help narrative story completion @xcite , commonsense inference @xcite , event infilling tasks @xcite . unified event-aware language models like @xcite would be promising and interesting directions. edge in the atomic. thus aser might roughly cover the knowledge in the atomic. moreover, aser also covers agentless events such as "the weather is good", which was partially covered by glucose @xcite however it contains noise compared with atomic. cocolmexperiments show atomic (877k edges) performs better than aser(4.4 m -5×larger). if-then types relations causal types if-event-then-state xintent cause xreact effect oreact effect if-event-then-event xneed cause xeffect effect xwant effect oeffect effect owant effect if-event-then-persona xattr stative