  Large-scale pre-trained language models have demonstrated strong knowledge
representation ability. However, recent studies suggest that even though these
giant models contains rich simple commonsense knowledge (e.g., bird can fly and
fish can swim.), they often struggle with the complex commonsense knowledge
that involves multiple eventualities (verb-centric phrases, e.g., identifying
the relationship between ``Jim yells at Bob'' and ``Bob is upset'').To address
this problem, in this paper, we propose to help pre-trained language models
better incorporate complex commonsense knowledge. Different from existing
fine-tuning approaches, we do not focus on a specific task and propose a
general language model named CoCoLM. Through the careful training over a
large-scale eventuality knowledge graphs ASER, we successfully teach
pre-trained language models (i.e., BERT and RoBERTa) rich complex commonsense
knowledge among eventualities. Experiments on multiple downstream commonsense
tasks that requires the correct understanding of eventualities demonstrate the
effectiveness of CoCoLM.
