  Existing pre-trained language models (PLMs) have demonstrated the
effectiveness of self-supervised learning for a broad range of natural language
processing (NLP) tasks. However, most of them are not explicitly aware of
domain-specific knowledge, which is essential for downstream tasks in many
domains, such as tasks in e-commerce scenarios. In this paper, we propose
K-PLUG, a knowledge-injected pre-trained language model based on the
encoder-decoder transformer that can be transferred to both natural language
understanding and generation tasks. We verify our method in a diverse range of
e-commerce scenarios that require domain-specific knowledge. Specifically, we
propose five knowledge-aware self-supervised pre-training objectives to
formulate the learning of domain-specific knowledge, including e-commerce
domain-specific knowledge-bases, aspects of product entities, categories of
product entities, and unique selling propositions of product entities. K-PLUG
achieves new state-of-the-art results on a suite of domain-specific NLP tasks,
including product knowledge base completion, abstractive product summarization,
and multi-turn dialogue, significantly outperforms baselines across the board,
which demonstrates that the proposed method effectively learns a diverse set of
domain-specific knowledge for both language understanding and generation tasks.
