K- PLUG : Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E -Commerce (2021)

Abstract: Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledge-injected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. Specifically, we propose five knowledge-aware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue. K-PLUG significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks. Our code is available.

## introduction pre-trained language models (plms), such as elmo @xcite , gpt @xcite , bert @xcite , roberta @xcite , and xlnet @xcite , have made remarkable breakthroughs in many natural * equal contribution. language understanding (nlu) tasks, including text classification, reading comprehension, and natural language inference. these models are trained on large-scale text corpora with self-supervision based on either bi-directional or auto-regressive pre-training. equally promising performances have been achieved in natural language generation (nlg) tasks, such as machine translation and text summarization, by mass @xcite , unilm @xcite , bart @xcite , t5 @xcite , pegasus @xcite , and prophetnet @xcite . in contrast, these approaches adopt transformerbased sequence-to-sequence models to jointly pretrain for both the encoder and the decoder. while these plms can learn rich semantic patterns from raw text data and thereby enhance downstream nlp applications, many of them do not explicitly model domain-specific knowledge. as a result, they may not be as sufficient for capturing human-curated or domain-specific knowledge that is necessary for tasks in a certain domain, such as tasks in e-commerce scenarios. in order to overcome this limitation, several recent studies have proposed to enrich plms with explicit knowledge, including knowledge base (kb) @xcite @xcite @xcite , lexical relation @xcite , word sense @xcite , part-of-speech tag @xcite , and sentiment polarity @xcite . however, these methods only integrate knowledge into the encoder, and the decoding process in many nlg tasks benefits little from these knowledge. to mitigate this problem, we propose a knowledge-injected pre-trained language model that is suitable for both natural language understanding and generation (k-plug). different from existing knowledge-injected plms, k-plug integrates knowledge into pre-training for both the encoder and the decoder, and thus k-plug can be adopted to both downstream knowledge-driven nlu and nlg tasks. we verify the performance of the proposed method in various e-commerce scenarios. in the proposed k-plug, we formulate the learning of four types of domainspecific knowledge: e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions (usps) @xcite of product entities. specifically, e-commerce kb stores standardized product attribute information, product aspects are features that play a crucial role in understanding product information, product categories are the backbones for constructing taxonomies for organization, and usps are the essence of what differentiates a product from its competitors. k-plug learns these types of knowledge into a unified plm, enhancing performances for various language understanding and generation tasks. to effectively learn these four types of valuable domain-specific knowledge in k-plug, we proposed five new pre-training objectives: knowledge-aware masked language model (kmlm), knowledge-aware masked sequenceto-sequence (kms2s), product entity aspect boundary detection (peabd), product entity category classification (pecc), and product entity aspect summary generation (peasg). among these objectives, kmlm and kms2s learn to predict the masked single and multiple tokens, respectively, that are associated with domain-specific knowledge rather than general information; peabd determines the boundaries between descriptions of different product aspects given full product information; pecc identifies the product category that each product belongs to; and peasg generates a summary for each individual product aspect based on the entire product description. after pre-training k-plug, we fine-tune it on three domain-specific nlp tasks, namely, ecommerce knowledge base completion, abstractive product summarization, and multi-turn dialogue. the results show that k-plug significantly outperforms comparative models on all these tasks. our main contributions are as follows: ## knowledge-injected pre-training in this section, we explain the data used to pretrain k-plug, its model architecture, and our pretraining objectives. ## conclusion ground-truth 采用高纯度无氧铜水箱，增大了加热面积，高效速热且温度持久稳定。 ::::::::::::: 封闭式燃烧技术， 提升了热效率，节能又高效。拥有控温功能，搭配 :::::::::: led数字大屏，让你能够根据需求来 随意调节水温。 (high concentration oxygen-free tank with added heating area boils water quickly with stability. input 劲霸男士t恤，夏季新品，撞色时尚，舒适，花卉印花，短袖，黑色，商务休闲，圆领， description 夏季，时尚都市，短袖t恤，悬垂感和耐穿性好，时尚圆领，简约大气，多色可选，莫代尔 棉针织面料，柔滑触感，清凉舒爽，花卉印花结合，珠片绣花，时尚大方... (k-boxing's men's t-shirt. this summer's new arrival. contrasting colors. soft and comfortable. flower print. short-sleeve. black. business casual. crewneck. summer. urban style. short-sleeve t-shirt. naturally draping with high durability. stylish crewneck. basic and causal. multiple color chioces. modal knit fabric. soft and elastic. loose fit. flower print. glitter embroidery. stylish and casual ....) reference 选用莫代尔棉针织面料， ::::::: 手感柔滑，上身 ::::::: incredibly soft and elastic. features a :::::::::::: slightly looser fit. natural draping with high durability. decorated with flower print and ::::::::::::: incredibly soft and stretchy. ::::::::::::: slightly looser fit. flower print and water-proof and durable. specially made smoothly meshing zippers. long shoulder strap with :::::::::::: adjustable length, made with leather for tas-approved locks protect your items and cause no damage during security check.)