K-PLUG: K NOWLEDGE - INJECTED P RE - TRAINED L ANGUAGE M ODEL FOR N ATURAL L ANGUAGE U NDERSTANDING AND G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks.
However, most of them are not explicitly aware of domain-speciﬁc knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios.
In this paper, we propose K-PLUG, a knowledgeinjected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks.
We verify our method in a diverse range of e-commerce scenarios that require domain-speciﬁc knowledge.
Speciﬁcally, we propose ﬁve knowledgeaware self-supervised pre-training objectives to formulate the learning of domainspeciﬁc knowledge, including e-commerce domain-speciﬁc knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities.
K-PLUG achieves new state-of-the-art results on a suite of domain-speciﬁc NLP tasks, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue, signiﬁcantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-speciﬁc knowledge for both language understanding and generation tasks.
The code, data, and models will be publicly available 1 .
I NTRODUCTION
Pre-trained language models (PLMs), such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019), have made remarkable breakthroughs in many natural language understanding (NLU) tasks, including text classiﬁcation, reading comprehension, and natural language inference.
These models are trained on large-scale text corpora with self-supervision based on either bi-directional or auto-regressive pre-training.
Equally promising performances have been achieved in natural language generation (NLG) tasks, such as machine translation and text summarization, by MASS (Song et al., 2019),
UniLM (Dong et al., 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2019), PEGASUS (Zhang et al., 2020), and ProphetNet (Yan et al., 2020).
In contrast, these approaches adopt Transformerbased sequence-to-sequence models to jointly pre-train for both the encoder and the decoder.
While these PLMs can learn rich semantic patterns from raw text data and thereby enhance downstream NLP applications, many of them do not explicitly model domain-speciﬁc knowledge.
As a result, they may not be as sufﬁcient for capturing human-curated or domain-speciﬁc knowledge that is necessary for tasks in a certain domain, such as tasks in e-commerce scenarios.
In order to overcome this limitation, several recent studies have proposed to enrich PLMs with explicit knowledge, including knowledge base (KB) (Zhang et al., 2019; Peters et al., 2019;
Xiong et al., 2020; Wang et al., 2019; 2020), lexical relation (Lauscher et al., 2019; Wang et al., 2020), word sense (Levine et al., 2020), part-of-speech tag (Ke et al., 2019), and sentiment polarity (Ke et al., 2019; Tian et al., 2020).
However, these methods only integrate domain-speciﬁc knowledge into the encoder, and the decoding process in many NLG tasks beneﬁts little from these knowledge.
To mitigate this problem, we propose a K nowledge-injected P re-trained L anguage model that is suitable for both Natural Language U nderstanding and G eneration (K-PLUG).
Different from existing knowledge-injected PLMs, K-PLUG integrates knowledge into pre-training for both the encoder and the decoder, and thus K-PLUG can be adopted to both downstream knowledge-driven NLU and
NLG tasks.
We verify the performance of the proposed method in various e-commerce scenarios.
In the proposed K-PLUG, we formulate the learning of four types of domain-speciﬁc knowledge: e-commerce domain-speciﬁc knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions (USPs) (Garrett, 1961) of product entities.
Speciﬁcally, e-commerce KB stores standardized product attribute information, product aspects are features that play a crucial role in understanding product information, product categories are the backbones for constructing taxonomies for organization, and USPs are the essence of what differentiates a product from its competitors.
K-PLUG learns these types of knowledge into a uniﬁed PLM, enhancing performances for various language understanding and generation tasks.
To effectively learn these four types of valuable domain-speciﬁc knowledge in K-PLUG, we proposed ﬁve new pre-training objectives: knowledge-aware masked language model (KMLM), knowledge-aware masked sequence-to-sequence (KMS2S), product entity aspect boundary detection (PEABD), product entity category classiﬁcation (PECC), and product entity aspect summary generation (PEASG).
Among these objectives, KMLM and KMS2S learn to predict the masked single and multiple tokens, respectively, that are associated with domain-speciﬁc knowledge rather than general information; PEABD determines the boundaries between descriptions of different product aspects given full product information; PECC identiﬁes the product category that each product belongs to; and PEASG generates a summary for each individual product aspect based on the entire product description.
After pre-training K-PLUG, we ﬁne-tune it on three domain-speciﬁc NLP tasks, namely, ecommerce knowledge base completion, abstractive product summarization, and multi-turn dialogue.
The results show that K-PLUG signiﬁcantly outperforms comparative models on all these tasks.
Our main contributions can be summarized as follows:
• We present K-PLUG that learns domain-speciﬁc knowledge for both the encoder and the decoder in a pre-training language model framework, which beneﬁts both NLG and NLU tasks.
• We formulate the learning of four types of domain-speciﬁc knowledge in e-commerce scenarios: e-commerce domain-speciﬁc knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities, which provide critical information for many applications in the domain of e-commerce.
Speciﬁcally, ﬁve self-supervised objectives are proposed to learn these four types of knowledge into a uniﬁed PLM.
• Our proposed model exhibits clear effectiveness in many downstream tasks in the ecommerce scenario, including e-commerce KB completion, abstractive product summarization, and multi-turn dialogue.
R ELATED W ORK 2. PLM S IN G ENERAL
Unsupervised pre-training language model has been successfully applied to many NLP tasks.
ELMo (Peters et al., 2018) learns the contextual representations based on a bidirectional LM.
GPT (Radford et al., 2018) predicts tokens based on the context on the left-hand side.
BERT (Devlin et al., 2019) adopts a bi-directional LM to predict the masked tokens.
XLNet (Yang et al., 2019) predicts masked tokens in a permuted order through an autoregressive method.
MASS (Song et al., 2019) pre-trains the sequence-to-sequence LM to recover a span of masked tokens.
UniLM (Dong et al., 2019) combines bidirectional, unidirectional, and sequence-to-sequence LMs.
T5 (Raffel et al., 2019) and BART (Lewis et al., 2020) present denoising sequence-to-sequence pre-training.
PEGASUS (Zhang et al., 2020) pre-trains with gap-sentence generation objective.
While humancurated or domain-speciﬁc knowledge is essential for downstream knowledge-driven tasks, these methods do not explicitly consider external knowledge like our proposed K-PLUG. Encoder [ C L S ] w a t e r … [ Refrigerator ] l o ss c o o l i n g f a s t
Encoder [ C L S ] w a t e r l o ss r e d u c e c o o l i n g f
{+a s t 0 1 0 
 Encoder [ C L S ] w a t e r l+} o ss c
o o l i n g r e d u c e f a s t [ M A S K ]
[Figure] Encoder Decoder d r a w e r c r i s p
e r w i t h w i n d o w v e n t
[ B O S ]
[ E O S ]
d r a w e r c r
i s p e r w i t h w i n d o w v e n t [ C L S
w a t e r r e d u c e l o
[Figure]
c o o l i n g f
a s t
[Figure]
Decoder l o
[Figure] a t e [Figure]
c [Figure]
unique selling propositions of product entities .
Pre-training objectives include knowledge-aware masked language model (KMLM), knowledge-aware masked sequence-to-sequence (KMS2S), product entity aspect boundary detection (PEABD), product entity category classiﬁcation (PECC), and product entity aspect summary generation (PEASG).
2. I NJECTING K NOWLEDGE INTO PLM S
Recent work investigates how to incorporate knowledge into PLMs for NLU.
ERNIE (Sun et al., 2019) enhances language representation with the entity/phrase-level masking.
ERNIE (Zhang et al., 2019) identiﬁes and links entity mentions in texts to their corresponding entities in KB.
Similar to ERNIE (Zhang et al., 2019), KnowBERT (Peters et al., 2019) injects KBs into PLM.
Xiong et al.
(2020) leverages an entity replacement pre-training objective to learn better representations for entities.
KEPLER (Wang et al., 2019) adopts the knowledge embedding objective in the pretraining.
Besides, SKEP (Tian et al., 2020), SenseBERT (Levine et al., 2020), SentiLR (Ke et al., 2019), and K-ADAPTER (Wang et al., 2020) propose to integrate sentiment knowledge, word sense, sentiment polarity, and lexical relation into PLM, respectively.
However, most of these studies are focused on integrating knowledge for language understanding task, work of utilizing domainspeciﬁc knowledge for pre-training for language generation tasks are limited.
Inspired by these work, we construct K-PLUG that learns domain-speciﬁc knowledge into a PLM for both NLU and
NLG tasks.
K NOWLEDGE - INJECTED P RE - TRAINING
In this section, we explain the data used to pre-train K-PLUG, its model architecture, and our pretraining objectives.
3. D ATA P REPARATION
We collect the pre-training data from a mainstream Chinese e-commerce platform 2 , which contains approximately 25 million textual product descriptions and covers 40 product categories.
With an average length of 405 tokens, these product descriptions constitute a corpus with a size of 10B Chinese characters.
Each product description consists of information on 10.7 product aspects on average, and each product aspect is accompanied with a summary highlighting its prominent features, as shown in Figure 1(a).
Additionally, the e-commerce KB and USPs (further explained below) used in our pre-training data are as speciﬁed by the e-commerce platform and its online stores.
3. M ODEL A RCHITECTURE
K-PLUG adopts the standard sequence-to-sequence Transformer architecture (Vaswani et al., 2017), consisting of a 6-layer encoder and a 6-layer decoder as Song et al.
We set the size of hidden vectors as 768, and the number of self-attention heads as 12.
More details about the experimental settings are in the appendix.
3. K NOWLEDGE F ORMULATION AND P RE - TRAINING O BJECTIVES
We formulate the learning of four types of knowledge in a uniﬁed PLM: e-commerce KB, aspects of product entities, categories of product entities, and USPs of product entities.
Speciﬁcally, ecommerce KB stores standardized product attribute information, e.g. , ( Material : Cotton ) and ( Collar Type : Pointed Collar ).
It provides details about the products (Logan IV et al., 2017).
Aspects of product entities are features of a product, such as the sound quality of a stereo speaker, etc.
(Li et al., 2020).
Categories of product entities such as Clothing and Food are widely used by e-commerce platforms to organize their products so to present structured offerings to their customers (Luo et al., 2020;
Dong et al., 2020) USPs of product entities are the essence of what differentiates a product from its competitors (Garrett, 1961).
For example, a stereo speaker’s USP exhibiting its supreme sound quality could be “ crystal clear stereo sound ”.
An effective USP immediately motivates the purchasing behavior of potential buyers.
We propose and evaluate ﬁve novel self-supervised pre-training objectives to learn the abovementioned four types of knowledge in the K-PLUG model (see Figure 1).
Knowledge-aware Masked Language Model (KMLM)
Inspired by BERT (Devlin et al., 2019), we adopt the masked language model (MLM) to train the Transformer encoder as one of our pre-training objectives, which learns to predict the masked tokens in the source sequence ( e.g. , “The company is [MASK] at the foot of a hill.”).
Similar to BERT, we mask 15% of all tokens in a text sequence; 80% of the masked tokens are replaced with the [MASK] token, 10% with a random token, and 10% left unchanged.
Particularly, given an original text sequence x = ( x 1 , ..., x m , ..., x M ) with M tokens, a masked sequence is produced by masking x m through one of the three ways explained above, e.g. , replacing x m with [MASK] to create (cid:101) x
= ( x 1 , ..., [MASK] , ..., x M ) .
MLM aims to model the conditional likelihood P ( x m | (cid:101) x ) , and the loss function is:
[Equation] [Equation]
The major difference from BERT is that our KMLM prioritizes knowledge tokens, which contain knowledge regarding product attributes and USPs, when selecting positions to mask and, in the case that the knowledge tokens make up less than 15% of all tokens, randomly picks non-knowledge tokens to complete the masking.
Knowledge-aware Masked Sequence-to-Sequence (KMS2S)
K-PLUG inherits the strong ability of language generation from the masked sequence-to-sequence (MS2S) objective.
The encoder takes a sentence with a masked fragment (several consecutive tokens) as the input, and the decoder predicts this masked fragment conditioned on the encoder representations ( e.g. , “The company [MASK] [MASK] [MASK] the foot of a hill.”).
Given a text sequence x = ( x 1 , ..., x u , ..., x v , ..., x M ) , a masked sequence (cid:101) x
= ( x 1 , ..., [MASK] , ..., [MASK] , ..., x M ) is produced by replacing the span x u : v , ranging from x u to x v , with the [MASK] token.
MS2S aims to model P ( x u : v | (cid:101) x ) , which can be further factorized into a product P ( x u : v | (cid:101) x ) = [Equation] P
[Equation] | [Equation] x ) according to the chain rule.
The loss function is:
[Equation]
We set the length of the masked span as 30% of the length of the original text sequence.
Similar to
KMLM, KMS2S prioritizes the masking of text spans that cover knowledge tokens.
Product Entity Aspect Boundary Detection (PEABD)
A product description usually contains multiple product entity aspects.
Existing work (Li et al., 2020) proves that product aspects inﬂuence the quality of product summaries from the views of importance, non-redundancy, and readability, which are not directly taken into account in language modeling.
In order to train a model that understands product aspects, we leverage the PEABD objective to detect boundaries between the product entity aspects.
It is essentially a sequence labeling task based on the representations of K-PLUG’s top encoder layer.
Given a text sequence x = ( x 1 , ..., x M ) , the encoder of K-PLUG outputs a sequence h
= ( h 1 , ..., h M ) , which is fed into a softmax layer, and generates a probability sequence y .
The loss function is:
[Equation]
where y ∈ { [0 , 1] } are the ground-truth labels for the aspect boundary detection task.
Product Entity Category Classiﬁcation (PECC)
Product entity categories are the backbones for constructing taxonomies (Luo et al., 2020; Dong et al., 2020).
Each product description document corresponds to one of the 40 categories included in our corpus, such as Clothing , Bags , Home Appliances , Shoes , Foods , etc.
Identifying the product entity categories accurately is the prerequisite for creating an output that is consistent with the input.
Given a text sequence x = ( x 1 , ..., x M ) , a softmax layer outputs the classiﬁcation score, y , based on the representation of the encoder classiﬁcation token, [CLS].
The loss function maximizes the model’s probability of outputting the true product entity category as follows:
[Equation]
where ˆ y is the ground-truth product category.
Product Entity Aspect Summary Generation (PEASG)
Inspired by PEGASUS (Zhang et al., 2020), which proves that using a pre-training objective that more closely resembles the downstream task leads to better and faster ﬁne-tuning performance, we propose a PEASG objective to generate a summary from the description of a product entity aspect.
Unlike extracted gap-sentences generation in PEGASUS, our method constructs a more realistic summary generation task because the aspect summary naturally exists in our pre-training data.
Given an aspect description sequence x = ( x 1 , ..., x M ) , and an aspect summary sequence y =
( y 1 , ..., y T ) , PEASG aims to model the conditional likelihood P ( y | x ) .
The loss function is:
[Equation]
E XPERIMENTS AND R ESULTS 4. P RE - TRAINED M ODEL V ARIANTS
To evaluate the effectiveness of pre-training with domain-speciﬁc data and with domain-speciﬁc knowledge separately, we implement pre-training experiments with two model variants: C-PLUG and E-PLUG, whose conﬁgurations are the same as that of K-PLUG
• C-PLUG is a pre-trained language model with the original objectives of MLM and MS2S, trained with a general pre-training corpus, CLUE (Xu et al., 2020), which contains 30GB of raw text with around 8B Chinese words.
• E-PLUG is a pre-trained language model with the original objectives of MLM and MS2S, trained with our collected e-commerce domain-speciﬁc corpus.
4. D OWNSTREAM T ASKS
We ﬁne-tune K-PLUG on three downstream tasks: e-commerce KB completion, abstractive product summarization, and multi-turn dialogue.
The e-commerce KB completion task involves the prediction of product attributes and values given product information.
The abstractive product summarization task requires the model to generate a product summary from textual product description.
The multi-turn dialogue task aims to output the response by utilizing a multi-turn dialogue context.
The domain-speciﬁc knowledge we deﬁned in this paper is essential for these tasks.
4.2. E- COMMERCE KB C OMPLETION
Task Deﬁnition.
E-commerce KB provides abundant product information that is in the form of ( product entity , product attribute , attribute value ), such as ( pid#133443 , Material , Copper Aluminum ).
For the E-commerce KB completion task, the input is a textual product description for a given product, and the output is the product attribute values.
Dataset.
We conduct experiments on the dataset of MEPAVE (Zhu et al., 2020).
This dataset is collected from a major Chinese e-commerce platform, which consists of 87,194 instances annotated with the position of attribute values mentioned in the product descriptions.
There are totally 26 types of product attributes such as Material , Collar Type , Color , etc.
The training, validation, and testing sets contain 71,194/8,000/8,000 instances, respectively.
Model.
We consider the e-commerce KB completion task as a sequence labeling task that tags the input word sequence x = ( x 1 , ..., x N ) with the label sequence y =
( y 1 , ..., y N ) in the BIO format.
For example, for the input sentence “ A bright yellow collar ”, the corresponding labels for “ bright ” and “ yellow ” are Color-B and Color-I , respectively, and O for the other tokens.
For an input sequence, K-PLUG outputs an encoding representation sequence, and a linear classiﬁcation layer with the softmax predicts the label for each input token based on the encoding representation.
[Table]
• ScalingUp (Xu et al., 2019) adopts BiLSTM, CRF, and attention mechanism to extract attributes.
• JAVE (Zhu et al., 2020) is a joint attribute and value extraction model based on a pre-trained BERT.
• M-JAVE (Zhu et al., 2020) is a multimodal JAVE model, which additionally utilizes product image information.
Result.
Table 1 shows the experimental results.
We observe that our K-PLUG performs better than baselines.
C-PLUG achieves signiﬁcantly better performance than BERT, which indicates that MS2S can also beneﬁt the NLU task.
E-PLUG outperforms C-PLUG, showing that training with domainspeciﬁc corpus is helpful.
K-PLUG further exhibits a 2.51% improvement compared with E-PLUG.
In short, we can conclude that the improvement is due to both the domain-speciﬁc pre-training data and knowledge-injected pre-training objectives.
4.2. A BSTRACTIVE P RODUCT S UMMARIZATION
Task Deﬁnition.
Abstractive product summarization task aims to capture the most attractive information of a product that resonates with potential purchasers.
The input for this task is a product description, and the output is a condensed product summary.
Dataset.
We perform experiments on the dataset of Li et al.
(2020), which contains 1.4 million instances collected from a major Chinese e-commerce platform, covering three categories of product:
Home Appliances , Clothing , and Cases & Bags .
Each instance in the dataset is a (product information, product summary) pair, and the product information contains an image, a title, and other product descriptions.
In our work, we do not consider the visual information of products.
Model.
Abstractive product summarization task is an NLG task that takes the product description as the input and product summary as the output.
[Table]
• LexRank (Erkan & Radev, 2004) is a graph-based extractive summarization method.
• Seq2seq (Bahdanau et al., 2015) is a standard seq2seq model with an attention mechanism.
• Pointer-Generator (PG)
(See et al., 2017) is a seq2seq model with a copying mechanism.
• Aspect MMPG (Li et al., 2020) is the-state-of-the-art method for abstractive product summarization, taking both textual and visual product information as the input.
Result.
Table 2 shows the experimental results, including ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) F1 scores (Lin & Hovy, 2003).
K-PLUG clearly performs better than other text-based methods.
E-commerce knowledge plays a signiﬁcant role in the abstractive product summarization task, and domain-speciﬁc pre-training data and knowledge-injected pre-training objectives both enhance the model.
K-PLUG achieves comparable results with the multimodal model,
Aspect MMPG.
The work of Li et al.
(2020) suggests that product images are essential for this task, and we will advance K-PLUG with multimodal information in the future. [Table] [Table]
M ULTI -T URN D IALOGUE
Task Deﬁnition.
The multi-turn dialogue task aims to output a response based on the multi-turn dialogue context (Shum et al., 2018).
The input for this task is the dialogue context consisting of previous question answering [Table] and the output is the response to the last question.
Dataset.
We conduct experiments on two datasets of JDDC (Chen et al., 2020) and ECD (Zhang et al., 2018).
JDDC is collected from the conversations between users and customer service staffs from a popular e-commerce website in China and contains 289 different intents [Table] which are the goals of a dialogue, such as updating addresses, inquiring prices [Table]
etc, from after-sales assistance. There are 1,024,196 multi-turn sessions and 20,451,337 utterances in total.
The average number of turns for each session is [Table] and the average tokens per utterance is about 7.4.
After pre-processing, the training, validation [Table] and testing sets include 1,522,859/5,000/5,000 (dialogue context, response) pairs, respectively.
ECD is collected from another popular e-commerce website in China and covers over 5 types of conversations based on 20 commodities.
Additionally, for each ground-truth response, negative responses are provided for discriminative learning.
The training, validation, and testing sets include 1,000,000/10,000/10,000 (dialogue context, response) pairs, respectively.
Model.
We test with two types of K-PLUG: retrieval-based K-PLUG on the ECD dataset and generative-based K-PLUG on the JDDC dataset.
For the retrieval-based approach, we concatenate the dialogue context and use [SEP] token to separate context and response.
The [CLS] representation is fed into the output layer for classiﬁcation.
The generative-based approach is a sequence-tosequence model, which is the same as the model adopted in the abstractive product summarization task.
Baselines.
The baselines also include both the retrieval-based (BM25, CNN, BiLSTM, and BERT) and generative-based approaches.
Other baselines are as follows.
• SMN (Wu et al., 2017) matches a response with each utterance in the context. • DUA (Zhang et al., 2018) is a deep utterance aggregation model based on the ﬁne-grained context representations. • DAM (Zhou et al., 2018) matches a response with the context based using dependency information based on self-attention and cross-attention. • IoI
(Tao et al., 2019) is a deep matching model by stacking multiple interactions blocks between utterance and response. • MSN (Yuan et al., 2019) selects relevant context and generates better context representations with the selected context.
Result.
Table 3 and 4 show the experimental results on the JDDC and ECD datasets, respectively.
We report ROUGE-L (RG-L) F1, BLEU, and recall at position k in n candidates ( R n @ k ).
We can observe that, both on the retrieval-based and generative-based tasks, K-PLUG achieves new state-ofthe-art results, and e-commerce knowledge presents consistent improvements.
K-PLUG is evidently superior to BERT, possibly due to BERT’s lack of domain-speciﬁc knowledge for pre-training with the general MLM objective. We further perform a human evaluation on the JDDC dataset.
We randomly choose 100 samples from the test set, and three experienced annotators are involved to determine whether K-PLUG outperforms E-PLUG with respect to (1) relevance between the response and the contexts and (2) readability of the response.
The results are shown in Table 5.
We can see that the percentage of “Win”, which denotes that the results of K-PLUG is better than E-PLUG, is signiﬁcantly larger than “Lose” (p-value < 0.01 for t-test).
Kappa values (Fleiss, 1971) conﬁrm the consistency for different annotators.
The results in the ﬁrst block are taken from
Chen et al.
[Table] [Table] [Table]
A BLATION S TUDIES
To better understand our model, we perform ablation experiments to study the effects of different pre-training objectives.
Result.
The ablation results are shown in Table 6.
We can conclude that the lack of any pretraining objective hurts performance across all the tasks.
KMS2S is the most effective objective for the abstractive product summarization and generative conversation tasks since this objective is highly close to the essence of NLG.
Product-aspect-related objectives,
i
[Table]
e. , PEABD and PEASG, contribute much to the abstractive product summarization task, which proves that this task requires comprehensively understanding the product description from the view of product aspects, going beyond individual tokens.
[Table]
C ONCLUSION
We present a knowledge-injected pre-trained model (K-PLUG) that is a powerful domain-speciﬁc language model trained on a large-scale e-commerce corpus designed to capture e-commerce knowl - edge, including e-commerce KB, product aspects, product categories, and USPs.
The pre-training framework combines masked language model and masked seq2seq with novel objectives formulated as product aspect boundary detection, product aspect summary generation, and product category classiﬁcation tasks. Our proposed model demonstrates strong performances on both natural language understanding and generation downstream tasks, including e-commerce KB completion, abstractive product summarization, and multi-turn dialogue. A A PPENDIX A. E XPERIMENTS S ETTINGS FOR P RE - TRAINING
We adopt GELU activation (Hendrycks & Gimpel, 2016) as in GPT (Radford et al., 2018).
We use
Adam optimizer (Kingma & Ba, 2015) with a learning rate of 5e-4, β 1 = 0.9, β 2 = 0.98, L2 weight decay of 0.01, learning rate warm-up over the ﬁrst 10,000 steps and linear decay of the learning rate. The dropout probability is 0.1. The maximum sequence length is set to 512 tokens. Pre-training was performed with 4 Telsa V100 GPUs.
The pre-training is done within 10 epochs, which takes around 10 days, and the ﬁne-tuning takes up to 1 day.
We use the beam search with a beam size of 5 for inference for the NLG tasks.
Other hyper-parameters can be found in our code.
A. C ASE STUDIES
We present some examples from the test set of each task, with comparisons of the ground-truth result and the outputs produced by the models of E-PLUG and K-PLUG.
[Table]
A21’s [sports]
[Table]
windbreaker jacket with [contrasting color] design style)
[Table]
sports
[Table]
windbreaker]
[Table]
jacket with [contrasting color style]
[Table]
A21’s sports [windbreaker] style jacket with [contrasting color] design style)
[Table]
合
[ 微弹 ] 弹 性 的 [ 棉 质 ] 材 质 面 料 手 感 柔 软 顺 滑 (made from [low-strech] elasticity [cotton fabric] material for a silky smooth touch)
[Table]
配 合 [ 微弹 ] 裤 型 的 [ 棉 质 ] 材 质 面 料 手 感 柔 软 顺 滑 (made from [low-strech] pants ﬁt
[cotton fabric] material for a silky smooth touch)
[Table]
配 合 [ 微弹 ] 弹 性 的 [ 棉 质 ] 材 质 面 料 手 感 柔 软 顺 滑 (made from [low
[Table]
The K-PLUG model generates summaries describing more information about e-commerce knowledge bases and
[Table]
selling propositions of product entities .
[Table]
This kitchen range hood with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) extra suction and ventilation is ideal for a clean and fresh kitchen.
Perfectly combining 310pa of air pressure and 17 m 3 of suction power, it ﬁlters out grease before you realize it.
Never have to worry again about keeping your loved ones from cooking fumes. )
E-PLUG
[Equation]
(This kitchen range hood from Sacon is made of stainless steel with cutting edge technology.
Its design adds contemporary style to your European style kitchen.
It features (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) extra suction and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ventilation for immediate ﬁltering of fumes and grease.
Its intelligent touch control panel makes it extremely easy to use. )
K-PLUG
[Table]
[Equation]
[Table]
[Equation]
[Table]
一个 清 新 的 厨 房 环 境 。 310pa 大 风 压 ， 保 持排 烟 畅 通 ， 防 止 油 烟 倒 灌 。 低 至 53 分 贝 的 运 行 声 音 ， 让 您 告 别 厨 房 的 嘈 杂 。
(This stainless steel kitchen range hood features (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) extra suction and ventilation of 17 m 3 for immediate ﬁltering of grease and fumes, leaving your kitchen fresh.
Its 310 pa air pressure ensures clear ﬁltering without backﬂow.
Its low working noise of 53 dB keeps your kitchen nice and quiet.)
Input
[Equation]
[Table]
冬 天
[Equation]
漱 再 也不 冻 手 ，
[Equation]
的 配
[Equation]
[Equation]
， LED 数 字
[Equation]
屏 ， 封
[Equation]
式 燃 烧
[Equation]
温清
[Equation]
[Equation]
每次
[Equation]
全 新 享 受 ...
(Canbo’s affordable 8L thermal control natural gas residential kitchen and bathroom water heater.
Provides warm water for doing dishes and cleaning up in winter.
High-quality accessories.
High concentration oxygen-free copper tank.
LED digital screen.
Closed boiling system for safety.
Easy temperature control.
Multiple security checks.
Oxygen-free copper tank.
Lets you enjoy your showers ... )
[Table]
[Equation]
[Table]
[Equation]
[Table]
封 闭 式 燃 烧 技 术 ， 提 升 了 热 效 率 ， 节 能 又 高 效 。 拥 有 控 温 功 能 ， 搭 配 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) LED
数
字 大 屏 ， 让 你 能 够 根 据 需 求 来 随
意 调 节 水 温 。 (High concentration oxygen-free tank with added heating area boils water quickly with stability.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Closed boiling technique increases efﬁciency and saves energy.
Thermal control and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
digital (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) screen for adjustable water temperature anytime. )
E
PLUG
[Equation]
下也
[Equation]
也 适 用 。 (This gas water heater has a large volume and provides warm water year round.
It adopts the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) closed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) boiling technique for faster warming up and shorter wait time in shower.
It works in low pressure condition and is suitable for older neighborhoods. )
K-PLUG 高 配 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) LED 数
字 大 屏 ， 控 温清 晰 可 见 ， 温 度 随 时 掌控 于 心 。 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
封 闭 式 燃 烧 技 术 ， 快 速 度 燃 烧 ， 加 热 效 率 更 高 。 高 纯 度 无 氧 铜 水 箱 ， 加 热 速 度 更 快 。
(Equipped with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) LED digital screen for clear thermal control and temperature adjustment.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Closed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) boiling technique speeds up warming and increases efﬁciency.
High concentration oxygen-free copper tank for faster warming up.)
[Table]
手 感 柔 滑 ， 上 身 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 清 凉 舒 爽 ， 悬 垂 感 好 ， 耐 穿 性 强 。 胸
前 撞 色花 卉 印 花 结 合 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 珠 片 绣 花 ， 时 尚 又 大 方 。 经 典 圆 领 设计 ， 自 然 突 显 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
型
男 风 范 。 (Made from modal knit fabric .
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Incredibly soft and elastic .
Features a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly looser ﬁt .
Natural draping with high durability.
Decorated with ﬂower print and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) glitter embroidery for a stylish and casual look.
Finished with a classy crewneck , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
[Table]
[Table]
[Table]
[Table]
men’s
Mid
Designed
[Table]
(Made from polyester
[Table]
Feel soft , lightweight, and breathable.
Keeps you dry and comfortable.
Filled primarily with white duck down , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬂuffy and light .
Features a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) thick yet slim-ﬁt design.
Keeps you warm in cold climates
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
。 (This down jacket features a mid-thigh length , keeping a stylish silhouette and giving you an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Reference
[Table]
(Made from imported PVC coated fabric.
[Table]
Soft and comfortable .
[Table]
Water-proof and durable .
Specially made smoothly meshing zippers.
Long shoulder strap with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
leather
[Table]
This tote
[Table]
is made friom imported high-qiuality PVC coated fabric.
It has delicate
[Table]
and ﬂower
[Table]
decorations, giving it an elegant look.
It can hold many items
[Table]
making it ideal
[Table]
day
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
wheels.
Essential for the
[Table]
Strong and durable.
Breaks
[Table]
wheel design.
Includes dividers for easy
[Table]
of
[Table]
items
[Table]
Stylish design without compromising on durability.
Classic yet fashionable.
Closes and seals nicely
[Table]
Reference
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
added stability and durability .
Aluminum alloy frame makes it strong and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) close and seal nicely .
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Triple locks for added stability and security.
Includes (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dividers for easy and organized packing.)
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
aluminum alloy
[Table]
design to provide extra security.
Stable and effortless steering thanks to
[Table]
8-wheel design
[Table]
protect your items and cause no damage during security check
[Table]
[Table]