B ayes-enhanced Lifelong Attention Networks for Sentiment Classification (2020)

Abstract: The classic deep learning paradigm learns a model from the training data of a single task and the learned model is also tested on the same task. This paper studies the problem of learning a sequence of tasks (sentiment classification tasks in our case). After each sentiment classification task is learned, its knowledge is retained to help future task learning. Following this setting, we explore attention neural networks and propose a Bayes-enhanced Lifelong Attention Network (BLAN). The key idea is to exploit the generative parameters of naive Bayes to learn attention knowledge. The learned knowledge from each task is stored in a knowledge base and later used to build lifelong attentions. The constructed lifelong attentions are then used to enhance the attention of the network to help new task learning. Experimental results on product reviews from Amazon.com show the effectiveness of the proposed model.

## introduction in recent years, deep learning methods have brought sentiment classification to a new height with big data, see a survey paper @xcite . however, these deep learning methods primarily focus on learning a model from the training data of one task and the learned model is tested on the same task. in this paper, we study the problem of learning a sequence of sentiment classification tasks. this learning setting is called lifelong learning. lifelong learning is a continual learning process where the learner learns a sequence of tasks; after each task is learned, its knowledge is retained and later used to help new/future task learning @xcite @xcite @xcite . lifelong learning has been introduced extensively in a book, see @xcite . the first lifelong learning method for sentiment classification (called lifelong sentiment classification) was introduced in @xcite . following @xcite , several other lifelong sentiment classification approaches were proposed respectively in @xcite @xcite ). we will discuss these and other related work in the next section. among these, @xcite is a pioneering lifelong sentiment classification method based on deep learning. following these existing works, we treat the classification in each domain (e.g., a type of product) as a learning task. thus, we use the terms domain and task interchangeably throughout the paper. formally, we study the following problem. problem definition: at any point in time, a learner has learned n sentiment classification tasks. each task t i | n @xmath0. the learned knowledge is retained in a knowledge base. when faced with a new sentiment classification task t n+1 , the knowledge in the knowledge base is leveraged to help learn the new task t n+1 . after t n+1 is learned, its knowledge is also incorporated into the knowledge base for future task learning. the above-defined problem is a reformulation from the work of @xcite . as can be seen, the main challenges of this problem are 2-fold: (1) what forms of knowledge should be retained from each task and (2) how does the learner use the knowledge to help future task learning? these are also the challenges faced by lifelong learning. this paper studies the problem of lifelong learning for sentiment classification. below let us consider an example in sentiment classification. "i bought this camera last month. it works good although a little heavy. would recommend this." we (humans) can observe that (1) the first sentence shows a neutral sentiment (or no sentiment), (2) the second sentence expresses both positive (i.e., good) and negative (i.e., heavy) sentiments, and (3) the third sentence shows a strong positive (i.e., recommend) sentiment. the overall sentiment polarity of this example is positive. one reason why we can understand these clearly is that we give different focuses/attentions to each word and each sentence. we give strong attentions to the words (good, heavy, and recommend) and the last two sentences. another crucial reason is that we have accumulated a large amount of knowledge in the past, and use it now to help understanding. for example, we know that the word "good" is positive for all products, and the word "heavy" is negative for camera but may be positive for some other products. such a problem is called domain generality and domain specificity as introduced in @xcite . the problem is particularly acute in lifelong learning as lifelong learning needs to tackle a large number of (possibly never ending) domains/tasks. thus, we motivate and propose to use attention from previous tasks as knowledge to help future task learning. attention has become enormously popular as an essential component of neural networks especially for a wide range of natural language processing tasks @xcite . attention neural networks was first introduced for machine translation @xcite . since then, a large body of sentiment classification models using attention mechanisms have been arrived -to name a few @xcite @xcite @xcite @xcite . however, none of them is a lifelong sentiment classification method. most of them learn in isolation as the classic deep learning paradigm learned, although some methods work for multi-domain sentiment classification which focus on leveraging data in source domain to help target domain. so, they are far from sufficient for our goal as we aim at building a model to learn a sequence of sentiment classification tasks and learn them successively. in addition, whether attention is explainable (or interpretable) was recently discussed in @xcite @xcite . @xcite and @xcite show the importance of attention for neural networks. in this paper, we introduce a knowledge-enhanced attention model which we call blan (as shorthand for bayes-enhanced lifelong attention network) for learning a sequence of sentiment classification tasks. the idea is that the model parameters of naïve bayes are first generated from each task. these parameters are then transformed to knowledge, retained in the knowledge base and later used to build lifelong attentions. frequent pattern mining @xcite is used in extracting lifelong attentions. the extracted lifelong attentions are used to enhance attention neural networks. this enhancement is especially useful when the training data of a task is small and thus, attention neural networks do not work well. the proposed lifelong attention mechanism using naïve bayes and frequent pattern mining is helpful and promising. these will be clear later. the experimental results demonstrate the superiority of the proposed model against multiple baseline methods. in summary, the paper makes the following contributions: ## related work lifelong learning, an advanced machine learning paradigm has been recently researched significantly in a relatively short time, see the book @xcite . in brief, lifelong learning aims to imitate "human learning" process and capability, i.e., accumulating the knowledge learned in the past and using it to help future learning and problem solving. lifelong learning for sentiment classification (called lifelong sentiment classification) was first introduced in @xcite , which is based on optimization considering the knowledge from previous tasks to help future task learning. following this work, @xcite investigated lifelong sentiment classification based on voting of each individual task classifier. @xcite proposed a heuristic naïve bayes which achieves forward knowledge transfer to improve any future task learning and also achieves backward knowledge transfer to improve the model of any past task without retraining using the past task training data. @xcite proposed a continuous naïve bayes learning framework for e-commerce product review sentiment classification by extending the parameter estimation mechanism in naïve bayes. @xcite introduced a deep learning method for lifelong sentiment classification by jointly training two networks, a feature learning network and a knowledge retention network. such a method is the first lifelong sentiment classification method based on deep learning. we have compared this deep learning method in our experiments and show that it is inferior to our proposed method. @xcite @xcite also proposed lifelong learning methods for sentiment classification, but they focus on aspect-based analysis, which is different from document-level sentiment classification (our focus in this paper). there are currently several research topics that are closely related to lifelong learning -most notably, multi-task learning and transfer learning. multi-task learning vs. lifelong learning: multi-task learning @xcite jointly optimizes the learning of multiple tasks. although it is possible to make it continual, multi-task learning does not retain any knowledge. lifelong learning is a continual learning process where the learner learns task incrementally, accumulates knowledge from each task and later uses it to help future task learning. transfer learning vs. lifelong learning: transfer learning (pan and yang, 2010) uses the source domain labeled data to help target domain learning which has no or little labeled data. unlike lifelong learning, transfer learning is not continual and has no knowledge retention. the source should be similar to the target (which are normally selected by the user). domain adaptation is a type of transfer learning by establishing knowledge transfer from a labeled source domain to an unlabeled target domain. domain adaptation with multiple sources (or called multi-source domain adaption) aims to use the labeled data collected from multiple sources to help target domain learning @xcite . however, these methods are only one-directional, i.e., sources helps target because the target has no or little labeled data. our work is also related to knowledge-enhanced attention neural networks. knowledge-enhanced attention models have been introduced in @xcite @xcite @xcite . however, they do not concern sentiment classification task. although there are a large body of attention models for sentiment classification as we mentioned in the previous section, none of them uses enhanced attention mechanisms. they mainly use self-attention learned from the networks. our proposed attention mechanism is a combination of self-attention and enhanced-attention. we detail our attention mechanism and the proposed model next. ## proposed model the proposed blan model is a hierarchical architecture as shown in figure 1 . the key characteristic of blan is the bayes-enhanced lifelong attention (to build knowledge-enhanced attention). this is done by using the generative model parameters p w i |c k of the naïve bayes, formulated as lifelong attention: suppose we are given a task sequence (t 1 , ..., t n , t n+1 , ...), our model has learned the first n tasks. the above two types of knowledge (i.e., r w|+ and r w|-) of each task has also been retained in the kb. the knowledge mining process is marked using a knowledge miner in figure 1 . again, we use the terms domain and task interchangeably throughout the paper because we treat the classification in each domain as a learning task following the existing lifelong sentiment classification works. we now focus on using the retained knowledge and the newly learned knowledge from the new/target task t n+1 to generate lifelong attentions (namely lifelong word-level attention and lifelong sentence-level attention as shown later) for the target task. as introduced early, we need to tackle the problems of domain-generality and domain-specificity. for each word w in the target task, we rely on the knowledge from all domains to solve the former. we filter/remove the domain-specific knowledge from previous domains to address the latter when helping new task/domain learning. here we follow @xcite and propose to filter the domain-specific knowledge using frequent pattern mining (fpm) @xcite . a frequent pattern is a set of items that appear frequently (above a minimum frequency threshold, e.g., 10, called minimum support) in a database of transactions. taking positive attention knowledge as an example, we treat the positive attention knowledge of all words in each domain as items. a set of items in one attention distribution form one transaction. given all transactions, fpm can find frequent items. then, we treat those infrequent items as domain-specific knowledge and filter out the domain-specific knowledge of each previous domain. let rt w|+ and rt w|-denote the results of the word w after performing fpm in the t-th previous domain/task. the lifelong word-level attentions foot_1 of this word for the target task t n+1 are formulated as assume that all words in a sentence s are conditionally independent given the class (which follows the naïve bayesian assumption). then, the lifelong sentence-level attentions of s are formulated as bayes-enhanced lifelong attention network: when the target task t n+1 comes, our model first extracts the above two types of lifelong attentions. the extracted lifelong attentions are used to enhance the hierarchical neural networks. given a document d with q sentences {s 1 , ..., s q } and the sentence s j with words {w j 1 , ..., w j a }, where j denotes the j-th sentence s j , the proposed blan works as follows: word-level learner (wll): for each word in the sentence s j , wll first looks up the embedding vector from pre-trained word embedding vectors. the embedding vectors of all words in this sentence are fed into a bidirectional gru (bi-gru) @xcite . then wll can learn a representation for each word (e.g., w j i ) by concatenating the forward hidden state -→ h j i and the backward hidden state ←h j i . the representations of these words in sentence s j with positive and negative attentions are fed into a positive sentence-level learner (psll in figure 1 ) and a negative sentence-level learner (nsll in figure 1 ), respectively. let h j @xmath1, ..., h j a ] and w @xmath2, ..., w j a ], then the network attentions αw j in a vector form are formulated as for each word w j i (@xmath3, ..., a) in w j , we propose to use the lifelong word-level attention to enhance the network's attention, formulated as - sentence-level learner (sll): two slls (positive and negative, i.e., psll and nsll in figure 1 ) receive the output (i.e., eq. ( 8 )) of wll. similar to wll, psll and nsll use bi-gru to learn representations (h then, the network's sentence-level attentions β+ s and βs in a vector form are formulated as similar to wll, each sll also learns a unified representation of the sentences {s 1 , ..., s q } in the document d by using knowledge-enhanced attentions. for each sentence s j (@xmath4, ..., q), we propose to use bayes-enhanced lifelong sentence-level attention to enhance the network's attention: document-level learner (dll): the learned d + and d -are fed into a full connection layer (fcl) to produce the positive sentiment score p +|d and the negative sentiment score p -|d for this document. dll learns the class label by performing arg max{p +|d , p -|d }. for the objective function, the motivation is that the proposed model should assign a large positive sentiment score p +|d (meanwhile, a small or even zero negative sentiment score p -|d ) to a positive document, and the reverse results to a negative document. thus, we propose to maximize the objection function below algorithm 1: blan in lifelong learning for sentiment classification input : target domain training documents d, knowledge r t w|+ and r t w|-(@xmath5, ..., n) of the previous n tasks (stored in the kb) output: blan in the target domain 1 learn knowledge r n+1 w|+ and r n+1 w|-of the target domain and store the newly learned knowledge in the kb ; 2 for each document d in a training batch do 3 for each sentence s j in d do 4 for each word w j i in the sentence s j do 5 learn word-level attention α + w j i and α - w j i using eq. (7) ; 6 end 7 learn unified representation s + j and s - j for the sentence s j using eq. ( 8 ) ; in practice, we solve a logarithmic minimization problem with a regularization term. formally, the objective function to train our model is formulated as follows ## conclusions in this work, we studied deep learning for a sequence of sentiment classification tasks. after each task learned, its knowledge was retained to help subsequent task learning. this problem is called lifelong sentiment classification. this paper proposed a novel knowledge-enhanced attention network model for lifelong sentiment classification. the proposed model learns to build two types of lifelong attentions by exploiting the model parameters of naïve bayes. the built lifelong attentions are used to enhance the overall attention networks. experimental results showed that the proposed model outperforms a wide range of baselines markedly. the proposed model works for two-class (positive and negative) problem. in our future work, we plan to extend it to multi-class problem. we also plan to address catastrophic forgetting problem @xcite in neural networks and bring our method to a new height.