Attentive Pooling with Learnable Norms for Text Representation (2020)

Abstract: Pooling is an important technique for learning text representations in many neural NLP models. In conventional pooling methods such as average, max and attentive pooling, text representations are weighted summations of the L1 or L∞ norm of input features. However, their pooling norms are always fixed and may not be optimal for learning accurate text representations in different tasks. In addition, in many popular pooling methods such as max and attentive pooling some features may be over-emphasized, while other useful ones are not fully exploited. In this paper, we propose an Attentive Pooling with Learnable Norms (APLN) approach for text representation. Different from existing pooling methods that use a fixed pooling norm, we propose to learn the norm in an end-to-end manner to automatically find the optimal ones for text representation in different tasks. In addition, we propose two methods to ensure the numerical stability of the model training. The first one is scale limiting, which re-scales the input to ensure non-negativity and alleviate the risk of exponential explosion. The second one is re-formulation, which decomposes the exponent operation to avoid computing the real-valued powers of the input and further accelerate the pooling operation. Experimental results on four benchmark datasets show that our approach can effectively improve the performance of attentive pooling.

## introduction in recent years, neural network based methods are widely used in the natural language processing (nlp) field to learn text representations @xcite . in these methods, pooling is a core technique to build the text representation vector from a collection of input feature vectors by summarizing their information @xcite . thus, an effective pooling method sentiment classification average pooling the movie is good, but not to my taste max pooling the movie is good, but not to my taste attentive pooling the movie is good, but not to my taste news topic classification average pooling fire on queensland island takes heavy toll on wildlife max pooling fire on queensland island takes heavy toll on wildlife attentive pooling fire on queensland island takes heavy toll on wildlife that can select salient features accurately will facilitate many nlp methods @xcite . among existing pooling methods, average pooling is a representative one which takes the average of the l 1 norm of input features @xcite . however, average pooling equally regards the input representation vector at each position and ignores their different informativeness for learning text representation, which may not be optimal @xcite . thus, other pooling methods such as max pooling @xcite and attentive pooling @xcite @xcite @xcite are widely used in neural nlp models. for example, @xcite proposed to apply max pooling to the contextual word representations learned by cnn networks to build the representations of the entire sentence. @xcite proposed to use attentive pooling at both word and sentence levels to learn informative sentence and document representations by selecting important words and sentences. however, these pooling methods use fixed average norms, i.e., l 1 norm for average and attentive pooling and l ∞ norm for max pooling, to build text representations, which may not be optimal when handling different tasks. our work is motivated by the following obser-vations. first, different contexts usually have different informativeness for learning text representations. for example, in fig. 1 foot_0 , the word "but" is very important for inferring the sentiment polarity of this sentence, while "the" is uninformative. thus, modeling the different informativeness of contexts and attending to them differently may help learn more informative text representations. second, different tasks and even different datasets have different characteristics. for example, in fig. 1 , sentiment and negation words may be the key clues for inferring the sentiment polarity of the first sentence, while the global contexts may be useful for understanding the topic of the second sentence. thus, using a fixed pooling norm for universal text representation learning is probably not optimal. third, in popular pooling methods such as max pooling and attentive pooling, some contexts may be over-emphasized, and other useful contextual information is not fullyrespected. for example, as shown in fig. 1 , the sentiment word "good" is highlighted, but other useful clues such as "but" and "not" do not gain sufficient attentions, which may not be optimal for learning accurate text representations. thus, a dynamically learnable degree of "hard" or "soft" for pooling may benefit text representation learning. in this paper, we propose an attentive pooling with learnable norms (apln) approach to enhance the learning of text representations foot_1 . instead of manually setting a fixed pooling norm, we propose to automatically learn it in a unified framework, which can find the optimal values to learn text representations for different tasks in an end-to-end manner. in addition, since the learning of pooling norm may be numerically unstable in some cases due to the exponent operation, we propose two methods to improve its computational stability. the first one is limiting the scale of input features, which aims to ensure their non-negativity and avoid exponential explosion. the second one is a re-formulation method, which aims to avoid computing the real-valued power of input features by decomposing the exponent operation into three safe and fast atomic operations. we conducted experiments on four benchmark datasets, and the results show that our approach can effectively improve the learning of text representation. ## related work neural networks are widely used to learn text representations from contexts @xcite . pooling is usually an essential step in these methods to build contextual representations by summarizing the information of input features @xcite . the simplest pooling method is average pooling, which is used in many approaches to construct text representations @xcite . for example, @xcite proposed to apply average pooling to the output of cnn filters to capture global contexts in a sentence. in addition, they also proposed to average the sentence representations learned by parallel cnn networks with different window sizes. in their another work @xcite , they proposed to apply average pooling to the sequence of sentence representations to build the representations of an entire document. although average pooling is computationally efficient, it cannot distinguish important contexts from unimportant ones, which may not be optimal for learning accurate text representations. there are also other popular pooling methods that can select salient features to learn more informative text representations, such as max pooling @xcite and attentive pooling @xcite , which are employed by many neural nlp methods @xcite @xcite @xcite @xcite @xcite @xcite . for example, @xcite proposed to learn representations of contexts within each window using feed forward neural networks, and used max pooling to build final text representations. @xcite proposed to apply max pooling over time to the contextual word representations learned by multiple cnn filters. @xcite proposed to build representations of the entire document using the summation of word representations weighted by their tf-idf scores. @xcite proposed a hierarchical attention network to first learn sentence representations from words and then learn document representations from sentences. they proposed to apply attentive pooling at both word and sentence levels to select informative words and sentences for more informative representation learning. @xcite proposed a hierarchical user and 𝛼 1 𝛼 2 𝛼 𝑁 1 𝑁 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 max-over-time ... 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝒒 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝑥 𝑝 𝑥 𝑝 𝑥 𝑝 ... 𝑥 1/𝑝 × × × σ 𝛼 1 𝛼 2 𝛼 𝑁 𝒒 × × × σ ... σ (a) average pooling. 𝛼 1 𝛼 2 𝛼 𝑁 1 𝑁 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 max-over-time ... 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝒒 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝑥 𝑝 𝑥 𝑝 𝑥 𝑝 ... 𝑥 1/𝑝 × × × σ 𝛼 1 𝛼 2 𝛼 𝑁 𝒒 × × × σ ... σ (b) max pooling. 𝛼 1 𝛼 2 𝛼 𝑁 1 𝑁 𝒗 1 𝒗 2 𝒗 𝑁 max-over-time ... 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝒒 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝒗 1 𝒗 𝑥 𝑝 × × × σ 𝛼 1 𝛼 𝒒 × ... σ (c) attentive pooling. item representation model with three-tier attention, which applies attentive pooling to simultaneously select important words, sentences and reviews. however, the pooling norms of max and attentive pooling are always fixed, which may not be optimal for universal text representation learning since the characteristics of different tasks may be different. in addition, both pooling methods may over-emphasize the most salient features, and other useful contextual information is not fully exploited, which may also be sub-optimal. there are a few methods to adapt the pooling norms in different tasks. for example, @xcite explored the influence of selecting different pooling norms on the performance of different image classification tasks. however, the norms in their method are manually tuned, which are usually very time-consuming and may not be optimal. different from all aforementioned methods, our approach can automatically optimize pooling norms in an end-to-end manner, and can effectively select important contexts to learn informative text representations. extensive experiments on four datasets with different characteristics validate the effectiveness of our approach. ## preliminaries in this section, we will first present a brief introduction to several popular pooling methods, i.e., average, max and attentive pooling. to make it easier to understand, we present an intuitive comparison of the mechanisms of these different pooling methods in fig. 2 . average pooling. average pooling is used to build contextual representations by taking the arithmetic mean of input features, as shown in fig. 2(a ). it uses the l 1 norm of the input. denote the input sequence of hidden representations as [h 1 , h 2 , ..., h n ], where n is the sequence length. the output representation is computed as: max pooling. max pooling aims to build contextual representations by selecting the most salient features via max-over-time operations, as shown in fig. 2(b) . it utilizes the l ∞ norm at the time dimension of input features. denote r j as the j-th value in the vector r, which is computed as: attentive pooling. as shown in fig. 2(c ), attentive pooling usually builds contextual representations by selecting important input features, which can also be regarded as a kind of l 1 norm average. it computes an attention weight α i for the input at each position to indicate its informativeness, which is formulated as follows: 𝑁 max-over-time 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝑁 𝒓 𝒗 1 𝒗 2 𝒗 𝑁 ... 𝑥 𝑝 𝑥 𝑝 𝑥 𝑝 ... 𝑥 1/𝑝 × 𝛼 1 𝛼 2 𝛼 𝑁 𝒒 × × × σ figure 3: architecture of our attentive pooling with learnable norms (apln) approach. ## attentive pooling with learnable norms in this section, we will introduce the details of our attentive pooling with learnable norms (apln) approach. in the aforementioned pooling methods, the pooling norm is always fixed (i.e., l 1 or l ∞ ). however, the characteristics of different nlp tasks and even different datasets should have some differences, and it may not be optimal to use a fixed pooling norm for universal text representation learning. in addition, tuning the pooling norm manually is usually very time-consuming, and it may also be sub-optimal. thus, it is an intuitive idea to automatically learn the pooling norm in an end-to-end manner to alleviate the efforts on hyperparameter searching and learn more informative text representations. the architecture of our apln approach is shown in fig. 3 . we will introduce its details as follows. since different contexts usually have different importance, modeling their informativeness may help learn more informative text representations. thus, similar to the vanilla attentive pooling, in our apln approach, we also compute an attention score for the input at each position. however, instead of using the simple weighted summation to build the contextual representation r, we propose to compute the l p norm 3 average of the input feature vectors weighted their attention weights, which is formulated as follows: to show the influence of p on the inputs of the apln module, we vary the value of p and illustrate the shape of the function @xmath0. 4 . according to fig. 4 , we can see when p is larger, the attention of apln is sharper and sparser since small values of α i h i will be suppressed, which indicates the attentive pooling is "harder". in contrast, if p is smaller, the attentions are more distributed, which indicates the attentive pooling is "softer". thus, in this manner, our apln model can automatically explore how "hard/soft" the attention should be when constructing text representations, which may help recognize important contexts and avoid the problem of over-emphasizing some features and not fully respecting other useful ones, both of which are important for learning accurate text representations. unfortunately, in most cases the training of apln is unstable if we directly use it for pooling. thus, we propose two methods to ensure the numerical stability of the model training. the first one is scale limiting, which is used to limit the range of the elements of α i h i . the second one is re-formulation, which is used to avoid the direct computation of the real-valued powers of the input features and accelerate the pooling operation. we will introduce the two methods as follows. ## conclusion and future work in this paper, we propose an attentive pooling with learnable norms (apln) approach for text representation. instead of using a fixed pooling norm for universal text representation learning, we propose to learn the norm in an end-to-end framework to automatically find the optimal ones for learning text representations in different tasks. in addition, we propose two methods to ensure the numerical stability of the model training. the first one is scale limiting, which limits the scale of input representations to ensure their non-negativity and avoid potential exponential explosion. the second one is re-formulation, which decomposes the exponent operation into several safe atomic operations to avoid computing the real-valued powers of input features with less computational cost. extensive experiments on four benchmark datasets validate the effectiveness of our approach. in our future work, we will explore several potential directions. first, we plan to explore why the model prefers "soft" attentions rather than "hard" ones, which is different from the findings in several prior works based on hard attention. second, we plan to study how to model the differences on the characteristics of different samples and use different pooling norms, which may have the potential to further improve our approach. third, we will explore how to generalize our approach to other modalities, such as images, audios and videos, to see whether it can facilitate more attention-based methods.