Why and when should you pool? Analyzing Pooling in Recurrent Architectures (2020)

Abstract: Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling on sequence classification tasks. However, the reasons for their enhanced performance are largely unexamined. In this work, we examine three commonly used pooling techniques (mean-pooling, max-pooling, and attention, and propose *max-attention*, a novel variant that captures interactions among predictive tokens in a sentence. Using novel experiments, we demonstrate that pooling architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases: (i) pooling facilitates better gradient flow than BiLSTMs in initial training epochs, and (ii) BiLSTMs are biased towards tokens at the beginning and end of the input, whereas pooling alleviates this bias. Consequently, we find that pooling yields large gains in low resource scenarios, and instances when salient words lie towards the middle of the input. Across several text classification tasks, we find max-attention to frequently outperform other pooling techniques.

## introduction pooling mechanisms are ubiquitous components in recurrent neural networks (rnns) used for natural language tasks. pooling operations consolidate hidden representations from rnns into a single sentence representation. various pooling techniques, like mean-pooling, max-pooling, and attention, have been shown to improve the performance of rnns on text classification tasks @xcite . despite widespread adoption, precisely how and when pooling benefits the models is largely under-explored. in this work, we perform an in-depth analysis comparing popular pooling methods, and proposed max-attention, with standard bilstms for several text classification tasks. we identify two key factors that explain the benefits of pooling techniques: learnability, and positional invariance. first, we analyze the flow of gradients for different classification tasks to assess the learning ability of bilstms ( § 5). we observe that the gradients corresponding to hidden representations in the middle of the sequence vanish during the initial epochs. on training for more examples, these gradients slowly recover, suggesting that the gates of standard bilstms require many examples to learn. in contrast, we find the gradient norms in poolingbased architectures to be free from this problem. pooling enables a fraction of the gradients to directly reach any hidden state instead of having to backpropagate through a long series of recurrent cells. thus we hypothesize, and subsequently confirm, that pooling is particularly beneficial for tasks with long input sequences. second, we explore the positional biases of bil-stms, with and without pooling ( § 6). across several classification tasks, and various novel experimental setups, we expose that bilstms are less responsive to tokens towards the middle of the sequence, when compared to tokens at the beginning or the end of the sequence. however, we find that this bias is largely absent in pooling-based architectures, indicating their ability to respond to salient tokens regardless of their position. third, we propose max-attention, a novel pooling technique, which combines the advantages of max-pooling and attention ( § 3.2). max-attention uses the max-pooled representation as its query vector to compute the attention weights for each hidden state. max-pooled representations are extensively used in the literature to capture prominent tokens (or objects) in a sentence (or an im-age) @xcite . therefore, using them as a query vector effectively captures interactions among salient portions in the input. max-attention is simple to use, and yields performance gains over other pooling methods on several classification setups. attention: first proposed as a way to align target tokens to the source tokens in translation @xcite , the core idea behind attentionlearning a weighted sum of the hidden states-has been widely adopted. as attention aggregates hidden representations, we consider it under the umbrella of pooling. recently, @xcite conjecture that attention offers benefits during training; our work explains, and provides empirical evidence to support the speculation. ## datasets & experimental setup we experiment with four different text classification tasks: (1) the imdb dataset @xcite contains movie reviews and their associated sentiment label; (2) yahoo! answers @xcite dataset comprises 1.4 million question and answer pairs, spread across 10 topics, where the task is to predict the topic of the answer, using the answer text; (3) amazon reviews @xcite contain product reviews from the amazon website, filtered by their category. we construct a 20-class classification task using these reviews 4 ; (4) yelp reviews @xcite is another sentiment polarity classification task. for these datasets, we only use the text and labels, ignoring any auxiliary information (like title or location). we select subsets of the datasets with sequences having greater than 100 words to better understand the impact of vanishing gradients and positional bias in recurrent architectures. a summary of statistics is presented in ## gradient propagation in this section, we study the flow of gradients in different architectures and training regimes. pooling techniques used in conjunction with bilstms provide a direct gradient pathway to intermediate hidden states. however for bilstms without pooling, it is crucial that the parameters for the input, output, and forget gates are appropriately learned so that the loss backpropagates across long input sequences, without the gradients vanishing. experimental setup: in order to quantify the extent to which the gradients vanish across different word positions, we compute the gradient of the loss function w.r.t the hidden state at every word position t, and study their ℓ 2 norm ( ∂l ∂ht ). to aggregate the gradients across multiple training exam-ples (of different lengths), we linearly interpolate the distribution of gradient values for each example to a fixed length between 1 and 100. the gradient values at each (normalized) position are averaged across all the training examples. we plot these values (on a log scale) after training on the first 500 imdb reviews to study the effect of gradient vanishing at the beginning of training (figure 2a ). to understand how the distribution of gradients (across word positions) changes with the number of training batches, we compute the ratio of the gradient norm corresponding to the word at the middle and word at the end: ∂l ∂h mid / ∂l ∂h end . 6 we call this the vanishing ratio and use it as a measure to quantify the extent of vanishing (where lower values indicate severe vanishing). each training batch on the x-axis in figures 2b , 2c corresponds to 64 training examples. results it is evident from figure 2a that the gradients vanish significantly for bilstm, with ∂l ∂ht falling to the order of 10 -6 as we approach the middle positions in the sequence. this effect is even more pronounced for the case of bilstm lowf , which uses the xavier initialization @xcite for the bias of the forget-gate. the plot suggests that specific initialization of the gates with best practices (such as setting the bias of forget-gate to a high value) helps to reduce the extent of the issue, but the problem still persists. in contrast, none of the pooling techniques face this issue, resulting in an almost straight line. ## positional biases analyzing the gradient propagation in bilstms suggests that standard recurrent networks are bi-ased towards the end tokens, as the overall contribution of distant hidden states is extremely low in the gradient of the loss. this implies that the weights of various parameters in an lstm cell (all cells of an lstm have tied weights) are hardly influenced by the middle words of the sentence. in this light, we aim to evaluate positional biases of recurrent architectures with different pooling techniques. ## discussion & conclusion through detailed analysis we identify why and when pooling representations are beneficial in rnns. while some of the results pertaining to gradient propagation in pooling-based rnns may be obvious in hindsight, we note that this is the first work to systematically and explicitly analyze the phenomenon.