A DAPTING
P RETRAINED
L ANGUAGE
M ODELS
FOR
L ONG D OCUMENT C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Pretrained Language Models (LMs) have shown excellent results in achieving human like performance on many language tasks.
However, the most powerful LMs have one signiﬁcant drawback: a ﬁxed-sized input.
Due to this constraint, these
LMs are unable to utilize the full input of long documents.
In this paper, we introduce a new framework to handle documents of arbitrary lengths.
We investigate the addition of different mechanisms to handle an extended input size and utilizing attention to identify the most discriminating segment of input.
We perform extensive validating experiments on patent and Arxiv datasets, both of which have long text.
We demonstrate our method consistently outperforms state-of-the-art results reported in recent literature.
I NTRODUCTION
Neural Network based Language Models (LMs) have seen a ﬂurry of work, where new design and implementation improvements have advanced state-of-the-art performance in a variety of natural language tasks over the past few years (Devlin et al., 2018; Dai et al., 2019; Radford et al., 2019;
Yang et al., 2019; Liu et al., 2019).
LMs are powerful tools because they process a collection of unlabeled text and learn a rich embedding of natural language without supervision.
This representation can be re-purposed on subsequent tasks such as classiﬁcation and sentiment analysis (Korde & Mahender, 2012).
This technique is essential for reaching state-of-the-art performance, as LM based systems are able to achieve much better results than techniques that only use a small, labeled dataset.
Modern LMs achieve thier success by utilizing a powerful mechanism called “The Transformer” (Vaswani et al., 2017).
The transformer learns strong dependencies between its inputs, and has the ability to be stacked as many times as the hardware can handle.
This mechanism allows LMs to take in relatively large, yet still ﬁxed ,
size input
For the largest LMs, the input size can reach up to four thousand tokens; however, this is still a limitation as they cannot process arbitrarily long documents.
On many natural language tasks, this ﬁxed input size is sufﬁcient.
For example, Reading comprehension tasks are often used to analyze the quality of LMs (Wang et al., 2019), contain relatively few words on average, and do not have sufﬁciently long dependencies.
Furthermore, there are multiple tasks where the input data is too long and must be truncated before being processed by these LMs (Lee & Hsiang, 2019).
Truncation is unsuitable because long complex text often contains inter-referential pieces of information.
For instance, reading the ﬁnal chapter of a book after all the previous ones, takes on a different meaning than reading the same text by itself.
Solving the problem of arbitrarily long input requires more than a cursory glance.
A ﬁrst intuition may be to take a pretrained LM, separate the text into segments, place a Recursive Neural Networks (RNN) after the embedding, and simply pass the segments sequentially.
This seems reasonable as
RNNs have been used in the past for sequential text-based tasks, e.g. sentiment analysis (Socher et al., 2011).
Unfortunately, using RNNs in this way causes two problems.
First, RNNs are typically trained via backpropagation through time, making them prone to the problem of vanishing or exploding gradients (Pascanu et al., 2012).
While many techniques exist to deal with this issue (Williams & Peng, 1990; Mujika et al., 2018), they do not solve the secondary problem of signiﬁcant memory requirements for LM parameter updates.
Each forward pass through the LM produces a set of parameter gradients.
With multiple forward passes, the number of stored gradients quickly grows as a segmented document increases in length.
To further complicate the issue, the number of parameters in transformer-based networks is quadratic in relation to the ﬁxed input size.
In order to solve these problems, we look at the brain’s ability to guide attention and provide behavioral updates, as a biological inspiration.
The brain processes an extraordinary amount of data, yet from moment to moment, much of that information is ﬁltered out.
What becomes ﬁltered is not arbitrary, but is directly inﬂuenced by one’s objective.
This ﬁltering can be observed when an individual is tasked with counting the number of basketball passes in a video, yet fails to register a large gorilla that appears in center frame.
(Chabris & Simons, 2011).
Even outside the moment to moment, changes in an individual’s behavior is guided by the structure of their values, self-selected or otherwise.
An individual’s value structure imposes a framework for determining signiﬁcance of events (Peterson, 1999).
An event that may have been insigniﬁcant in the past can go on to take a new meaning once a new value has been gained or an old one has changed (Laudet et al., 2006).
This increase in valence may cause a behavioral change, a reorientation of goals, or a shift in the interpretation of experiences.
Unfortunately, when it comes to modeling this selective learning in Artiﬁcial Intelligence systems, common gradient based methods fall short.
A typical neural network model will update every parameter based on all of the inputs to minimize an objective function.
While this is desired for many applications, full input based learning can cause issues for other applications (Pascanu et al., 2012).
Ke et al.
(2018) attempt to solve this issue via selective attentive backtracking, though they focus on remembering long-term dependencies rather than utilizing the full context of the input.
In this work, we introduce a method that performs objective-based ﬁltering during learning, but still utilizes the entire input.
We focus on two domains speciﬁcally for classiﬁcation: patents and scientiﬁc papers.
Patent reading is a typical activity for lawyers trying to ﬁnd relevant documents.
The number of patents ﬁled increases nearly every year since 2003 (WIPO, 2018); and using an automated system to perform classiﬁcation is a continuously growing area of interest (Trappey et al., 2006).
Scientiﬁc papers are a signiﬁcant area of investigation.
Many papers are uploaded to the internet every day, and their automatic categorization is becoming a necessity.
A statistics paper may not be categorized as machine learning by its authors, but could be of interest to the machine learning community.
Additionally, Tshitoyan et al.
(2019) show material science concepts can be learned from scientiﬁc papers by a language model.
They demonstrated materials for functional applications could be recommended several years before their discovery.
In this work, we use an attention mechanism to discover the signiﬁcant portions of text to perform updates over a pretrained LM.
We ﬁnd attention-based updates to be essential because the most signiﬁcant portion of an input sequence may occur anywhere throughout the document.
While the datasets we study often start with highly discriminative features (titles and abstracts), we conduct experiments to show our attention mechanism can ﬁnd the important parts of text even when it does not occur in the ﬁrst segment.
In either case, using the gradients from only the ﬁrst input segment for learning performs well– often better than just using a baseline of the original language model with the input truncated to ﬁt the max size.
However, we ﬁnd using an attention mechanism consistently improves performance and achieves the best results in our experimented language models.
Therefore, our contributions are as follows:
We introduce a new framework for performing inference over arbitrary length documents.
We perform extensive validating experiments of our methods, showing how our attentionbased framework consistently outperforms alternative methods.
We demonstrate the attention mechanism can be utilized to perform selective language model parameter updates.
R ELATED W ORKS
Language model pretraining is a popular method for tackling many natural language understanding tasks; Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) and
Generative Pre-Training (GPT) (Radford et al., 2018) are two well-known language model pretraining methods that we utilize in this work.
BERT is trained by selectively conditioning on most of the input sequence, masking the rest, and attempting to predict the masked tokens.
BERT is concurrently trained with a next sentence prediction task.
The token used for the next sentence prediction task is reused for classiﬁcation after BERT has been trained.
GPT trains via next word prediction by conditioning on a sequence of text and trying to predict the next token.
By training on Wikipedia and books, GPT is able to generate novel sequences of text.
GPT is able to perform classiﬁcation by appending a special classifying token at the end of an input, encoding the token into a latent representation, and using the representation as input to new linear classiﬁer set speciﬁcally for the task.
These language model pretraining methods became state-of-art on natural language processing benchmarks such as GLUE (Wang et al., 2019) and SQUAD (Rajpurkar et al., 2016), achieving close to human-level performance.
As both BERT and GPT are based on Transformers, their computational and storage costs scale quadratically with the input sequence length.
This limits their application to mostly relatively short pieces of texts.
To the best of our knowledge, we are not aware of any works applying these pretrained language models for long document classiﬁcation.
Next, we discuss a few other deep learning-based text classiﬁcation approaches, with special attention to the classiﬁcation of scientiﬁc papers and patents.
Dai & Le (2015) consider pretraining recurrent neural networks with large corpus of texts, and show improved performance on several text classiﬁcation tasks.
Kim (2014) introduced convolutional neural networks (CNN) for text/sentence classiﬁcation.
Yang et al.
(2016) introduced a hierarchical attention mechanism for document classiﬁcation that attends to interesting sentences and words in a document.
However, the length of documents considered in latter two works are relatively short, with the corpus consisting of mostly individual sentences or online reviews.
For patent classiﬁcation, Li et al.
(2018) present a deep learning algorithm called Deep Patent, which is based on convolutional neural networks and continuous skip-gram embedding.
They were the ﬁrst to apply deep learning to large scale real-world patent classiﬁcation.
Lee & Hsiang (2019) used the pretrained BERT model to classify patents at the section and subclass level, only taking the title and abstract, or the ﬁrst claim, as input.
For scientiﬁc paper classiﬁcation,
He et al.
(2019) introduce a relatively large scientiﬁc paper dataset and perform classiﬁcation through a multi-network approach.
They introduce a RNN Attentionbased Learning reinforcement learning scheme.
They select short text sequences to be parsed by a
CNN, and the representation learned by the CNN is sequentially fed back into the RNN for subsequent text selections and eventual classiﬁcation.
Very recently, Cohan et al.
(2019) construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences.
However, their task is classiﬁcation at the sentence level for a single text input.
M ETHODS
Our approach for classifying long documents is to divide the long document into a sequence of segments, each of which is short enough to be processed by a pretrained language model.
The in- formation from the language model’s representation of each segment is utilized in different manors to produce a classiﬁer (see Figure 1(right)).
We develop multiple strategies to combine these representations, progressively enhancing the combinations– starting from simple concatenation to an attention-based strategy.
In this work, one of our primary contributions is to investigate the effectiveness of these different enhancements.
Let x = ( x 1 , x 2 , . . .
, x m ) be a document, where x i is a ﬁxed-length sequence of tokens (segment), and m the number of segments in the document.
Let y ∈ Y be the respective labels in a k -class classiﬁcation problem.
We use z i =
LM( x i ) ∈
R d to denote the d -dimensional latent representation, of the segment x i , for classiﬁcation (e.g. the representation of the “CLS” token in BERT) from a selected language model LM .
Let C W , b ( v ) =
σ ( W v + b ) be a linear classiﬁer followed by the softmax function, and let p the vector of probabilities of x being assigned to each class.
Base Language Model (Base LM)
In the usual application of deep language models such as
BERT and GPT for text classiﬁcation, the input text is truncated at a ﬁxed length (256, 512, etc) due to limits in the size of the model.
This corresponds to our basic model:
[Equation]
where W ∈ R k × d , b ∈ R k , and we assume the segment length of x 1 equals to the input size limit of the language model.
This model is depicted in Figure 1(left).
Next we describe the three progressive enhancements to the base LM, shown in Figure 2.
Concatenated Language Model (Cat-LM)
The ﬁrst enhancement is a natural extension to improve the basic model by including information from more segments x 2 , . . .
, x m .
A very simple way to do this is to concatenate the representation z 1 , z 2 , . .
z m before the classiﬁcation layer.
This leads to the model:
[Equation]
where W ∈ R k × md , b ∈ R k .
This model is difﬁcult to perform backpropagation directly because we cannot hold m copies of the LM parameters in memory at the same time.
We solve this problem by stopping the backpropgation paths of some of the segments, namely z 2 , . . .
, z m .
Section 3. discusses this approximation.
RNN-augmented Language Model (RNN-LM)
The third model we want to consider is one that summarizes the information from z 1 , . . .
, z m using a bidirectional LSTM (Hochreiter & Schmidhuber, 1997).
Let ( h 1 , h 2 , . . .
, h m )
=
biLSTM(z 1 , . .
. , z m ) be the q dimensional hidden state representations from a bidirectional LSTM, where h i ∈ R q .
The biLSTM-based model can be written as:
[Equation]
where W ∈ R k × ( d +2 q ) , b ∈ R k .
For this model we also stop the gradient computation at z 2 , . . .
, z m , and do not backpropagate beyond the LSTM parameters.
We do not include z 2 , . .
z m as input to the classiﬁer as the size of W and b scale directly with m , and with a large m the number of parameters would grow counter-productively.
However, we still include z 1 as the text from that segment typically contains the most discriminative features.
By including z 1 there is a direct path for gradient updates to the language model.
Additionally, we found that including h 1 , instead of h 2 , in the classiﬁer consistently produced worse results.
We believe this is from the forward direction of h 1 not providing any contextual information.
Comparatively, the backwards direction of h 2 provides enough context that when concatenated with z 1 achieve the best performance.
Attention-based Language Model (ATT-LM)
For our attention based model, we utilize the structure of RNN-LM and add the attention mechanism as described in Yang et al.
We deﬁne our attention variables as follows:
[Equation]
where W s ∈ R 2 q × q , b s ∈ R q , u s ∈ R q are learned attention parameters for attention over segments.
Let M ( α ) =
arg max 1 ≤ i ≤ m α
i be the index of the segment that gives the highest attention weight.
These deﬁnitions give us our equation for the attention-based model by concatenating a set of relevant features:
[Equation]
where W ∈ R k × (2 d +3 q + m ) , b ∈ R k are the parameters of the linear classiﬁer.
For this model, we stop the gradient computations paths for all z
i ’s apart from z 1 and the selected z M ( α ) .
For all four methods, we let n be the number of training documents and use the negative log likelihood as the loss function:
[Equation]
3. P ARAMETER U PDATES
We run mini-batch stochastic gradient descent for parameter updates.
We treat the parameters of the language models and the other parameters (weight matrix W for classiﬁcation, LSTM parameters, etc) differently.
We perform full gradient computation on non-LM parameters, and only approximate gradient computation for the LM’s parameters by stopping backpropagation on selected segments (see the model description above and the Appendix Section A).
E XPERIMENTS
Implementation Details.
We use PyTorch (Paszke et al., 2017) to conduct all our experiments.
For our pretrained language models, we use the HuggingFace (2019) implementation of the BaseBERT model (110M parameters) and use the GPT model of similar size (117M parameters).
Due to computational constraints, we do not use BERT-Large or larger GPT models.
We use BERT’s lower-case tokenizer and GPT’s tokenizer with an added classiﬁcation token.
Both models take a ﬁxed input size of 256 tokens, contain 12 transformer blocks, and have a hidden size of 768 neurons.
We use an bidirectional LSTM for our recurrent neural network in the recurrent and attention based methods.
We apply dropout (Srivastava et al., 2014)
( p = .
1 ) before the ﬁnal linear layer.
Unless otherwise noted we use a learning rate of 2 e − 5 , use a scheduled ADAM optimizer (Kingma & Ba, 2015), train for 3 epochs over each training dataset, use a training mini-batch size ofdocuments, and set all other hyper parameters to their default values.
While the number of segments m does not need to be ﬁxed, for ease of processing we set m = 8 for the patent datasets as patents have a relatively constrained length and 8 allows for minimal padding to be used.
We also use m = 16 segments for the Arxiv dataset as one document typically contains 6k words, many of which are removed as non-meaningful or are intentionally truncated as part of the bibliography.
Next, we describe the different datasets we use in our experiments.
[Table]
PatentBert
[Table]
Local
[Table]
Patents.
Patents can be broken down into multiple levels of resolution according to the International Patent Classiﬁcation System (IPC): Section, Class, Subclass .
The most broad category,
Section , has eight labels (A-H).
For example, Section A is concerned with Human Necessities , while
Section H is concerned with Electricity .
We also perform classiﬁcation experiments on a more detailed level of categorization: Subclass , which contains 638 labels.
Patents were gathered from the
Google Patents Public Dataset via SQL queries.
We gathered all documents from the United States Patent Ofﬁce (USPTO) from 2006-2014 as our training set and use all patents from 2015 as our test set.
We have 1,917,334 training and 296, testing documents, where 15,172 and 1,835 documents were respectively skipped for missing abstracts.
The text of a patent is composed of different parts: title, abstract, and a list of claims.
For our purposes, we consider one patent to be ﬁrst the title, then the abstract, followed by each claim in order– claim 1, claim 2, ... until the last claim.
(Inverted) Wireless Patents.
We selected a subset of patent data to perform additional experiments.
We chose the wireless (H04) Class due to its large number of training and test examples (the second most of all the Class data).
The Class with the most examples (computing) was not chosen, due to its large imbalance in Subclasses (dominated by one label with over 75% examples).
Additionally, we use the wireless Class to construct an inverted patent dataset, where a single patent starts with its last claim, up to the ﬁrst claim in the reverse order, then the abstract, and lastly the title.
It is commonly believed that the abstract and the ﬁrst claim is the most useful in classifying a patent.
We create this dataset to present information in reverse order of relevance to test models that bias towards the beginning of documents (e.g., models that truncate beyond a ﬁxed number of tokens).
After processing, the wireless Class contains 250,982 training and 42,892 testing documents, where 15,172 and 97 documents were skipped, respectively, for missing abstracts.
Arxiv papers.
We study the long document dataset provided by He et al.
It consists of 33,388 papers, from 11 different categories, downloaded from the scientiﬁc article hosting website
Arxiv.
The least occurring category is “math.
AC” with 2885 documents, and the most occurring is ”cs.
DS” with 4136 documents.
We call this dataset Arxiv-11.
They also provide a subset of the data using four categories and 12,195 documents, which we refer to as Arxiv-4.
All downloaded pdf documents were converted to txt ﬁles, with no document less than 1,000 words.
We randomly sample 90% for training and use the remaining 10% for test.
R ESULTS
Patents and Arxiv Datasets.
We report our main results in Table 1, where the numbers are microF1 scores.
We make three observations.
First, we compare our methods to the previous work using DeepPatent (Li et al., 2018), PatentBert (Lee & Hsiang, 2019) and Local Word Glimpses (He
[Table]
et al., 2019).
Table 1 shows the Base LMs perform well, and we note our Base-BERT our performs
PatentBert as they do not combine the title, abstract, and claim– but only the title and abstract or just the ﬁrst claim.
Second, we compare Base LMs against the three enhancements acrossvariants of the patent dataset and 2 variants of the Arxiv dataset.
The RNN-based and attentionbased LMs show consistent improvement over base LMs, while the simple concatenation-based is not consistent.
Furthermore, the usage of attention is superior in all cases (highlighted in the last line).
Lastly, it is also interesting to note that BERT-based models consistently outperform GPTbased models.
This is likely due to the use of bidirectional contexts in BERT.
Inverted Patents.
As shown in recent work (Lee & Hsiang, 2019; Li et al., 2018), only the abstract or ﬁrst claim on a patent is needed for good classiﬁcation performance.
To analyze the effect of attention discovering the location of discriminative content, we invert the structure of the patents in our wireless dataset and train new models.
Comparing the results from the second last column (wireless) and last column (inverted wireless) in Table 1, we see that Base LMs, which only take into account of ﬁrst 256 tokens, suffer from a drop of F1 scores of more than 2 . 0 .
Cat-LMs and RNN-LMs reduce the gap in F1 scores to about 1 . 0 , while the attention-based models perform the best, with the gap between wireless and inverted wireless reducing to smaller than 0 .
Exploration of Hyper-parameters.
Next, we investigate the effects of different hyper-parameters on the various models using a small set of training data.
For this analysis, we use the Arxiv-4 dataset.
While the attention based models seem to do well, table 2 demonstrates the unpredictable nature of using different learning rates along with different training epochs.
This behavior aligns with the claims of Devlin et al.
(2018), who also found ﬁne-tuning on small datasets sometimes lead to unstable results.
Training and Evaluation Time.
Lastly, we compare the training time and evaluation speed of our models on the Arxiv-11 dataset.
As shown in Table 3, due to the multiple forward passes, all of the enhanced variants of the LMs require nearly 3 - 4 x training time and over 2 x to evaluate.
Considering there are substantially more operations required to process the full input text, this slow down is better than expected.
The difference in training time between the attention-based models and non-attention-based is surprising, given the fact that the LM’s parameters must be updated with more than one set of gradients.
This points towards further gradient computations being feasible for architectures, and hardware, that can handle the additional required memory
[Table] [Table]
5. A BLATION E XPERIMENT
In order to understand the effects of the attention, we introduce an ablated model that allows us to investigate how attention inﬂuences and guides back-propagation.
The model we use is similar to
ATT-BERT, but we remove LM ( x 1 ) , h 2 , and h m from the classiﬁer.
Therefore it can be written as:
[Equation]
with W ∈ R k × ( q + m + d ) , b ∈ R k being the parameters of the linear classiﬁer.
The full results of this model can be seen in table 5 in the appendix.
But here, we use this model to carry out two additional experiments.
Shufﬂing Experiment.
First, we perform a shufﬂed input experiment to examine the effect of using attention to guide language model parameter updates.
We use all the same setup as the experiments for the Arxiv-11 column in Table 1, except for each training iteration, we randomly permute each segment x 1 between all mini-batch examples.
This means methods that use x 1 will no longer be able to rely upon those gradients to give an informative update to the model’s parameters.
Table 4 shows how the non-attention based methods guess the maximum occurring class when gradients are updated using x 1 .
And it also shows how these methods perform poorly without gradients to update the language model parameters.
On the other hand, the attention-based methods are able to perform well despite this loss of information, and ATT-BERT is relatively unaffected by the loss of gradients from x 1 . [Figure] a [Figure]
Attention α Comparisons.
Second, we measure and compare the α values on the wireless and inverted wireless patent dataset.
We average the α value for each of the eight input segments over the entire test set.
Figure 3 (a) comparatively shows the α values of our attention based model on the wireless and inverted wireless dataset; this clearly demonstrates how the attention mechanism is able to accurately pick up on the important sections of text.
Figure 3 (b) shows a comparison between ATT-BERT and the ablated model on both the wireless and inverted wireless datasets.
The α values for both models on the inverted dataset seem relatively similar, with the ablated model placing heavier emphasis on the ﬁrst few segments.
However, the effect of including the gradients of x 1 can been seen on the wireless dataset, where the α 1 value is over 0 .
7 on the ablated model.
This means enabling backpropagation to occur for input x 1 has a positive effect for ATT-BERT and, moreover, that the title and abstract in a patent are of high discriminative importance.
C ONCLUSIONS
In this work, we achieve state of the art results on multiple long document classiﬁcation tasks by utilizing pretrained language models with attention based enhancements.
With language modeling continuing to see improvements every month, we show how different models can be utilized with our method.
We performed numerous experiments to clearly demonstrate the value added by using attention to learn from important segments.
Furthermore, we showed that the additional gradient computation as a result of attention is marginal when compared to the consistent improvement of results; and we analyzed the effects of the attention mechanism through the loss of input information via both shufﬂing and a carefully constructed dataset augmentation.
A G RADIENT C OMPUTATION FOR L ANGUAGE M ODEL P ARAMETERS
Let us denote our language model representation be z
i
=
LM( x i ) =
f θ ( x i ) , where θ are the language model parameters.
Let h φ be the function we compute on top of the LM representations, e.g., the classiﬁer, LSTM, etc.
The parameter φ can contain the classiﬁcation weights W and the
LSTM weights.
The models considered in this paper can be written as:
[Equation]
Coupled with the loss function l (log loss) and the target label y , we have
[Equation]
Computing the gradient over the LM parameters θ , by chain rule we have ∂
[Equation]
By stopping the gradient computation over x 2 , . . .
, x m , we are dropping the terms related to ∂∂ θ f θ ( x i ) , i ≥ 2 , from the above formula.
In optimization, we say q is a descent direction if (cid:104) q, ∂ θ l (cid:105) < 0 .
The negative gradient − ∂ θ l is clearly a descent direction.
The gradient above in
Equation (8) is in the form ∂ θ l = c (cid:80) mi =1 g i , where g i are the gradient term of the i th segment in the equation.
By dropping terms in the backpropagation, we are assuming either the contribution from g 1 (or g M ( α ) in the attention model of Equation 5) dominates the contributions from other segments, or the gradients g i from different segments point towards similar directions, so that the truncated gradient is still a descent direction.
In some cases, if the storage of partial gradients ∂∂ u h φ ( z 1 , . . . , u , . . .
, z m ) is feasible, it is possible to compute the full gradient in two passes over the mini-batch.
For example, in the basic Cat-BERT model for classiﬁcation, this partial gradient is W i scaled by the derivative of softmax, where i is the index of the segment.
We can compute these W i with the derivative of the softmax as scaling factors in a ﬁrst pass, and accumulate gradients over f θ ( x i ) scaled by these factors in a second pass over the mini-batch.