L ANGUAGE M ODEL P RE - TRAINING FOR H IERARCHICAL D OCUMENT R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Hierarchical neural architectures can efﬁciently capture long-distance dependencies and have been used for many document-level tasks such as summarization, document segmentation, and sentiment analysis.
However, effective usage of such a large context can be difﬁcult to learn, especially in the case where there is limited labeled data available.
Building on the recent success of language model pretraining methods for learning ﬂat representations of text, we propose algorithms for pre-training hierarchical document representations from unlabeled data.
Unlike prior work, which has focused on pre-training contextual token representations or context-independent sentence/paragraph representations, our hierarchical document representations include ﬁxed-length sentence/paragraph representations which integrate contextual information from the entire documents.
Experiments on document segmentation, document-level question answering, and extractive document summarization demonstrate the effectiveness of the proposed pre-training algorithms.
I NTRODUCTION
While many natural language processing (NLP) tasks have been posed as isolated prediction problems with limited context (often a single sentence or paragraph), this does not reﬂect how humans understand natural language.
When reading text, humans are sensitive to much more context, such as the rest of the document or even other relevant documents.
In this paper, we focus on tasks that require document-level understanding.
We build upon two existing separate lines of work that are useful for these tasks: (1) document-level models with hierarchical architectures, which include sentence representations contextualized with respect to entire documents (Ruder et al., 2016; Cheng &
Lapata, 2016; Koshorek et al., 2018; Yang et al., 2016), and (2) contextual representation learning with language-model pretraining (Peters et al., 2018; Radford et al., 2018).
In this work, we show for the ﬁrst time that these two ideas can be successfully combined.
We ﬁnd that pretraining hierarchical document representations raises several unique challenges compared to existing representation learning.
First, it is not clear how to pretrain the higher level representations.
Using existing techniques, it is possible to initialize the lower level of the hierarchical representation, to represent words in the context of individual sentences e.g. using ELMo (Peters et al., 2018), or, in addition, initialize sentence vectors that are not dependent on the context from the full document using Skip-Thought vectors (Kiros et al., 2015), but it is not possible to initialize the full hierarchical neural structure.
Second, bidirectional representations are standard in state-of-the-art NLP tasks (Wu et al., 2016; Weissenborn et al., 2017), and may be particularly important for document-level representations due to the long-distance dependencies.
However, current pretraining algorithms often use left-to-right and right-to-left language models to pretrain the representations.
This restricts the expressiveness of the hierarchical representations, as left and right contextual information do not fuse together when forming higher-level hierarchical representations.
In this paper, we address these challenges by proposing two novel approaches to pre-train documentlevel hierarchical representations.
Both methods pre-train representations from unlabeled documents containing thousands of tokens.
Our ﬁrst approach generalizes (Peters et al., 2018) to pre-train hierarchical left-to-right and right-to-left document representations.
To allow the hierarchical representations to learn to fuse left and right contextual information from abundant unlabeled text, our second approach is a new pretraining algorithm called masked language model , which allows efﬁcient training of bidirectional hierarchical document representations.
We evaluate the impact of the novel aspects of our pre-trained representations on three tasks: document segmentation, answer passage retrieval for document-level question answering, and extractive text summarization.
We ﬁrst pretrain document-level representations on unlabeled documents and then ﬁne-tune with task-speciﬁc labeled data and a light-weight task network.
Experiments show that pre-training hierarchical document representations brings signiﬁcant beneﬁts on all tasks, and that pretraining the higher level of the document representation results in larger improvements than pre-training only the locally contextualized lower word level in most tasks.
On the CNN/Daily Mail summarization task, we obtain strong results without using reinforcement learning methods employed by the previous state-of-the-art models. On the TriviaQA answer passage retrieval task, our model improves over a strong baseline (Clark & Gardner, 2018) by 6% absolute.
B ACKGROUND
In this section, we review pretraining of contextual token representations using language models and discuss how the language model probability decomposition imposes constraints on the architectures.
Language Model Pretraining Given a sequence of tokens ( x 1 , . . .
, x n ) , a representation module V θ encodes each word into a vector:
V θ ( x 1 , . . .
, x n )
= ( v 1 , . .
. , v n ) , where each word-level contextual representation v i could potentially depend on the whole sequence.
Common choices for the representation modules are LSTMs, CNNs, and self-attention architectures (Hochreiter &
Schmidhuber, 1997; Vaswani et al., 2017; LeCun & Bengio, 1998).
A representation module is parameterized by an underlying neural architecture with parameters θ .
The typical left-to-right language model aims to maximize the probability of observed sequences:
[Equation]
The pretrained parameters θ can then be used to initialize the model for a downstream task.
Uni-directionality constraint The formulation of the language model implicitly imposes a directionality constraint on the V θ , which we term uni-directionality constraint .
In Equation 1, the word x t can only depend on the (representations of) the previous words.
Therefore, the contextual representations v 1 , . .
. , v t − 1 can not depend on the representations of x t , . . .
, x n , resulting in the uni-directionality constraint on the representations.
One common type of representation modules that satisfy the uni-directionality constraint are unidirectional recurrent neural networks (RNNs).
Given a sequence of words, a left-to-right LSTM model generates a sequence of representations V θ ( x 1 , . . .
, x n )
= ( −→ h 1 , . .
, −→ h n ) that satisﬁes the uni-directionality constraints, where −→ h t only depends on −→ h 1 , −→ h 2 , . . .
−→ h t − 1 .
Modules that satisfy the uni-directionality constraint can also be derived from self-attention, convolution, and feedforward architectures (e.g. (Radford et al., 2018)).
As mentioned in the introduction, for many downstream tasks, bidirectional representations bring large improvements over uni-directional ones.
Therefore, Peters et al.
(2018), pre-trained two encoders using two language models: a left-to-right and a right-to-left model, using a left-to-right and a right-to-left LSTM encoder, respectively.
The token representations are formed by the concatenation of outputs from the two uni-directional encoders.
We term this style of language model pre-training
L + R -LM.
More speciﬁcally, the representation of the t -th word becomes v t = (cid:104) −→ h t ←− h t (cid:105) , where ←− h t is generated from a right-to-left LSTM.
Tasks using these input representations must learn to merge the left
The general strategy we adopt in this paper is to pretrain the hierarchical representations on unlabeled data and then to ﬁne-tune the representations with the task-speciﬁc labeled data.
Therefore, the uni-directionality constraint will also impacts the task-speciﬁc networks.
I-th block
Lo c a l L S T M x 1 i C ha r CNN
x
2 i x 3 i x T i
h
i h 
 i h
i h T i x 1 i+1 C ha r CNN h
i+1 h
i+1 h 3 i+ h T i+ i+1 block x 2 i+ x 3 i+ x T i+
Lo c a l L S T M
Lo c a l L S T M Lo c a l L S T M
Pool + Project Pool + Project
GlobalLSTM p i p i+ [Figure]
I-th [Figure]
Lo c a l L S T M x 1 i C ha r CNN
x
2 i x 3 i x T i
h
i h 
 i h
i h T i x 1 i+1 C ha r CNN h
i+1 h
i+1 h 3 i+ h T i+ i+1 block x 2 i+ x 3 i+ x T i+
Lo c a l L S T M
Lo c a l L S T M Lo c a l L S T M
Pool + Project Pool + Project
GlobalLSTM p i p i+
[Figure]
H IERARCHICAL D OCUMENT -L EVEL R EPRESENTATIONS
As our goal is to learn ﬁxed-length contextualized representations of text segments to be used in document-level tasks, we choose hierarchical neural architectures that build representations of tokens and text segments in context.
Here we describe the speciﬁc architectures studied in this paper.
The novel contribution of our work – pre-training such representations from unlabeled text, will be detailed in Section 4.
We choose LSTM-based architectures with a standard hierarchical structure that has been useful for capturing long-term context in document-level tasks (Serban et al., 2016; Koshorek et al., 2018).
We experiment with two related document-level representations.
The ﬁrst one, called BI -HLSTM, is a standard hierarchical bidirectional LSTM encoder.
BI -HLSTM does not satisfy the uni-directionality constraint and cannot be pre-trained using a left-to-right language model likelihood objective function.
Our second document-level representation is L + R -HLSTM, which consists of two concatenated uni-directional versions of BI -HLSTM, and can be seen as a hierarchical document-level extension of ELMo (Peters et al., 2018).
Both document-level representations view documents as sequences of text blocks, each consisting of a sequence of tokens.
The text blocks can be sentences, paragraphs, or sections, depending on the task.
Here we experiment with sentences and paragraphs as text blocks.
The BI -HLSTM architecture fuses left and right contextual information more effectively and directly corresponds to state-of-the-art architectures used for multiple NLP tasks.
BI -HLSTM
BI -HLSTM has a two-level hierarchy.
Every block is ﬁrst independently processed by a bidirectional LSTM (Bi-LSTM) to encode the local context.
The contextual representations of the words in each block are then pooled and projected to a single vector to represent the block.
The vector representations of the sequence of blocks in the document are then processed by a global BiLSTM, resulting in block representations contextualized by the entire document.
The overall design is shown in Figure 1a, and the encoding procedure is formalized next.
A document D consists of K blocks D =
{ U 1 , U 2 , . . .
U K } , and each block consists of a sequence of tokens where U
i =
{ x i 1 , x i 2 , . . .
x iT } .
For a block
U i , we start with character representations of the words, where ¯ x it =
CNN char ( x it ) .
Then character-based word representations are processed by the local Bi-LSTM, where { h i 1 , . . .
, h iT } =
B I -LSTM Local ( { ¯ x i 1 , . . .
, ¯ x iT } ) .
Before generating the globally-contextualized representations, the encoder generates representations of all blocks by pooling the locally-contextualized word representations generated from the local encoder.
The block representations are obtained using both max and average-pooling on the word representations, followed by a feed-forward transformation:
[Equation]
The resulting local block representations c
i are then processed by the global Bi-LSTM to generate document-contextualized block representations p
[Equation]
: { p
[Equation]
, p K
[Equation]
= [Equation]
I -LSTM Global ( { c 1 , . . .
, c K } ) .
In downstream tasks, this document representation is used by inputting the text block contextual vector representations { p i } to light-weight task-speciﬁc networks.
L + R -HLSTM
As previously mentioned, when using left-to-right language models to pre-train representations, the representation of the i -th word can only depend on the representations of the previous words ( i − 1) words.
Similarly, the representation of the j -th text block can only depend on the representations of the previous text blocks.
To design a document-level representation similar to the one deﬁned by the BI -HLSTM
but that can still be trained using a uni-directional language model decomposition, we replace all of the bi-directional LSTMs in BI -HLSTM with either left-to-right or right-to-left LSTMs.
The left-to-right version of the resulting hierarchical encoder is presented in
Figure 1b.
Given a document D
= { U 1 , U 2 , . . .
U K } , let the encoding results of the left-to-right HLSTM be denoted as {−→ p 1 , . .
, −→ p K } =
L- TO -R-HLSTM ( D ; θ l ) .
We use another right-to-left hierarchical encoder to produce representations: {←− p 1 , . . .
, ←− p K } = R- TO -L-HLSTM ( D ; θ r ) .
We then concatenate the ﬁnal representations of the two encoders: ˆ p i = (cid:2) −→ p i
←− p
i (cid:3) .
We term the resulting document-level encoder L + R -HLSTM.
Note that the L + R -HLSTM combines the left and right information only at the top level, and there is no interaction between the two unidirectional contextualizers.
While at the bottom-most layer, the BI -HLSTM similarly concatenates left and right information, the second local BI -LSTM layer can fuse the two directions.
Similarly, the left and right contexts at the text segment level are fused at the second global BI -LSTM layer.
In principle, both BI -HLSTM and L + R -HLSTM can capture long distance dependencies via their hierarchical structure, but it is unclear whether such dependencies can be effectively learned from limited amounts of supervised data labeled for down-stream tasks of interest.
In the next section, we discuss how such document encoders can be trained to provide rich text segment and token representations using large unlabeled document collections.
P RE - TRAINING H IERARCHICAL D OCUMENT -L EVEL R EPRESENTATIONS
Here we propose language model-based methods for pre-training hierarchical document-level representations.
In order to train the hierarchical representations with language models, our main idea is conceptually simple: we combine sentence representations contextualized with respect to the whole document and the token representations with respect to the sentence to perform missing word predictions.
Intuitively, from the loss of word prediction, both the contextual sentence and token representations will be updated.
Note that there are interactions between the contextual sentence and token representations, as part of the contextual sentence representations are constructed from token representations.
Therefore, the language model pretraining algorithms needed to be designed carefully.
In Section 4.1, we present ways to pre-train uni-directional hierchical document-level representations.
In Section 4.2, we propose a language prediction model that can pre-train bi-directional representations.
4. U NI - DIRECTIONAL R EPRESENTATIONS
Standard language model pre-training ( L + R -LM) can be used to pre-train the uni-directional tokenlevel representations in local text block context.
Starting from Equation 1, and using a standard
LSTM language model, we can deﬁne the predictive probability of the next word given prior text block-internal context as: P local L -LM ( x it | x i t − 1 , . . .
x i 1 ) =
P ( x it | −→ h
i t − 1 ) , and train the representations us- ing this language model.
As in Peters et al.
(2018), this only pre-trains token representations in local context and does not provide a way to derive contextualized single vector text block representations.
Here we propose to use hierarchical language models to pre-train uni-directional hierarchical document-level representations.
We refer to this algorithm as L + R -LM global .
When pre-training with hierarchical language models, we treat the uni-directional document-level representation in Figure 1b as the representation used by a hierarchical language model and use entire unlabeled documents to pre-train it.
The hierarchical language model predicts the next word in a document using the contextual encoder representations.
The left-to-right hierarchical model can use the contextual information from previous tokens in the same document to predict the next word.
The previous tokens with respect to a current token can be categorized into two sets: the preceding tokens in the same text block, and all tokens in the preceding text blocks.
More speciﬁcally, the predictive probability for the next token is deﬁned as: P [Equation] , U [Equation]
p
[Equation]
Therefore, the objective function for L + R -LM global is as follows:
[Equation]
where we use a feed-forward network to combine the information from ( −→ h it − 1 , −→ p i − 1 ) and ←− h it +1 , ←−
p i +1 , respectively.
These feed-forward networks are not used in down-stream tasks.
4. B I - DIRECTIONAL R EPRESENTATIONS
We have shown a method to pre-train uni-directional encoders with hierarchical language models.
However, it is still not clear how we can pre-train representations that do not satisfy the uni-directionality constraint from unlabeled text.
Here we present text prediction models we term masked language models ( MASK -LM), which alleviate the architectural constraints imposed by standard language model pre-training.
They can be seen as a special form of denoising auto-encoders (Vincent et al., 2008), where the decoder network has minimal parameters, the only type of noise is word masking, and the decoder only makes predictions for positions with noise (from the terminology in that paper, full emphasis is on reconstructing the corrupted dimensions).
[Figure]
Figure 2 demonstrates the method of training representations with MASK -LM.
For a given document, we ﬁrst generate a set of random indices and mask out (hide) the words at the corresponding positions in the document.
After a document has been masked, we provide it as input to a document-level encoder to produce contextual word and text block representations.
These contextual representations are used to predict the words at the masked positions.
More formally, let Z denote a set of indices for the masked words.
Let ζ ( D ; Z ) =
{ ζ ( x 11 ) , ζ ( x 12 ) , . . .
, ζ ( x it ) , . . .
ζ ( x KT ) } indicate the masked word sequence, where:
[Equation]
Note that is a special symbol to indicate that the word is masked out (hidden).
Given that Z is generated randomly, masked language models essentially optimize the following objective function where θ represent the parameters of the representation:
[Equation]
Unlike the auto-encoder method of Hill et al.
(2016), in MASK
-LM we assume the probabilities of all masked words in a sequence can be predicted independently and do not employ a separate auto-regressive decoder network.
There are two ways to apply MASK -LM to pre-train parts of the BI -HLSTM.
Analogously to the local setting of L + R -LM, we could pre-train with MASK -LM using local context.
We deﬁne the probabilities of masked words to depend only on the locally contextualized token representations, and apply MASK -LM to only pre-train the local bi-LSTM by using: P local MASK -LM
( x it | ζ ( D ; Z ); θ ) =
P ( x it | h it ; θ ) , where the h it is the word-level contextual representation for masked word x it .
Note that if ( i, t ) ∈
Z , ζ ( x it )
= .
Therefore, h it , which is generated by the bi-directional local LSTM encoder, needs to carry the contextual information in order to recover the masked word x it .
Similarly, in the MASK -LM global setting, the predictive probabilities are deﬁned as:
P global MASK -LM ( x it | ζ ( D ; Z ); θ ) = P ( x it | h
it , p i ; θ ) , where we use another feed-forward network to combine the information from h it and p i .
D OWN - STREAM T ASKS
We perform experiments on three document-level downstream tasks, which require predictions at the level of text blocks in full documents.
The predictions are made using light-weight task speciﬁc networks which take as input single-vector document-contextualized text block representations p i , generated by the hierarchical document encoders described in Section 3.
We brieﬂy deﬁne the downstream tasks and the task-speciﬁc architectures used.
We provide additional details in the
Experiments section.
Document Segmentation In the document segmentation task, we take as input a document represented as a sequence of sentences.
The goal is to predict, for each sentence, whether it is the last sentence of a group of sentences on the same topic, thereby segmenting the input document in topical segments.
We use a feed-forward network with one hidden layer with RELU activations on top of sentence representations p i to deﬁne scores of the task speciﬁc labels (segment boundary or not).
Answer Passage Retrieval Given a document and a question, the task of answer passage retrieval is to select passages from the document that answer the question.
While most QA systems focus on predicting short answer spans given a relevant answer passage, answer passage retrieval is a prerequisite task for open-domain question answering.
While there are a multitude of architectures possible for this task, we propose a modular approach that enables extremely fast answer passage retrieval during serving time.
Our model works as follows.
The given document is encoded and its contextual passage representations p i are generated.
The questions are treated as documents consisting of a single sentence and are similarly encoded in representations q .
Inspired by Peters et al.
(2018), instead of using the representations p
i directly, we use a linear combination of the layers of p i with positive weights summing to 1: p (cid:48)
i =
(cid:80) l s l p
i,l , where s l is a task-speciﬁc learned scalar weight for the l -th layer, and p i,l represents the l -th layer of the representations of the i -th block.
To score a passage-question pair, we use the following architecture using a learned task-speciﬁc network combining the following features: s 0 ( p i , q ) =
(cid:104) p (cid:48)
i , q (cid:48) (cid:105) is a dot product of the passage and question representations, s 1 , . . .
, s 5 are ﬁve light-weight features such as paragraph position and tf-idf borrowed from Clark & Gardner (2018).
These features are sent to a feed-forward network with one hidden layer with RELU activations to generate the ﬁnal score.
Extractive Summarization Given a document, the task of extractive summarization is to select representative sentences to form a summary for the whole document (Kupiec et al., 1995).
Our summarization model is quite similar to our answer passage selection model with two key differences.
First, we add another randomly initialized LSTM on top of all sentence representations to form reﬁned sentence representations.
We found that the extra layer generally improves results.
Second, we perform max-pooling over the contextual representations of all sentences p (cid:48)
i to form a ﬁxed-length vector representation d for the entire document.
The representation d is then treated like the ”question” representation in the passage selection task.
Third, instead of using a dot-product as the scoring function, for each sentence, we concatenate d and p (cid:48)
i to form a combined vector which is sent to a feed-forward network with one hidden layer with RELU activations to generate the ﬁnal score.
As for the Answer Passage Retrieval task, we add several task-speciﬁc features such as the position of the sentence in the document and the number of words in the sentence.
E XPERIMENTS
The main goal of our experiments is to examine the value of unsupervised pre-training of hierarchical document-level representations, focusing on the novel aspects of our methods.
We ﬁrst present the experimental settings and pre-training details in Section 6.1.
We then present and analyze the experimental results on the three tasks.
Among the pre-training methods, L + R -LM global , MASK -LM and MASK -LM global are proposed in this paper.
The local version of L + R -LM is similar to ELMo (Peters et al., 2017), but its effectiveness when used to initialize document-level hierarchical representations has not been studied before.
6. S ETTINGS
We use standard LSTM cells for our sequence encoders.
For word embeddings, we use characterbased convolutions similarly to (Peters et al., 2018), and project the embeddings to 512 dimensions.
Our local encoder is a 2-layer LSTM with 1024 hidden units in each direction.
Our global encoder is another 2-layer LSTM with 1024 hidden units.
Since L + R -LM requires the representation to satisfy the uni-directionality constraint, while there are no directionality constraints on the representations when pre-training with MASK -LM, in the experiments we always pair each encoder with its natural pre-training method: we pre-train BI -HLSTM with MASK -LM, and L + R -LM with L + R -HLSTM.
For each of the two encoders, we compare no pre-training on unlabeled data, versus pre-training at the local sentence level only, versus pre-training at the global document level.
If we initialize the BI -HLSTM representation with locally pre-trained MASK -LM, this means that we initialize the local Bi-LSTM and learn the global Bi-LSTM only based on the labeled down-stream task data.
If we initialize both the local and global Bi-LSTMs of BI -HLSTM with MASK -LM global pre-trained models, only the task-speciﬁc feed-forward network is randomly initialized, and all parameters are ﬁne-tuned with the labeled set.
When comparing hierarchical (global) and nonhierarchical (local) pretraining methods, we use the same document-level representation architecture for the downstream tasks, pre-train on the same amount of unlabeled text, and then ﬁne-tune with the same task-speciﬁc network architectures.
In all tasks, we also included comparisons to skip-thought vectors (Kiros et al., 2015), where the model is learned to construct sentence embeddings.
Additionally, we compare to EML O pool , which forms a representation for a text block by pooling the token-level representations generated by the model pretrained in (Peters et al., 2018).
Perone et al.
(2018) have shown that EML O pool produces strong sentence representations and outperforms many other sentence representation models including skip-thought vectors.
4 At a high level, the contextualized token representations obtained by our local L + R -LM implementation are similar to the ones obtained by ELMo.
However, ELMo was trained on a larger dataset and uses higher-dimensional hidden vector representations. To pre-train the hierarchical representations, we use documents in Wikipedia and ﬁlter out ones with fewer than three sections or less than 800 tokens.
We sample from the ﬁltered document set and form a collection containing approximately 320k documents with about 0.9 billion tokens.
We select the most frequent 250k words as the vocabulary for all models.
We limit documents to at most 75 sentences and limit each sentence length to 75 tokens.
Sentences are used as text blocks for pretraining and pre-trained models are applied to both sentence-block and paragraph-block documents in the down-stream tasks.
To capture the full hierarchical information, we did not use truncated back-propagation-through-time (tBPTT) (Rumelhart et al., 1985), but used full back propagation on documents with up to 5k tokens.
For MASK -LM, we randomly mask 20% of the words in the documents for training.
All of the down-stream task experiments are performed using the same set of pre-trained models.
6. D OCUMENT S EGMENTATION
We create labeled data for the segmentation task by taking Wikipedia articles and forming the labels using the section information.
Given a sentence in a Wikipedia article, the label of the sentence
[Table]
is positive if it is the last sentence of a Wikipedia section.
To ensure fair comparisons, the section boundary information is never used during pre-training.
We select 5k documents each for training, development, and test sets.
In our dataset, only 5% of the sentences are positively labeled; we use F score as the evaluation metric due to this imbalanced class distribution.
All reported experimental results for the segmentation task are averages of ﬁve different training runs on the task-speciﬁc labeled data.
Results
The experimental results for the segmentation experiments are in Table 1a.
Our baseline system that does not use any pretraining of representations is similar to the one proposed in Koshorek et al.
(2018) with the difference that our system uses character information to generate the word embeddings.
Note that pretaining results in substantial improvements over the baseline systems in all settings.
We next compare pre-training with hierarchical language models (the global setting) versus pre-training with non-hierarchical language models (the local setting).
Global pre-training of the hierarchical document-level representations improves upon local pre-training of the sentence-contextualized token representations.
In the case of the L + R -HLSTM encoder paired with L + R -LM global , the improvement is over 6% F1.
For this task, we found that L + R -LM global is much better than MASK -LM global .
We hypothesize that by predicting words in the next sentence, the L + R -LM global model is more sensitive to the topical changes between sentences.
The Table also shows results of experiments with a setting where we applied another Bi-LSTM on top of the
EML O pool and skip-thought vectors; as seen both methods perform worse than our approaches.
6. A NSWER P ASSAGE R ETRIEVAL
We apply the document-level encoders to the task of answer passage retrieval in the next set of experiments.
To evaluate the impact of pre-training a document-level encoder for this task, we use the Wikipedia subset of the TriviaQA dataset (Joshi et al., 2017), a large scale distantly supervised question answering dataset.
While TriviaQA was originally designed for short answer detection, we use it to evaluate the answer-passage selection task, as it takes the full documents as input and requires retrieving passages.
We use the paragraph breaker implementation used in Clark & Gardner (2018).
As in other work on TriviaQA, we set 400 to be the maximum number of tokens in a paragraph during training.
We also use 400 tokens in testing.
We report the top-1 passage selection accuracy on the development set as our main metric.
As the TriviaQA test data labels are not publicly available, we cannot obtain passage accuracy test metrics and thus do not report these.
The passage retrieval results are in Table 1b.
Note that pre-training leads to substantial improvements in all settings.
For both MASK -LM and L + R -LM, global pre-training of the full hierarchical representations improves upon local pre-training of the lower-level contextual token representations.
Our models improve upon the passage selection model proposed by (Clark & Gardner, 2018) by more than 6% absolute.
The latter work focuses on evaluation of their end-to-end short answer selection [Table] L + R [Table]
HLSTM
[Table] [Table] [Table] [Table]
[Table]
system and does not report passage retrieval performance for TriviaQA-Wiki.
We measured the performance using the authors’ codebase 6 .
Analysis Our answer passage retrieval model uses a dot-product between the question and passage representations to score passages.
If we restrict our models to not use additional features apart from this dot product of the question and passage representations, we can derive task-speciﬁc models in a ”zero-shot” setting, where we do not use labeled data from TriviaQA to ﬁne-tune pre-trained representations.
We use this zero-shot setting to further investigate whether the pre-trained representations have learned to assess the semantic similarity between questions and passages based on unlabeled text.
The results are presented in Table 2b.
In this setting, MASK -LM global achieves 66.2, outperforming L + R -LM global , EML O pool and skip-thought vectors.
The max-pooled version of our local L + R -LM method under-performs EML O pool , which is explained by the difference in size of the pre-trained representations.
6. E XTRACTIVE D OCUMENT S UMMARIZATION
We use the CNN/DailyMail dataset (Hermann et al., 2015) to evaluate our summarization model.
We follow the standard data pre-processing and evaluation settings for extractive summarization from (See et al., 2017), and use ROUGE (Lin, 2004) as the evaluation metric.
The results of our summarization system are presented in Table 3.
Compared to other extractive summarization systems, our model improves upon the previous model (Narayan et al., 2018) by 0. in ROUGE-1 and 0.7 in ROUGE-L, without using the sophisticated reinforcement learning techniques employed by that work.
Our model does not perform as well as NeuSum (Zhou et al., 2018), which employed an auto-regressive sentence selection model.
Recently, an abstractive summarization system was able to outperform extractive ones (Paulus et al., 2017). We expect that the novel aspects of our work can also bring improvements in that non-extractive setting and are complementary to the use of auto-regressive sentence selectors. We also compared our models to models using
EML O pool and skip-thought vectors with Bi-LSTM.
As seen these methods are also competitive, but are not as strong as our global hierarchical pre-training approach.
The impact of pretraining is analyzed in Table 2a.
For brevity, we only include ROUGE-L scores, but ROUGE-1 and ROUGE-2 show similar trends.
We also show the accuracy of the top-rank sentence.
As for answer passage retrieval, MASK -LM global is the best overall pretraining algorithm.
R ELATED W ORK
Prior work in supervised learning of neural representations for full documents has shown the effectiveness of hierarchical neural structures, which contain representations of both the sentences and their component words in a two-level hierarchy.
In the lower hierarchy level, the contextual word representation is formed with respect to the sentence.
In the higher level, the contextual sentence representation is formed with respect to the document.
Such structures enable the models to integrate long-distance context from the documents and have been used for labeling sentence sentiment (Ruder et al., 2016), document summarization (Cheng & Lapata, 2016), text segmentation (Koshorek et al., 2018), and text classiﬁcation (Yang et al., 2016), inter alia .
These hierarchical neural representations have been largely learned based on task-speciﬁc labeled data, posing a challenge for applications with a limited number of annotated examples.
Unsupervised pretraining for hierarchical document representations has received relatively little attention despite the recent success on language model pre-training (Peters et al., 2018;
Radford et al., 2018).
Prior work has shown how to use unlabeled text to pre-train representations of individual words, e.g. (Mikolov et al., 2013), or ﬂat (relatively short) sequences of words, where each word representation is contextualized with respect to the sequence (Peters et al., 2017; 2018; Salant & Berant, 2018;
Radford et al., 2018).
Similarly, ﬁxed-length vector representations of full sentences/paragraphs have also been pre-trained from unlabeled text (Le & Mikolov, 2014; Kiros et al., 2015; Dai & Le, 2015; Logeswaran & Lee, 2018), where the sentence/paragraph representations are generated based on the content of the sentence/paragraph alone, ignoring other sentences in the document.
Li et al.
(2015) showed how to train hierarchical document representations from unlabeled text using document auto-encoders but did not use such representations in extrinsic downstream documentlevel tasks.
To the best of our knowledge, no prior work has transferred hierarchical unsupervised document representations to downstream tasks, and evaluated the impact of pretraining local and global document-level contextualizers.
In addition, bidirectional hierarchical representations have not been pretrained without a sophisticated decoder in an auto-encoder framework before.
C ONCLUSION
In this paper, we proposed methods for pre-training hierarchical document representations, including contextual token and sentence/paragraph representations, integrating context from full documents.
We demonstrated the impact of pre-training such representations on three document-level downstream tasks: text segmentation, passage retrieval for document-level question answering, and extractive summarization.