= = =
= = =
= = =
= =
= =
= =
= =
= =
= =
= =
= =
= =
= =
=
==
G ENERATING M ULTI -S ENTENCE A BSTRACTIVE S UMMARIES OF I NTERLEAVED T EXTS
Anonymous Submission
A BSTRACT
In multi-participant postings, as in online chat conversations, several conversations or topic threads may take place concurrently.
This leads to difﬁculties for readers reviewing the postings in not only following discussions but also in quickly identifying their essence.
A two-step process, disentanglement of interleaved posts followed by summarization of each thread, addresses the issue, but disentanglement errors are propagated to the summarization step, degrading the overall performance.
To address this, we propose an end-to-end trainable encoderdecoder network for summarizing interleaved posts.
The interleaved posts are encoded hierarchically, i.e., word-to-word (words in a post) followed by post-to-post (posts in a channel).
The decoder also generates summaries hierarchically, threadto-thread (generate thread representations) followed by word-to-word (i.e., generate summary words).
Additionally, we propose a hierarchical attention mechanism for interleaved text.
Overall, our end-to-end trainable hierarchical framework enhances performance over a sequence to sequence framework by 8-10% on multiple synthetic interleaved texts datasets.
I NTRODUCTION
Interleaved texts are becoming more common with new ways of working and new forms of communication, e.g., multi-author entries for activity reports, meeting minutes and social media conversations, such as Slack.
Quickly getting a sense of or following the content of different threads in interleaved texts, where posts belonging to different threads occur in one sequence, is often difﬁcult.
An example of two threads with multiple posts interleaved to form a sequence is:
Post1-Thread
→ Post1-Thread2 → Post2-Thread1 → Post3-Thread
→ Post
[Equation]
Thread
[Equation]
Post
[Equation]
Thread
This intermingling leads to difﬁculties in not only following discussions but also in retrieving their essence.
In conversation disentanglement, interleaved posts are grouped by the thread; e.g., the previous example could be rearranged as:
[Equation]
In analyzing interleaved texts, Shang et al.
(2018) went a step further and proposed summarization of the interleaved texts.
They designed an unsupervised two-step system and evaluated the system on meeting texts.
In the ﬁrst step, a conversation disentanglement component disentangles the texts thread-wise.
Then, in the second step, a multi-sentence compression component compresses the thread-wise posts to single sentence summaries.
However, this system has a major disadvantage, in that the disentanglement obtained through either supervised Jiang et al.
(2018) or unsupervised
Wang & Oard (2009) methods propagate its errors to the downstream summarization task, and thus, degrades the overall performance.
We aim to tackle this issue of error propagation through an end-to-end trainable encoder-decoder system that takes a variable length input, e.g., interleaved texts, processes it and generates a variable length output, e.g., a multi-sentence summary; see Figure 1 for an illustration.
An end-to-end system eliminates the disentanglement component, and thus, the error propagation.
The encoder ﬁrst performs word-to-word encoding to embed each post, followed by post-to-post encoding to embed the overall content of the posts and represent the discourse structure of the interleaved texts.
The decoder has a thread-to-thread decoder to generate a representation for each thread, and the thread representation is given to a word-to-word decoder to generate a summary sentence.
We also propose to use hierarchical attention similar to Nallapati et al.
(2016), but instead of computing post-level attention at every word, attentions are only computed when decoding new sentences.
Further, the attention networks are trained end-to-end.
Despite the availability of a multitude of real-world interleaved texts, a major challenge to train encoder-decoder models is the lack of labels (summaries).
As labeling is difﬁcult and expensive
Barker et al.
; Aker et al.
; Verberne et al.
(2018), we synthesize two separate corpora.
Each is derived by mixing texts and associated summaries from a corpus of documents, where the mixed text has a structure reﬂective of a multi-party conversation with inter-leaved threads and the summary highlights the information in the threads.
We ﬁnd abstracts and titles of randomized controlled trial (RCT) articles, a PubMed corpus, and also for Stack Exchange posts and titles.
To summarize, our contributions are threefold:
• We propose to combine a hierarchical encoder and decoder to obtain multi-sentence summaries of interleaved texts.
• We propose a novel hierarchical attention mechanism that is integrated with the hierarchical encoder-decoder architecture and which equips the decoder to disentangle the interleaving further.
• We use two synthetic datasets to verify the ideas and show our end-to-end trainable architecture addresses not only the issue of error-propagation observed in competitive methods but also improves the performance
R ELATED W ORK
Quite often multi-party conversations, e.g. news comments, social media conversation and activity report, have tens of posts discussing several different matters pertaining to a subject.
Ma et al.
Aker et al.
; Shang et al.
(2018) designed methodologies to summarize posts in order to provide an overview on the discussed matters.
They broadly follow the same approach: cluster the posts and then extract a summary from each cluster.
There are two kinds of summarization: abstractive and extractive.
In abstractive summarization, the model utilizes a corpus level vocabulary and generates novel sentences as the summary, while extractive models extract or rearrange the source words as the summary.
Abstractive models based on neural sequence-to-sequence (seq2seq) (Rush et al.
(2015)) proved to generate summaries with higher
ROUGE scores than the feature-based abstractive models.
Integration of attention into seq2seq
(Bahdanau et al.
(2014)) led to further advancement of abstractive summarization (Nallapati et al.
(2016); Chopra et al.
There are many possible patterns of organization of the information in texts, e.g., chronological pattern.
News articles have an inverted pyramid pattern, i.e., core information in the lead sentences and the extra information in later sentences.
A seq2seq model is appropriate for summarization of a news article as it encodes and decodes sequentially.
However, in interleaved texts, related information maybe separated; thus a seq2seq model may be competent but not sufﬁcient.
Li et al.
(2015) proposed an encoder-decoder (auto-encoder) model that utilizes a hierarchy of networks: word-to-word followed by sentence-to-sentence.
Their model is better at capturing the underlying structure than a vanilla sequential encoder-decoder model (seq2seq).
Krause et al.
Jing et al.
(2018) showed multi-sentence captioning of an image through hierarchical Recurrent
Neural Network (RNN), topic-to-topic followed by word-to-word, is better than seq2seq.
These works suggest a hierarchical encoder, with word-to-word encoding followed by post-to-post, will better recognize the dispersed information in interleaved texts.
Similarly, a hierarchical decoder, thread-to-thread followed by word-to-word, will intrinsically disentangle the posts, and therefore, generate more appropriate summaries.
Nallapati et al.
(2016) devised a hierarchical attention mechanism for a seq2seq model, where two levels of attention distributions over the source, i.e., sentence and word, are computed at every step of the word decoding.
Based on the sentence attentions, the word attentions are rescaled.
Hsu et al.
(2018) slightly simpliﬁed this mechanism and computed the sentence attention only at the ﬁrst step.
Our hierarchical attention is more intuitive and computes new sentence attentions for every new summary sentence, and unlike Hsu et al.
(2018), is trained end-to-end .
[Figure]
P ROBLEM S TATEMENT
We aim to design a system that when given a sequence of posts, C = (cid:104) P 1 , . .
. , P | C | (cid:105) , produces a sequence of summaries, T = (cid:104) S 1 , . .
, S | T | (cid:105) .
For simplicity and clarity, unless otherwise noted, we will use lowercase italics for variables, uppercase italics for sequences, lowercase bold for vectors and uppercase bold for matrices.
Figure 2 illustrates the proposed hierarchical encoder-decoder framework.
In the framework, ﬁrst, a low-level, word-to-word encoder converts a sequence of words in a post, P j , to a sequence of representations, H j = (cid:104) h
w 2 w j 0 , . . .
h w 2 w j | P j | (cid:105) .
Subsequently, a top-level, post-to-post encoder converts those representations, (cid:104) H 0 , . . .
, H | C | (cid:105) , to a sequence of top-level post representations (cid:104)
h p 2 p 1 , . . .
, h p 2 p | C | (cid:105) .
These encoded representations are then passed to the decoder, which utilizes a top-level, threadto-thread, decoder to disentangle them into a sequence of thread representations (cid:104) h t 2 t 1 , . . .
, h t 2 t | T | (cid:105) .
Finally, a low-level, word-to-word, decoder takes a feed-forward mapped thread representation h t 2 t i and generates a sequence of summary words (cid:104)
w i 1 , . . .
, w i | S i | (cid:105) .
The maximum number of posts in a sequence of interleaved texts is denoted by n and threads by m .
We limit the number of words in posts and summaries to ﬁxed lengths by either truncating or stop ββ np <sos>
D w 2 w D t 2 t h t 2 t i h w 2 w 00 h t 2 t i − h t 2 t i − h t 2 t i − β ∗ W h w 2 w ih w 2 w i −1, q s i h
w 2 w 0 p h w 2 w np padding, and denote them by p and q respectively.
The hidden states of the and decoder have dimensionality
E NCODER
The hierarchical encoder used in Figure 2 is based on Nallapati et al.
(2017), where word-toword and post-to-post encoders are bi-directional LSTMs.
The word-to-word Bi-LSTM encoder ( E w 2 w ) runs over word embeddings of any post, P j , and generates a set of hidden representations ( (cid:104)
h
E w 2 w j, . .
, h E w 2 w jp (cid:105) ); see left middle section in Figure 2.
The average pooled value of the wordto-word representations of post P j ( 1 p (cid:80) pk =
h E w 2 w jk ) are input to the post-to-post Bi-LSTM encoder ( E t 2 t ), which then generates a set of representations ( (cid:104)
h
E p 2 p, . .
, h E p 2 p n (cid:105) ) corresponding to those posts.
Overall, output word-to-word representations, W , has dimension n × p × 2 l and post-to-post representations, P , has dimension n × 2 l for a given channel, C .
D ECODER
Our hierarchical decoder (see Figure 2) is based on Li et al.
(2015), where both thread-to-thread and word-to-word decoders are uni-directional LSTMs.
The initial state h D t 2 tof the thread-to-thread LSTM decoder ( f D t 2 t ) is set with a feedforwardmapped representation of an average pooled post representations ( c (cid:48)
= 1 n (cid:80) nk =
h p 2 p k ).
At each step i of the decoder, a sequence of attention weights, (cid:104)
ˆ β i, 0 , . . .
, ˆ β i,n × p (cid:105) , corresponding to the set of encoded word representations, (cid:104)
h
w 2 w, . .
, h w 2 w n × p (cid:105) are computed utilizing the previous state, h D t 2 t i − 1 .
We will elaborate the attention computation in the next section.
A weighted representation of the words (crossed blue circle) is then computed: (cid:80) nj =1 ˆ β ij W ij , and used as one input to LSTM
f D t 2 t .
Additionally, we use the last hidden state h D w 2 w i − 1 · of the wordto-word decoder LSTM ( D w 2 w ) of the previously generated summary sentence as the second input to compute the next state of thread-to-thread decoder, i.e., h D t 2 t i .
The motivation is to provide information about the previous sentence.
The current state h D t 2 t i is passed through a single layer feedforward network and a distribution over
STOP=1 and CONTINUE=0 is computed:
[Equation]
where g is a feedforward network.
In Figure 3, the process is depicted by a red circle.
The threadto-thread decoder keeps decoding until p STOPi is greater than 0.5.
Additionally, the current state h D t 2 t i is passed through another single layer feedforward network k followed by tanh activation to compute the thread representation s i = tanh( k (cid:16) h D t 2 t i (cid:17) ) .
Given a thread representation s i , the word-to-word decoder generates a summary for the thread.
Our word-to-word decoder is
Bahdanau al.
attentional
H IERARCHICAL A TTENTION
Our novel dynamic hierarchical attention is based on two popular concepts: attention scores should integrate in top-down manner (Nallapati et al.
and two distributions over same words must be modeled together (See et al.
(2017)), not just in isolation.
Our hierarchical attention works at 3 levels, the post level (corresponding to posts), i.e., γ , and higher level (corresponding to source tokens), i.e., β , are computed while obtaining a thread representation, s .
The lower level attention (also corresponding to source tokens), i.e., α , is computed while generating a word, y , of a summary, S .
; see yellow rectangles in Figure 4.
At step i of thread decoding, we also compute elements of post-level attention, i.e., γ i · as below.
[Equation]
At the same step, we also compute elements of high-level attention, i.e, β i · as below.
[Equation]
W
[Equation]
j
[Equation]
where k :
= j ∗ l and j ∈ { 1 , . .
. , n } , l ∈
[Equation]
p
[Equation]
where a k is computed as in Eq.
W in Eq.
4 is a word-to-word encoder representations of dimension n × p × 2 l , add aligns a post representation to its constituting word representations and does element-wise addition, and attn β is a feedforward network that aligns the current thread decoder state h D t 2 t
i − 1 with all n ∗ p number of a k vectors.
Then, we use γ to rescale high-level attention, β as below.
[Equation]
At step t of word-to-word decoding of summary thread i , we compute elements of low level attention, i.e., α it · as below.
[Equation]
where e
[Equation]
, and a k is same as in Eq.
4 and attn α is a feedforward network that aligns the current word decoder state
h
D
w 2 w t −with all n ∗ p number of a k vectors.
Finally, we use rescaled high-level word attentions, ˆ β , for rescaling low level attention, α as below:
[Equation]
Table 1 compares our hierarchical attention with previous work on hierarchical attention in the text summarization domain.
[Table] [Table]
[Table] [Table]
[Table]
T RAINING O BJECTIVE
We train our hierarchical encoder-decoder network similarly to an attentive seq2seq model Bahdanau et al.
(2014), but
with an additional weighted sum of sigmoid cross-entropy loss on stopping distribution; see Eq. 1.
Given a summary, Y i = (cid:104) w i 0 , . . .
, w iq (cid:105) , our word-to-word decoder generates a target ˆ Y =
(cid:104)
y
i 0 , . . .
, y iq (cid:105) , with words from a same vocabulary U .
We train our model end-to-end by minimizing the objective given in Eq. 8.
[Equation]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
D ATASET
Obtaining labeled training data for conversation summarization is challenging.
The available ones are either extractive (Verberne et al.
(2018)) or too small (Barker et al.
; Anguera et al.
to train a neural model.
To get around this issue and thoroughly verify the proposed architecture, we synthesized a dataset by utilizing a corpus of conventional texts for which summaries are available
We create two corpora of interleaved texts: one from the abstracts and titles of articles from the
PubMed corpusand one from the questions and titles of Stack Exchange questions.
A random interleaving of sentences from a few PubMed abstracts or Stack Exchange questions roughly resembles interleaved texts, and correspondingly interleaving of titles resembles its multi-sentence summary.
The algorithm that we devised for creating synthetic interleaved texts is deﬁned in detail in the
Appendix.
The number of abstracts to include in the interleaved texts is given as a range (from a to b ) and the number of sentences per abstract to include is given as a second range (from m to n ).
We vary the parameters as below and create three different corpora for experiments: Easy ( a =2, b =2, m =5 and n =5), Medium ( a =2, b =3, m =2 and n =5) and Hard ( a =2, b =5, m =2 and n =5).
Tableshows an example of a data instance in the Hard Interleaved RCT corpus.
E XPERIMENTS
we report ROUGE-1, ROUGE-2, and ROUGE-L as the quantitative evaluation of the models.
The
Pubmed Medium and Hard corpora train, test and validation have approximately 290k, 6k and 1.5k instances, respectively.
The Stack-Exchange Medium corpus train has 180k and Hard has 210k instances, while the both corpora have test and validation of approximately 5k and 4k instances, respectively.
The remaining hyper-parameters are described in detail in the Appendix.
B ASELINES
We ﬁrst trained and tested it on the Easy corpus.
We also ran Shang et al.
’s unsupervised two-step system on the test set of the Easy Interleaved Pubmed corpus.
Additionally, we also utilized
Shang et al.
’s clustering component to ﬁrst cluster the interleaved texts of the corpus, and then the disentangled corpus is used to train the seq2seq model.
We refer to the latter as cluster → seq2seq.
The performance comparison of Shang et al.
and the two seq2seq models are shown in Table 3.
Clearly, seq2seq performs better than Shang et al.
(2018), the reason being a seq2seq model trained on a sufﬁciently large dataset is better at summarization than the unsupervised sentence com-
[Table] [Table]
[Table] [Table]
pression (extractive) method.
The lower performance of cluster → seq2seq in comparison to seq2seq shows that a disentanglement component is unnecessary in easy scenario.
Furthermore, we utilize the ground-truth labels and disentangle the interleaved posts of a sampled (150k) Hard Corpus, and re-run the seq2seq model.
The results (see Table 4) show seq2seq (row 3) takes advantage of cluster labels but hier2hier (row 4) does more than it.
S EQ 2 SEQ VS . HIER 2 HIER MODELS
We then compare the proposed hierarchical approach against the seq-to-seq approach in summarizing the interleaved texts by experimenting on the Medium and Hard corpora.
Table 5 shows the experimental results on several corpora; clearly, an increase in the complexity of interleaving scantly impacts the performances.
Though, in case of Pubmed, seq2seq depreciates by 1-2 points while hier2hier remains nearly same with increase of complexity.
In Stack Exchange corpora, as Hard corpora has more training data ( + 30k) seq2seq improves slightly.
However, a noticeable improvement is observed on changing the decoder to hierarchical, i.e., 1.5-3 Rouge points in Pubmed and 2-4.5 points in Stack Exchange depending on the interleaving complexity.
[Table]
To understand the impact of hierarchy on the hier2hier model, we perform an ablation study and use the Hard Pubmed corpus for the experiments, and Table 6 shows the results.
Clearly, adding hierarchical decoding already provides a boost in the performance.
Hierarchical encoding also adds some improvements to the performance; however, the enhancement attained in training and inference speed by the hierarchical encoding is much more valuable.
We will discuss it in depth later.
Further, we also compiled corpora to mimic real world conversation interleaving, wherein the sequence of summaries for interleaved posts may not follow the sequential occurrence of posts, see 17-18 in Algorithm.
An example, a post mention an action in the very beginning of conversation, but does only elaborates it at the end.
Therefore, summary corresponding to the action should be at the end.
So, we again create Medium and Hard real world mimicking corpora of the Stack Exchange dataset as it is more similar to real world conversations, and perform the same set of experiments.
As seen in Table 7, the results show that both the seq2seq and hier2hier models performance are slightly lower as it is tougher task; see Table 5.
In addition, the hier2hier model is still consistently better than the seq2seq model.
Importantly, hier2hier converges much earlier than the seq2seq and also reaches a lower training loss (Figure 5 in Appendix).
hier2hier takes ≈ 1.5 days for training on a Tesla V100 GPU, while seq2seq takes ≈ 4.5 days.
Therefore, the hier2hier model not only achieves greater accuracy but also reduces training and inference time.
[Table] [Table]
H IERARCHICAL ATTENTION
The contribution of the post-level and high-level attentions in a hierarchical decoder is two-fold: computing the thread representation and rescaling the word-level attentions.
To understand the impact of hierarchical attention on the hier2hier model, we perform an ablation study of post-level attentions ( γ ) and high-level attentions ( β ), using the Hard corpus for the experiments.
[Table]
Table 8 shows the performance comparison.
Clearly, γ attention improves the performance (0.5-1) of hierarchical decoding but not a lot.
The high-level attention, i.e., β is very important as without it the model performances is noticeably reduced (Rouge values decrease from 2-3).
Additionally, we also include Li, Luong and Jurafsky, 2015 post-level attention technique in the comparison, as it is the closest hierarchical attention to ours (see Table 1).
Though, (Li, Luong and Jurafsky) utilize a softmax obtained β to compute thread representation, and in contrast to us, they do not reuse β for re-scaling α , and thereby, closest to hier2hier (+ γ − β ) .
Evidently, re-utilization gives hier2hier (+ γ − β ) an edge over (Li, Luong and Jurafsky, 2015).
Lastly, removing both the γ and β ) makes the hier2hier similar to seq2seq, except a few more parameters, i.e., two additional LSTM, and the performance is also very similar.
D ISCUSSION
Table 9 shows an output of our hierarchical abstractive system, in which interleaved texts are in the top, and ground-truth and generated summaries in the bottom.
Table 9 also shows the top two post indexes attended to by the post-level attention ( γ ) while generating those summaries, and they coincide with relevant posts.
Similarly, the top 10 indexes (words) of the high-level attention ( β )
[Table]
this
study was conducted to evaluate the inﬂuence of excessive sweating during long-
distance
running on the urinary concentration of caffeine
[Table]
to assess the effect of a program of supervised ﬁtness walking and patient education on functional status , pain , and... . . .
[Table]
a total of 102 patients with a documented diagnosis of primary osteoarthritis of one or both knees participated...
we examined the effects of intensity of training on ratings of perceived exertion
[Table] [Table]
caffeine in sport .
inﬂuence of endurance exercise on the urinary caffeine concentration .
0,2 effect of excessive [UNK] during [UNK] running on the urinary concentration of caffeine .
supervised ﬁtness walking in patients with osteoarthritis of the knee .
a randomized , controlled trial .
1,4 effect of a physical ﬁtness walking on functional status , pain , and pain the effect of training intensity on ratings of perceived exertion .
6,8 effects of intensity of training on perceived [UNK] in [UNK] athletes .
is directly visualized in the table through the color coding matching the generation.
The system not only manages to disentangle the interleaved texts but also to generate appropriate abstractive summaries.
Meanwhile, β provides explainability of the output.
As interleaving in the table includes abstracts from the same category, e.g. Physical Activities, the interleaving is complex and approximates the real-world conversations.
Despite that, the end-toend hierarchical system tackles the task to a large extent.
The next, future stage in this research is transfer learning of the hierarchical system to more types of conversations.
We ﬁnd models trained for 50k iteration on Pubmed Hard corpora (a=6, b=10, m=2 and n=3) and then ﬁne-tuned on the popular meeting AMI corpus (McCowan et al.
(2005)) already reaches competitive results (Rouge1=39.81, Rouge-2=11.35) against SOA system on AMI (Shang et al.
(2018) Rouge-1=37.
and
Rouge-2=07.84).
Additionally, we aim to modify hier2hier to include some of the recent additions of seq2seq models, e.g., See et al.
(2017) pointer mechanism.
C ONCLUSION
We presented an end-to-end trainable hierarchical encoder-decoder architecture which implicitly disentangles interleaved texts and generates a multi-sentence abstractive summary covering the text threads.
Furthermore, the architecture addresses the error propagation issue that occurs in the twostep architectures.
Our proposed novel hierarchical attention further boosts both disentanglement and summary generation.
A PPENDIX I NTERLEAVE A LGORITHM
In Algorithm.
1, I NTERLEAVE takes a set of concatenated abstracts and titles, C = (cid:104)
A 1 ; T 1 , . . .
, A | C | ; T | C | (cid:105) , minimum, a , and maximum, b , number of abstracts to interleave, and minimum, m , and maximum, n , number of sentences in a source, and then returns a set of concatenated interleaved texts and summaries.
WINDOW takes a sequence of texts, X , and returns a window iterator of size | X |− w t
+ 1 , where w and t are window size and sliding step respectively.
window reuses elements of X , and therefore, enlarges the corpus size.
Notations U refers to a uniform sampling, [ · ] to array indexing, and R EVERSE to reversing an array.
1: procedure I NTERLEAVE
2: ˆ C , Z ← WINDOW (
, Array
) 3: while
C (cid:54) = ∅ do 4:
C (cid:48) , A (cid:48) , T (cid:48) , S ← ˆ C .
N EXT () , Array(), Array(), {} 5: r ∼ U ( a , b ) 6: for j
1 to r do (cid:46) Selection 7:
A , T ← ˆ C [ j ] 8:
T (cid:48) .
A DD ( T ) 9: q ∼ U ( m , n ) 10:
A (cid:48) .
A DD ( A [1: q ]) 11:
S ← S ∪ { j × q } 12: ˆ A , ˆ T ← Array(), Array() 13: for 1 to | S | do (cid:46)
Interleaving 14: k ← U ( S ) 15:
S ← S \ k 16:
I ← R EVERSE ( A (cid:48)
[ k ]) .
POP () 17: ˆ A .
A DD ( I ) 18:
J ← T (cid:48)
[ k ] 19: if J (cid:54)∈ ˆ T then : 20: ˆ T .
A DD ( J ) 21:
Z .
A DD (ˆ A ; ˆ T ) 22: return Z
P ARAMETERS
For the word-to-word encoder, the steps are limited to 20, while the steps in the word-to-word decoder are limited to 14.
The steps in the post-to-post encoder and thread-to-thread decoder depend on the corpus type, e.g., Medium has 15 steps in post-to-post and 3 steps in thread-to-thread.
In seq2seq experiments, the source is ﬂattened, and therefore, the number of steps in the source is limited to 300.
We initialized all weights, including word embeddings, with a random normal distribution with mean 0 and standard deviation 0.1.
The embedding vectors are of dimension 80.
The vocabulary size is limited to 5000 and 15000 for Pubmed and Stack Exchange corpora respectively.
All hidden states of the encoder and decoder in the models are set to dimension 100.
We pad short sequences with a special token, (cid:104) PAD (cid:105) .
We use Adam Kingma & Ba (2014) with an initial learning rate of .0001 and batch size of 64 for training.
Texts are lowercased and numbers are replaced by the special symbol % .
D ATA POINTS AND R ESULTS
T RAINING : SEQ 2 SEQ VS HIER
2 HIER
[Table]
Interleaved Texts
botulinum toxin a is effective for treatment of idiopathic detrusor overactivity ( [UNK] )
the [UNK] is generally [UNK] because of the theoretical risk of [UNK] reﬂux ( [UNK] ) , although studies assessing...
most [UNK] injection drug users ( idus ) have been infected with hepatitis c virus ( hcv ) , but...
[UNK] differences in the incidence of bleeding and its relation to subsequent mortality in patients with st-segment elevation myocardial infarction...
[Table]
optimal hcv management approaches for idus remain unknown
GroundTruth/Generation
prospective randomised controlled trial comparing trigone-sparing versus trigone-including intradetrusor injection of abobotulinumtoxina for refractory idiopathic detrusor overactivity.
[Table] [Table]
efﬁcacy
UNK UNK
patients with
[Table]
detrusor
[Table] [Table]
rationale and design of a randomized controlled trial of directly observed hepatitis c treatment delivered in methadone clinics.
[Table]
of a point-of-care hepatitis injection drug injection drug , hcv
[Table]
and
[Table]
of incidence of bleeding and mortality of men versus women
with
-elevation myocardial infarction treated with ﬁbrinolysis .
[Table]
of
with
[Table]
myocardial
with
[Table]
myocardial infarction .
[Table]
Interleaved Texts
[Table]
[Table] [Table]
women
[Table]
kenya received short-course zidovudine ( zdv ) , single-dose nevirapine ( [UNK] ) , combination [UNK] or short-course...
[Table]
breast milk samples were collected two to three times weekly for 4-6 weeks ....
[Table]
and
[Table]
response in
[Table] [Table]
[Table]
-1 persists on hiv-1 rna in human immunodeﬁciency virus-infected individ- uals with hivpatterns of individual and population-level adherence to antiretroviral therapy and risk factors for poor adherence in the ﬁrst year of the dart trial in uganda and zimbabwe .
[Table]
impact of a antiretroviral treatment algorithm on adherence to antiretroviral therapy in [UNK] , efﬁcacy
[Table]