A Generative Approach to Titling and Clustering W ikipedia Sections (2020)

Abstract: We evaluate the performance of transformer encoders with various decoders for information organization through a new task: generation of section headings for Wikipedia articles. Our analysis shows that decoders containing attention mechanisms over the encoder output achieve high-scoring results by generating extractive text. In contrast, a decoder without attention better facilitates semantic encoding and can be used to generate section embeddings. We additionally introduce a new loss function, which further encourages the decoder to generate high-quality embeddings.

## introduction automated information labeling and organization has become a desirable way to process the copious amounts of available text. we develop methods for producing text headings and section-level embeddings through a new task: generation of section titles for wikipedia articles. this task is useful for improving wikipedia, an active area of research due to the long tail of poor quality articles, including articles lacking section subdivisions or consistent headings @xcite @xcite . additionally, the types of labels used to denote sections can be useful for organizing other unstructured collections of text. we approach this task in two ways: first we train a text generation model for producing section titles, and second, we leverage our model architecture to extract section embeddings, which offer a useful mechanism for comparing and clustering sections with similar information @xcite @xcite . this approach provides a flexible framework for creating paragraph-level embeddings, in which the type of information encoded in the embedding can be controlled by changing the generation task. section title generation is similar to existing tasks, such as generating titles for newspaper articles @xcite . however, wikipedia section titles contain a unique mix of short abstractive headings like "history" and longer extractive headings like song titles, where many of the words in the section title also appear in the section text. the variations in the type of headings makes this dataset useful for analyzing how models perform on different subsets of the data. a common state-of-the-art model for many existing text generation tasks uses an encoderdecoder framework where the encoder is initialized with bert and the decoder is also a transformer @xcite @xcite . the entire output of the encoder is passed to the decoder, which allows the decoder to attend over the entire input sequence during each generation step. in contrast, we explore using transformer encoders with rnn decoders and show that rnn decoders better generate short abstractive titles while transformer decoders perform better on longer extractive titles. embeddings extracted from the rnn decoders also perform better in clustering evaluations, which suggests that the attention-based mechanisms in the transformer facilitate copying input text into the output, but the rnn architecture better facilitates encoding semantic meaning. we additionally introduce a new loss function for the rnn decoder that encourages the start and end states of the rnn to be similar. this loss function encourages the model to encode meaningful information into a single state, which further improves the quality of the generated section-level embeddings. we first describe our models (section 2) and our data set (section 3) and then present results, evaluating our models on a held-out test corpus (section 5). our main contributions include: (1) the introduction of a new short-text generation task that is useful for information labeling and organization; ## models our primary task is to generate section titles, and our secondary task is to generate section-level embeddings. all models use an encoder-decoder architecture, where the encoder is initialized with bert @xcite . we use 4 decoder variants, including one trained with a novel loss function. trans this model contains a (randomly initialized) transformer decoder, with hyperparameters identical to the bert-base model. the hidden states generated by the encoder for the entire input sequence are passed to the decoder, thus allowing the decoder to attend over the entire input sequence during each decoding step. this model serves as our primary baseline, as it is identical to the bert2rnd model in @xcite . we use the same hyperparameters as @xcite , which were selected after extensive tuning. rnn instead of a transformer decoder, we use an rnn, specifically a gated recurrent neural network (gru) @xcite , as the decoder. unlike the transformer decoder, which computes attention over the full input sequence, we do not use any attention mechanisms over the input to the decoder. instead, we only pass the last hidden layer for the first token ("cls" token), forcing the model to encode all meaningful information about the input sequence into this single state. the rnn decoder, which consists of a single decoder layer, is substantially smaller than the transformer decoder used in the trans model. rnn+sc our third model uses the same architecture as the rnn model, but we add an additional component to the loss function that encourages the start state and the end state of the decoder to be similar, which we call a state constraint (sc). the primary intuition behind this loss function is that it encourages the decoder to stay "on topic" while generating text, as it discourages the rnn from wandering too far away from where it started. it further encourages the start state to encode all information needed to generate the entire output se-quence, rather than allowing the start state to focus on information in the beginning of the sequence and the end state to encode information for the end of the sequence. here, f is a gru, @xmath0, . . . , t } is the target token position, and h 0 is initialized to the cls token of the bert source encoder. the formula for the state constraint function is given in equation 2: the normalization terms force the loss term to focus on embedding direction rather than magnitude; they are necessary to account for the arbitrary magnitude of model states. during training, we multiply the state constraint loss, l sc , by a fixed scalar (α) and add it to the standard cross-entropy (ce) loss function. the final loss function is then given by: rnn+attn our final model also uses a transformer encoder and an rnn decoder. however, unlike the previous model, we pass the entire last layer of the encoder to the decoder and add an attention mechanism over this input sequence @xcite . this model and the trans model are attention-based decoders, while the rnn and the rnn+sc models do not use attention over the decoder input. ## data our primary data set consists of articles from english language wikipedia collected on june 25, 2019. we filter out articles that contain the word "redirect" and omit any section whose title has fewer than 2 characters. we extracted sections and section titles from each article and randomly divided the data into train, test, and development sets, using an 80/10/10 split (11.43m/1.43m/1.44m articles). wikipedia articles are often hierarchical, containing multiple subsections. however, we make no distinction between titles that are complete sections and titles that are subsections. this lack of distinction makes the generation task harder, as our models are not able to take advantage of hierarchical information and also allows our models and results to better generalize to other data sets that do not have this hierarchy. more detailed statistics on the data set are shown in in the wikipedia corpus, across 14.3m data points, there are only 6.5m distinct headings (45.25% of all titles). approximately 6m headings (41.82%) occur only 1 time in the data, meaning the other 0.5m headings are reused multiple times across 8.3m articles to constitute the remainder of the corpus. the most common heading, "history", occurs 480k times in the data set, making up 3.35% of the total corpus. other common headings include "career" (181k), "biography" (151k), "early life" (111k), "background" (102k) and "plot" (96k). in contrast, the titles in gigaword are generally longer and more distinctive than the wikipedia section titles, with 80.45% of all titles being unique. however, in the absence of generic abstract headings like "history", the gigaword corpus tends to be more extractive, meaning there is high tokenoverlap between articles and their titles. the wikipedia corpus is also much larger than gigaword, which facilitates analyses. ## experimental setup for all encoders, we use the bert-base uncased model. thus, we lowercase all text and use wordpiece tokenization from the public bert wordpiece vocabulary @xcite . we use the same preprocessing pipeline, including word-piece tokenization, when computing target text length and extractive scores. for all models, we limit the encoder input size to 128 tokens and the decoder output size to 32 tokens and use a batch size of 32. we generally use a learning rate of 0.05 with square root decay, 40k warm-up steps, and the adam optimizer; however, for the rnn models with the gigaword data, we use 100k warm-up steps, clip gradients to 20, and optimize with adagrad, which we found to produce smoother training curves. for the state constraint models, we start by setting the scalar α = 0, and linearly increase α to 1, between 100k and 200k training steps. we train the rnn models for 2m steps using v100 gpus, and we train the transformer models for 500k steps using tpus. in practice, we find that the rnn performance stops improving within 1m steps and the transformer performance stops within 50k steps. ## related work while we introduce the task of wikipedia section heading generation, the task of article headline generation using the gigaword corpus has been well-studied, primarily using an encoder-decoder architecture with additional modules like attention or copy mechanisms @xcite . @xcite further explore how to leverage the pretrained bert model for abstractive summarization, primarily using the cnn/daily mail data set. rothe et al. (2019) perform a comprehensive assessment of pretrained language models for text generation tasks, including the gigaword task. our trans model is identical to their bert2rnd model and achieves comparable results over the gigaword corpus. the high level of extraction in existing text generation tasks has motivated the use of mechanisms that explicitly copy input text into the output @xcite or the introduction of new data sets @xcite . furthermore, models trained for extractive summarization often outperform abstractive models on abstractive data sets @xcite @xcite . our work extends these results by showing that even abstractive models are implicitly learning extraction, as they perform better on extractive text. our metric for measuring extractiveness is similar to the 'novel n-gram percentage' proposed by @xcite ; however, we use the same input pipeline for computing this metric as for training our models, and we correlate extractive score with performance, rather than just using it as an extrinsic measure of abstraction @xcite . in our wikipedia section heading generation task, the prevalence of generic headings makes the task more abstractive than datasets like gigaword @xcite , or even other short-text generation tasks, like email subject prediction @xcite , which makes it a useful dataset for analyzing model performance. it is also extrinsically useful -most automated methods for improving wikipedia focus on creating new content, such as through multi-document summarization @xcite or generating text from structured data @xcite . however, less than 1% of all english wikipedia articles are considered to be of quality class good, suggesting there is a need for improving existing articles. @xcite show that many low quality articles consist of 0-1 sections and present a method for recommending new sections for an author to add to the article. our approach offers a way to label existing paragraphs as distinct sections. our approach also results in document embeddings, which we show can be used to cluster sections. document embeddings are useful for a variety of tasks including news clustering @xcite , argument clustering @xcite , and as features for downstream tasks like text classification @xcite . while tf-idf vectors have historically been a popular construction method @xcite , more recent methods have focused on distributive representations, particularly doc2vec, a generalization of the word2vec algorithm @xcite @xcite . finally, the growing popularity of pretrained language models like bert has led to numerous investigations on what these models learn @xcite @xcite . most investigations involve using targeted probing tasks. while our work shares similar goals, in that we investigate what type of information these models learn, we focus on data subsets and performance analysis. ## future work our work offers several avenues for future exploration. we focus only on english wikipedia. however, there are numerous language editions of wikipedia, many of which have far fewer articles than the english edition and could benefit from tools for text generation. 2 additionally, while we discard the hierarchical nature of wikipedia sections, this information could offer a way to improve model performance (potentially at the cost of generalizability to other data sets). furthermore, while we evaluate the performance of our generated section embeddings for clustering, more work is needed to assess their usefulness on other tasks, such as retrieving relevant sections from a query, measuring section similarities, or as features for text classification. ## conclusions overall, our work introduces the task of generating section titles for text. we also introduce the rnn+sc model and demonstrate how rnn decoders can be utilized for short text generation and improved section embeddings. specifically, our method for generating text embeddings, which involves leveraging internal states of models trained for generation, allows the embeddings to contain targeted information that maximizes their usefulness for specific tasks.