Q -learning with Language Model for Edit-based Unsupervised Summarization (2020)

Abstract: Unsupervised methods are promising for abstractive textsummarization in that the parallel corpora is not required. However, their performance is still far from being satisfied, therefore research on promising solutions is on-going. In this paper, we propose a new approach based on Q-learning with an edit-based summarization. The method combines two key modules to form an Editorial Agent and Language Model converter (EALM). The agent predicts edit actions (e.t., delete, keep, and replace), and then the LM converter deterministically generates a summary on the basis of the action signals. Q-learning is leveraged to train the agent to produce proper edit actions. Experimental results show that EALM delivered competitive performance compared with the previous encoder-decoder-based methods, even with truly zero paired data (i.e., no validation set). Defining the task as Q-learning enables us not only to develop a competitive method but also to make the latest techniques in reinforcement learning available for unsupervised summarization. We also conduct qualitative analysis, providing insights into future study on unsupervised summarizers.

## introduction automatic text summarization 2 is an attractive technique for helping humans to grasp the content of documents effortlessly. while supervised neural methods have shown good performances @xcite , the unsupervised approach is starting to attract interest due to its advantage of not requiring costly parallel corpora. however, the empirical performance of unsupervised methods is currently behind that of state-of-theart supervised models (zhao et al., 2018; baziotis 1 our codes are available at https://github.com/ kohilin/ealm 2 we refer to abstractive summarization in this paper. reconstruction evaluation compression evaluation compression encoder-decoder editorial agent reconstruction encoder-decoder editorial actions for each word [remove, replace, keep …] language model converter q-learning approach with language model conventional approach with two generators machine learning is not perfect ai is imperfect machine learning is not perfect ai is imperfect machine learning is not perfect compression reconstruction machine learning is not perfect et @xcite . unsupervised text summarization is still developing and is now at the stage where various solutions should be actively explored. one previous unsupervised approach extends neural encoder-decoder modeling to the zero paired data scenario, where a model is trained with a paradigm called compression-reconstruction (cr) learning @xcite @xcite . the mechanism is similar to that of the back-translation @xcite : the model consists of a compressor (i.e., summarizer) and a reconstructor, and they are co-trained so that the reconstructor can recover the original sentence from the summary generated by the compressor @xcite ; the left side of figure 1 ). experimental results showed that such an unsupervised encoder-decoder-based summarizer is able to learn the mapping from a sentence to a summary without paired data @xcite . reinforcement learning (rl) is also a potential solution for the no paired data situation. in related fields, for example, there are unsupervised methods for text simplification and text compression with policy-gradient learning @xcite . recent rl techniques take a value-based approach (e.g., q-learning) such as dqn @xcite or the combination of policy and value-based approaches such as asynchronous advantage actor-critic @xcite . a critical requirement to leverage a valuebased method is a value function that represents the goodness of an action on a given state @xcite . we can naturally define the value function by utilizing the cr-learning paradigm, and it makes the latest value-based approaches available for unsupervised text summarization. in this paper, we propose a new method based on q-learning and an edit-based summarization @xcite ; right side of figure 1 ). the edit-based summarization generates a summary by operating an edit action (e.g., keep, remove, or replace) for each word in the input sentence. our method implements the editing process with two modules: 1) an editorial agent that predicts edit actions, and 2) a language mmodel (lm) converter that deterministically decodes a sentence on the basis of action signals, which we call ealm. the cr learning is defined on the qlearning framework to train the agent to predict edit actions that instruct the lm converter to produce a good summary. although a vast action space causing sparsity in reward, such as the word generation of an encoder-decoder model, is generally difficult to be learned in rl, our method mitigates this issue thanks to its fewer edit actions and the deterministic decoding of a language model. moreover, the formulation by q-learning enables us to incorporate the latest techniques in rl. the main contribution of this paper is that we provide a new solution in the form of an unsupervised edit-based summarization leveraging qlearning and a language model. experimental results show that our method achieved a competitive performance with encoder-decoder-based methods even with truly no paired data (i.e., no validation set), and qualitative analysis brings insights as to what current unsupervised models are missing. also, the problem formulation on q-learning enables us to import the latest techniques in rl, which leads to potential improvements in future research. ## task definition we begin by formally defining the problem of unsupervised summarization with the cr learning. the goal of the task is to produce an informative summary y consisting of m words y 1 , y 2 , ..., y m for a given input sentence x consisting of n words x 1 , x 2 , ..., x n where @xmath0. the challenge in this task is to learn the transformation from x to y with only the input sentence x. to tackle this, the cr learning introduces an additional transformation called reconstruction. the reconstruction requests to reproduce the input sentence x from the generated summary y where x is the reproduced sentence consisting of n words x1 , x2 , ..., xn . in terms of the generated sentences y and x, let c be the compression function and r be the reconstruction function: ## previous method the previous approaches use a generative encoderdecoder model @xcite , for the compression and reconstruction functions @xcite @xcite . although the objective functions and implementation details differ depending on the study, the underlying motivation entails the same hypothesis as the cr learning. for example, @xcite introduced four objective functions -discrepancy of y from a pretrained language model, topical distance of x and y, and the length of y and probability difference of x i and xi -where the former threes can be regarded as f (x, y) and the final one as g(x, x). while such an encoder-decoder model has performed well on many generation tasks, it suffers from inherent difficulties related to repetition @xcite , length control @xcite , and exposure bias @xcite . it also runs into convergence problems when co-training multiple generators @xcite . ## proposed method our proposed method, which we call ealm, consists of two essential modules: the editorial agent and the lm converter. the agent sends action signals (keep, remove, or replace each word in a sentence) to the conveter, which then deterministically transforms the input sentence according to the signals. we train the agent to find action signals so that the lm converter produces sentences demanded by the cr learning. in the following sections, we first share the background of q-learning ( §4.1) and then present how to put the task and our approach on the q-learning framework ( §4.2). we next explain the core algorithmic details ( §4.3) and finish with explanations about training and inference ( §4.4). ## experiment baselines. we compare our proposed approach with three baselines: lead-n, which simply takes the beginning n words as the summary, seq3, a recent encoder-decoder model @xcite , and cmatch, a new approach without explicit reconstruction learning @xcite . to conduct qualitative analysis on generated summaries, we ran the baselines ourselves with a replicated model for seq3 4 and the provided model for cmatch. 5 also, we test two types of seq3 models: one tuned with a validation set (seq3 + ) and the other with parameters at the last iteration in the training (seq3 -). this is because ealm and cmatch do not need paired data even for validation. proposed method. we implemented ealm as follows. the q-network of the agent consists of a two-layered mlp with 200 units per layer and relu. we used the adam optimizer with the learning rate of 0.001 and apply gradient clipping by 1. for the epsilon-greedy strategy, we first set the exploring probability to 0.9 and decay it by multiplying by 0.995 every 100 updates until it reaches the minimum exploration rate of 0.03. we set the discount factor γ to 0.995. the size of the replay buffer is 2000, and we sample 128 experiences as 4 https://github.com/cbaziotis/seq3 . we ran the training script with the same configuration as the original paper except for decreasing the batch size from 128 to 32 due to our gpu limitation. we trained three models and obtained slightly lower scores than the ones reported in the original paper. we report the averaged score among the three models. dataset. the same as @xcite , we train our model on the gigaword corpus (giga, @xcite . however, we used only 30k sentences randomly picked from sentences with less than 50 words for the training of ealm. this is because the whole data, 3.8m sentences, is too large to expose the agent to different experiences from the same sentence. 6 note that we used the entirety of sentences for the training of the seq3 models. we followed @xcite in the evaluation as well, using the test set consisting of the giga (1897 sentences) and duc datasets (duc3 with 624 sentences, duc4 with 500 sentences; @xcite . all models follow the same tokenization policy: the default tokenization in giga, duc3, and duc4. although bert (which ealm uses) has its own vocabulary based on subwords, we do not apply subwording to go along with a single tokenization policy. therefore, words not in the bert vocabulary are interpreted as unknown words, and the ratio of unknown words was around 10% in giga. evaluation. in our quantitative analysis, we examine the rouge scores. 7 to mitigate the bias to longer sentences in rouge calculation, we capped all summaries at the first 75 bytes. note that the averaged sentence length of gold summaries after the capping were 8.58, 9.59, and 10.25 for giga , duc3, and duc4, respectively. also, we examine sentence length (len) and count of new words (nw; number of words that are used in a generated summary but do not appear in the input sentence). additionally, we show qualitative comparisons with a manual check of generated summaries. although a questionnaire survey is often conducted to assess the deeper quality of summaries such as informativeness and readability, this still hides the exact points of model's strengths and weaknesses. we consider that specific indications provide insights on future work for the current unsupervised summarizers. we manually checked more than 200 summaries for each model and each dataset and include a few samples in appendix (a.6). ## conclusion we brought the q-learning framework into unsupervised text summarization and proposed a new method ealm that is an edit-based unsupervised summarizer leveraging a q-learning agent and a language model. the experments showed that ealm performed competitively with the previous encoder-decoder-based methods. however, in qualitative analysis, we found that the quality of the generated summaries of any unsupervised model was not sufficient, and there are individual limitations for each model. these issue must be overcome as the step forward to generating practically available summaries without paired data. in particular for ealm, there is room for improvement by importing the latest techniques in rl. our work paves the way for further research on bridging q-learning and unsupervised text summarization.