We describe an application of an encoder-decoder recurrent neural network with LSTM units and attention to generating headlines from the text of news articles. We find that the model is quite effective at concisely paraphrasing news articles. Furthermore, we study how the neural network decides which input words to pay attention to, and specifically we identify the function of the different neurons in a simplified attention mechanism. Interestingly, our simplified attention mechanism performs better that the more complex attention mechanism on a held out set of articles. Recurrent neural networks have recently been found to be very effective for many transduction tasks -that is transforming text from one form to another. Examples of such applications include machine translation [1,2] and speech recognition [3] . These models are trained on large amounts of input and expected output sequences, and are then able to generate output sequences given inputs never before presented to the model during training. Recurrent neural networks have also been applied recently to reading comprehension [4] . There, the models are trained to recall facts or statements from input text. Our work is closely related to [5] who also use a neural network to generate news headlines using the same dataset as this work. The main difference to this work is that they do not use a recurrent neural network for encoding, instead using a simpler attention-based model. We use the encoder-decoder architecture described in [1] and [2] , and shown in figure 1. The architecture consists of two parts -an encoder and a decoder -both by themselves recurrent neural networks.
REF used encoder-decoder with attention method to generate news headlines.
Generating News Headlines with Recurrent Neural Networks
ArXiv
ArXiv
Computer Science