MART : Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning (2020)

Abstract: Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Recurrent Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph generation. Extensive experiments, human evaluations, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than baseline methods, while maintaining relevance to the input video events.

## introduction in video captioning, the task is to generate a natural language description capturing the content of a video. recently, dense video captioning @xcite has emerged as an important task in this field, where systems first generate a list of temporal event segments from a video, then decode a coherent paragraph (multi-sentence) description from the generated segments. @xcite simplifies this task as generating a coherent paragraph from a provided list of segments, removing the requirements for generating the event segments, and focusing on decoding better paragraph captions from the segments. as noted by @xcite ; @xcite , generating paragraph descriptions for videos can be very challenging due to the difficulties of having relevant, less redundant, as well as coherent generated sentences. towards this goal, @xcite proposed a variant of the lstm network @xcite ) that generates a new sentence conditioned on previously generated sentences by passing the lstm hidden states throughout the entire decoding process. @xcite further augmented the above lstm caption generator with a set of three discriminators that score generated sentences based on defined metrics, i.e., relevance, linguistic diversity, and inter-sentence coherence. though different, both these methods use lstms as the language decoder. recently, transformers @xcite ) have proven to be more effective than rnns (e.g., lstm @xcite , gru @xcite , etc.), demonstrating superior performance in many sequential modeling tasks @xcite @xcite @xcite . @xcite first introduced the transformer model to the video paragraph captioning task, with a transformer captioning module decoding natural language sentences from encoded video segment representations. this transformer captioning model is essentially the same as the original transformer @xcite for machine translation, except that it takes a video representation rather than a source sentence representation as its encoder input. however, in such design, each video segment caption is decoded individually without knowing the context (i.e., previous video segments and the captions that have already been generated), thus often leading to inconsistent and redundant sentences w.r.t. previously generated sentences (see figure 3 for examples). @xcite recognize this problem as context fragmentation in the task of language modeling, where the transformers are operating on separated fixed-length segments, without any information flow across segments. therefore, to generate more coherent video paragraphs, it is imperative to build a model that can span over multiple video segments and capture longer range dependencies. hence, in this work, we propose the memory-augmented recurrent transformer (mart) model (see section 3 for details), a transformer-based model that uses a shared encoder-decoder architecture augmented with an external memory module to enable the modeling of the previous history of video segments and sentences. compared to the vanilla transformer video paragraph captioning model @xcite , our first architecture change is the unified encoder-decoder design, i.e., the encoder and decoder in mart use shared transformer layers rather than separated as in @xcite ; @xcite . this unified encoderdecoder design is inspired by recent transformer language models @xcite @xcite to prevent overfitting and reduce memory usage. additionally, the memory module works as a memory updater that updates its memory state using both the current inputs and previous memory state. the memory state can be interpreted as a container of the highly summarized video segments and caption history information. at the encoding stage, the current video segment representation is enhanced with the memory state from the previous step using cross-attention @xcite . hence, when generating a new sentence, mart is aware of the previous contextual information and can generate paragraph captions with higher coherence and lower repetition. transformer-xl @xcite is a recently proposed transformer language model that also uses recurrence, and is able to resolve context fragmentation for language modeling @xcite . different from mart that uses a highly-summarized memory to remember history information, transformer-xl directly uses hidden states from previous segments. we modify the transformer-xl framework for video paragraph captioning and present it as an additional comparison. we benchmark mart on two standard datasets: activitynet captions @xcite and youcookii @xcite . both automatic evaluation and human evaluation show that mart generates more satisfying results than previous lstm-based approaches @xcite @xcite and transformer-based approaches @xcite . in particular, mart can generate more coherent (e.g., coreference and order), less redundant paragraphs without losing paragraph accuracy (visual relevance). one line of work @xcite addresses the video paragraph captioning task by decoding each video event segment separately into a sentence. the final paragraph description is obtained by concatenating the generated single sentence descriptions. though individual sentences may precisely describe the corresponding event segments, when put together the sentences often become inconsistent and redundant. another line of works @xcite use the lstm decoder's last (word) hidden state from the previous sentence as the initial hidden state for the next sentence decoding, thus enabling information flow from previous sentences to subsequent sentences. while these methods have shown better performance than their single sentence counterpart, they are still undesirable as the sentence-level recurrence is achieved at word-level, and the context history information quickly decays due to vanishing gradients @xcite problem. additionally, these designs also have difficulty modeling long-term dependencies @xcite . in comparison, the recurrence in mart resides in the sentence or segment level and is thus more robust to the aforementioned problems. advinf @xcite augments the above lstm word-level recurrence methods with adversarial inference, using a set of separately trained discriminators to re-rank the generated sentences. the techniques in advinf can be viewed as an orthogonal way of generating captions with better quality. transformers transformer @xcite is used as the basis of our approach. different from rnns (e.g., lstm @xcite , gru @xcite , etc) that use recurrent structure to model long-term dependencies, transformer relies on self-attention to learn the dependencies between input words. transformers have proven to be more efficient and powerful than rnns, with superior performance in many sequential modeling tasks, including machine translation @xcite , language modeling/pre-training @xcite @xcite and multi-modal representation learning @xcite @xcite . additionally, @xcite have shown that a transformer model can generate better captions than the lstm model. however, transformer architectures are still unable to model history information well. this problem is identified in the task of language modeling as context fragmentation @xcite , i.e., each language segment is modeled individually without knowing its surrounding context, leading to inefficient optimization and inferior performance. to resolve this issue, transformer-xl @xcite introduces the idea of recurrence to the transformer language model. specifically, the modeling of a new language segment in transformer-xl is conditioned on hidden states from previous language segments. experimental results show transformer-xl has stronger language modeling capability than the non-recurrent transformer. transformer-xl directly uses all the hidden states from the previous segment to enable recurrence. in comparison, our mart uses highly summarized memory states, making it more efficient in passing useful semantic or linguistic cues to future sentences. ## methods though our method provides a general temporal multi-modal learning framework, we focus on the video paragraph captioning task in this paper. given a video v , with several temporally ordered event segments [e 1 , e 2 , ..., e t ], the task is to generate a coherent paragraph consisting of multiple sentences [s 1 , s 2 , ..., s t ] to describe the whole video, where sentence s t should describe the content in the segment e t . in the following, we first describe the baseline transformer that generates sentences without recurrent architecture, then introduce our approach -memory-augmented recurrent transformer (mart). besides, we also compare mart with the recently proposed transformer-xl @xcite in detail. ## experiments we conducted experiments on two popular benchmark datasets, activitynet captions @xcite and youcookii @xcite . we evaluate our proposed mart and compare it with various baseline approaches. ## conclusion in this work, we present a new approach -memory-augmented recurrent transformer (mart) for video paragraph captioning, where we designed an auxiliary memory module to enable recurrence in transformers. experimental results on two standard datasets show that mart has better overall performance than the baseline methods. in particular, mart can generate more coherent, less redundant paragraphs without any degradation in relevance. vanilla transformer he continues speaking while holding the violin and showing how to play his hands. he continues playing the instrument while looking down at the camera. he continues playing the violin and then stops to speak to the camera.