A hierarchical approach to vision-based language generation: from simple sentences to complex natural language (2020)

Abstract: Automatically describing videos in natural language is an ambitious problem, which could bridge our understanding of vision and language. We propose a hierarchical approach, by first generating video descriptions as sequences of simple sentences, followed at the next level by a more complex and fluent description in natural language. While the simple sentences describe simple actions in the form of (subject, verb, object), the second-level paragraph descriptions, indirectly using information from the first-level description, presents the visual content in a more compact, coherent and semantically rich manner. To this end, we introduce the first video dataset in the literature that is annotated with captions at two levels of linguistic complexity. We perform extensive tests that demonstrate that our hierarchical linguistic representation, from simple to complex language, allows us to train a two-stage network that is able to generate significantly more complex paragraphs than current one-stage approaches.

## introduction automatic video to text translation is a very difficult, still unsolved problem, at the intersection of natural language processing, machine learning and computer vision which has been widely researched in the last years @xcite . its applications are varied and far reaching, with impact in virtually all aspects of our lives. one of the main challenges is that for a given visual input, there are infinitely many ways to describe it in natural language. every human utterance is unique @xcite , meaning that, while the classic machine learning paradigm assumes, for a given input, a certain "label" from a finite set, language is much more complex and versatile. recurrent neural networks (rnn) establish the current basis for research in video to language translation. using long-short term memory (lstm) @xcite to produce the final text description has become the standard approach. the most common architecture is the sequence-tosequence model @xcite , which usually produces good results. however, it is still unclear what the neural network learns, w.r.t language representation, semantics and diversity. recent works @xcite have clearly pointed out that automated systems have a significantly less diverse vocabulary than humans. it has also been shown that as the sentence becomes longer, the relation to the actual video content drops @xcite . our goal is to study whether a dual, intermediate linguistic description at the level of short sentences, describing simple actions in the form of @xcite could help in the generation of more complex and fluent language. common sense and the challenges encountered in current research, strongly suggest that the transition from vision to language could benefit from a more gradual approach: we should first detect the actors and objects in the scene, then understand and describe their actions in short sentences and only after, put everything together into a larger, coherent story, described in fluent natural language. note that the end description should not necessarily contain the initial simple sentences. thus, we propose a dual-stage video-to-language representation: at the first stage we describe the video as a sequence of events, in the form of simple (subject, verb, object) sentences, which are also well localized in time and space. with a slight abuse of terminology, we refer to such simple sentences that describe simple @xcite actions as svo sentences, even though they may sometimes contain more than three words. then, the second-level description sums up the video content within a coherent paragraph, which describes the visual content in a more elaborate way, by adding to the first level information, causal and semantic relationships between actors and events (without having to repeat the initial svo sentences). in order to train our models to fully capture the role of the simple sentence descriptions within the more complex video to natural language translation problem, we introduce a novel and relatively large videos-to-paragraphs dataset (with appropriate annotations at both linguistic levels), which contains indoor videos from a well-contained "universe". the scenes, actors and their activities are in the context of what typically happens in a classroom. then, we conduct extensive experiments, which show that our two-stage video to language generation system (fig. 2 ) greatly benefits from the intermediate simple sentence representations, which stands between the pure visual interpretation (at the level of single objects or actions as simple "action labels") and the more complex interpretations in natural language (at the level of paragraphs). ## related work the task of video captioning can be split in two stages: understanding the video and generating the text. for generating the text, the most used approach is based on rnns, which uses a video embedding to generate the text @xcite . for creating the embedding, the literature varies from average pooling over frame-based features @xcite to using other various rnns @xcite and attention models @xcite . recently, reinforcement learning was also employed to improve pre-trained models by defining policies and rewards specific to image @xcite @xcite or video @xcite captioning metrics. interestingly enough, recent literature has also shown that the actual encoder architecture affects the results less than the high-level, pretrained image, video and audio features used at input @xcite . this suggests that a more complex, stage-wise encoder-decoder pathway could be needed between vision and language, which further justifies our approach. the idea of generating textual descriptions of simpler events has also been studied @xcite @xcite . however, those methods finish at that first level of short sentences, without producing longer and more complex paragraphs, beyond the simple concatenation of the sentences. moreover, the published datasets do not provide annotations at both levels of simpler sentences and complex paragraphs. another recent idea that is related to ours @xcite is to "divide and conquer" the video content, by first breaking the long caption into many small segments, then employ a sequence to sequence model to generate the final caption. they introduce a transformed version of the charades dataset @xcite . actors were given a script and were asked to record a video that follows it as closely as possible. different than us, they identify actions by class labels (which is a standard approach), not describe such actions by short sentences (which is an intermediate vision-to-language translation approach). moreover, their final captions are significantly shorter than ours (ours: 40.03 vs theirs: 24.13 words on average per caption). thus, they focus on the intermediate prediction of "action labels", combined with a reinforcement learning approach. we generate more sophisticated textual descriptions by going through an initial stage of descriptions in simple sentences and, thus, take a longer step towards bridging the gap between vision and natural language. the lack of other datasets in the literature that provide annotations at both levels of simple sentences and more complex language, limits, for now, the experimental comparisons with methods such as @xcite . however, by introducing our new videos-to-paragraphs dataset, with full annotations at both levels, we are able to compare to published state of the art methods by retraining them on our dataset. our approach is to the best of our knowledge novel. however, there are other methods that proposed to have an intermediate representation @xcite , but, different than our first-level svo represen-figure 1 : examples from videos-to-paragraphs dataset. we present annotations for two different videos. for each we present a few svo (subject, verb, object) events along with: their start frame, end frame, corresponding space-time bounding box that contains the event, and its simple sentence description. at the bottom we present the paragraph-level video description. the paragraphs contain longer and more complex sentences than the svos. tation, that intermediate representation is a hidden rnn state that does not have an explicit, explainable meaning. the two levels we propose are valid linguistic representations at different levels of spatiotemporal context. one is limited to local svos, the other is a compact paragraph at the whole video level. then, other published works predict only simple semantic representations of video content (similar to our proposed svos) @xcite . ## our videos-to-paragraphs dataset below we describe our videos-to-paragraphs dataset and provide details regarding annotations and various statistics of the dataset. ## proposed approach our key idea is to generate natural language descriptions from video input in a hierarchical, stage-wise manner. starting from describing atomic actions in simple sentences, then we generate a more compact and coherent final text, in more complex language. intuitively, we relate this approach to visual object recognition, in which we recognize whole objects by first detecting their parts and sub-parts. we view language generation from a similar, stage-wise perspective. in fact, by describing the world at many different layers of semantic abstraction in space and time, we expect language to be more complex than the visual recognition of physical objects. thus, we propose a hierarchical representation of the vision-language world, in which objects first do simple things described in simple sentences, before relating these simple events and creating larger stories. these stories contain more complex connections, descriptions and explanations, requiring more complex language. we present an overview of our system in fig. 2 . there are two pathways: one that receives the whole video, processes it and creates a video embedding and the other that processes svos and creates a svos embedding. in the end, features from both pathways are combined and fed to a "paragraph description generator" module. next we define each component. ## experimental setup we validate experimentally the benefit of the two-stage text representation and generation approach for video to language translation. thus, we design experiments that highlight the relevance of key elements of our method and use the standard language metrics in the literature: bleu @xcite , rouge @xcite , cider @xcite and meteor @xcite . ## conclusions and future work in this paper we introduced a novel, hierarchical approach to language generation from videos. it starts with generating simple sentences ordered in time, followed by the generation, at a second stage, of a longer and more complex description, which semantically relates the simpler events in space and time using more coherent, natural language. this is to our best knowledge, the first approach of this kind in the literature, with two explainable stages for video to language generation. additionally, we introduce a novel dataset, videos-to-paragraphs, with full annotations at both levels of textual descriptions, again, the first of its kind in the literature. the dataset is instrumental in comparing our work to the standard, direct video-to-language published approaches, which are not able to generate longer coherent paragraphs. our extensive experiments strongly suggest that the idea of generating explicit language in several phases, going from simpler sentences, to longer paragraphs and then to complex stories, could offer an interesting direction towards strong common vision and language representations.