Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference (2021)

Abstract: This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form (subject, predicate, object) that can be easily translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.

## introduction multimodal understanding tasks @xcite @xcite have attracted rapidly growing attention from both computer vision and natural language processing communities, and various multimodal tasks combining visual and linguistic reasoning, such as visual question answering @xcite and image caption generation @xcite , have been introduced. with the development of the multimodal structured datasets such as visual genome @xcite , recent studies have been tackling a complex multimodal inference task such as visual reasoning @xcite and visual-textual entailment (vte) @xcite , a task to judge if a sentence is true or false under the situation described in an image. the recently proposed multimodal logical inference system @xcite uses first-order logic (fol) formulas as unified semantic representations for text and image information. the fol formulas are structured representations that capture not only objects and their semantic relationships in images but also those complex expressions including negation, quantification, and nu-figure 1 : inference example between a video and sentences. the description of this video is: the woman tried to put on her outerwear though she could not, because its zipper was not open completely. as an example, consider a video-and-language inference example in figure 1 . this video consists of scene1, where the sentence the woman puts on her outerwear is true, and scene2, where the sentence the woman takes off her outerwear is true. note that the entire video represents richer information as expressed by the sentence the woman tries to put on her outerwear. to judge whether this sentence is true, it is not enough to simply combine two actions, putting on outerwear and taking off outerwear. to capture this dynamic aspect of human action, it is necessary to take into account the information expressed by intentional phrases such as trying to put on outerwear. towards such a complex multimodal inference between video and text, we build a new japanese video-and-language dataset with human actions. we annotate videos with action labels written in triplets of the form ⟨subject, predicate, object⟩, where object can be empty (indicated by ϕ). action labels contain not only basic expressions such as ⟨person, run, ϕ⟩ and ⟨person, hold, cup⟩, but also expressions including intentional phrases such as ⟨person, try to eat, food⟩. an advantage of using triplets ⟨subject, predicate, object⟩ is that a triplet itself can serve as the semantic representation of a video and can be translated into logical formulas (see section 3). this paper introduces a method to create a video-and-language dataset involving aspectual and intentional phrases. we collect a preliminary dataset labeled in japanese for human actions. we also analyze to what extent our dataset contains various aspectual and intentional phrases. our dataset will be publicly available at https://github.com/rikos3/humanactions . ## related work there have been several efforts to create human action video datasets in the field of computer vision. charades @xcite ) contains 9,848 videos of daily activities annotated with free-text descriptions and action labels in english. charades sta @xcite is a dataset built by adding sentence descriptions with start and end times to the charades dataset. for japanese video datasets, stair actions @xcite is a dataset that consists of 63,000 videos with action labels. each video is about 5 seconds and has a single action label from 100 action categories. action genome @xcite ) is a large-scale video dataset built upon the charades dataset, which provides action labels and spatiotemporal scene graphs. violin @xcite introduces a multimodal inference task between text and videos: given a video with aligned subtitles as a premise, paired with a natural language hypothesis based on the video content, a model needs to judge whether or not the hypothesis is entailed by the given video. the violin dataset mainly focuses on conversation reasoning and commonsense reasoning, and the dataset contains videos collected from movies or tv shows. compared to the existing datasets, our dataset is distinctive in that action labels are written in structured representations ⟨subject, predicate, object⟩ and contain various expressions such as continue to eat and try to close that support complex inference between videos and texts. fol structures (also called first-order models) are used to represent semantic information in images @xcite ). an fol structure for an image is a pair (d, i) where d is a domain consisting of all the entities occurring in the image, and i is an interpretation function that describes the attributes and relations holding of the entities in the image @xcite . to extend fol structures for images to those for videos, we add to fol structures a set of scenes @xmath0, s 2 , . . . , s n } that makes up a video, ordered by the temporal precedence relation. this structure may be considered as a possible world model for standard temporal logic @xcite . thus, a video is represented by (s, d, i), where s is a set of scenes linearly ordered by the temporal precedence relation, d is a domain of the entities, which is constant in all scenes, and i is an interpretation function that assigns attributes and relations to the entities in each scene. we assign personal ids (d 1 , d 2 , . . . , d n ) to people appearing in each scene. since the purpose of our dataset is to label human actions, we assign ids to people, but not to non-human objects. to facilitate the annotation of the attributes and relations holding of the entities in each scene, we use triplets of the form ⟨subject, predicate, object⟩ given to each scene s i as action labels, where object may be empty. this form itself can be seen as a semantic representation of videos. furthermore, it can also be translated into an fol formula, in a similar way to the standard translation of modal logic to fol @xcite . the following examples show a translation from triplets in scenes into fol formulas. here each predicate has an additional argument for a scene variable. (1) means that the entity d 1 runs in scene s 1 ; each triplet can be translated into an fol formula by using this method and thus serve as a semantic representation of a video usable in the semantic parser and inference system for the vte task presented in @xcite . though it is left for future work, the dataset in which each scene of a video is annotated with triplets will be useful to evaluate the vte system for videos. ## conclusion we introduce a video-and-language dataset with human actions for multimodal inference. we annotate human actions in videos in the free format and collect 1,942 action categories for 200 videos. our dataset contains various action labels for videos, including those predicates characteristic of videos such as predicates for utterance, predicates for intention and attitude, and aspectual predicates. in future work, we analyze recent ac-tion recognition models using action genome @xcite with our dataset. we will also work on building a multimodal logical inference system between texts and videos.