Multimodal Intent Discovery from Livestream Videos (2022)

Abstract: Individuals, educational institutions, and businesses are prolific at generating instructional video content such as “how-to” and tutorial guides. While significant progress has been made in basic video understanding tasks, identifying procedural intent within these instructional videos is a challenging and important task that remains unexplored but essential to video summarization, search, and recommendations. This paper introduces the problem of instructional intent identification and extraction from software instructional livestreams. We construct and present a new multimodal dataset consisting of software instructional livestreams and containing manual annotations for both detailed and abstract procedural intent that enable training and evaluation of joint video and text understanding models. We then introduce a multimodal cascaded cross-attention model to efficiently combine the weaker and noisier video signal with the more discriminative text signal. Our experiments show that our proposed model brings significant gains compared to strong baselines, including large-scale pretrained multimodal models. Our analysis further identifies that the task benefits from spatial as well as motion features extracted from videos, and provides insight on how the video signal is preferentially used for intent discovery. We also show that current models struggle to comprehend the nature of abstract intents, revealing important gaps in multimodal understanding and paving the way for future work.

## introduction instructional videos have become increasingly ubiquitous as users generate diverse "how-to", diy, and tutorial videos. a pew research center 2018 survey of u.s. adult youtube users @xcite found that over half of surveyed users use video content to learn how to do things they had not done before. these instructional videos convey both abstract and specific intent for physical tasks such as cooking where e.g., an abstract culinary intent is "let's bring out the flavor" and a detailed intent is "add a pinch of nutmeg". thus, a key task in instructional video understanding is to discover both abstract and detailed intents. by discovering these intents, we can enable or improve important tasks such as semantic indexing of videos @xcite , knowledge graph creation for video search and recommendations @xcite , intent highlighting, and video summarization @xcite . an important domain with rich and complex examples of both abstract and detailed intent types are software training videos for creative tasks such as making photo or video effects. these types of software training videos have been shown to be effective for enhanced learning @xcite and are also considered a valuable resource in the era of online learning @xcite . existing video and phrase datasets such as howto100m @xcite cover a wide variety of tutorials for visual tasks demonstrated by humans; however, software-based instructional videos are not a part of such corpora. hence, in this paper, we present a new corpus of software-instructional videos containing instructional intents, which are derived from behance livestreams demonstrating the use of adobe photoshop software. 2 intent detection has been well-studied in dialogue systems @xcite , but is less explored for instructional video content, especially emerging livestream content @xcite . while rich in complex procedural instruction and intent, the interactive and social nature of livestreams poses unique challenges. analyzing language features alone will provide only limited information about the actual instructional intent and the tools and commands used. for instance, the phrase "flipping the canvas" in "are you flipping the canvas?" indicates a tool intent, but a closer look at the video clip reveals that it is in fact part of livestream chit-chat and does not take place on-screen. incorporating both language and video modalities can enhance intent extraction of such ambiguous intents. hence, in this paper, we present a new joint language-video intent discovery task and a multimodal dataset consisting of: behance intent discovery, and the behance livestream video and transcript corpus that intents are found in. we frame intent discovery as a sequence labelling task; each sample in the intent discovery dataset contains a transcribed phrase annotated with token-level tags for abstract and detailed intents, and an associated video timestamp. our goal is to predict the instructional intents from the transcript in each video. to perform intent discovery within instructional videos, we propose a multimodal cascaded crossattention model to predict both the abstract and detailed procedural intents that are present. additionally, we use late fusion of multimodal embeddings to prevent the visual modality from overwhelming the textual signal, and show significant improvements on the video-based intent detection task using unimodal and multimodal pretrained models like hero @xcite . further, we compare the performance of various video feature extractors as well as different video lengths, and present benchmark results on the proposed dataset. we find that discovery of tool intents benefit from sparsely-sampled spatial features while creative intents benefit from densely-sampled motion features. in the absence of motion features, most models struggle to utilize the video signal for identification of creative intents. further, visualization of crossattention and visual gate modules in the late fusion model suggests strong and meaningful interaction between the two modalities. our contributions are: intent discovery has been widely studied in the context of dialog modelling and generation wherein it has been framed as a binary or multi-class classification problem. the snips @xcite and atis @xcite datasets consist of concise single-sentence texts containing intents with constrained vocabulary and attributes. several works have explored intent classification of internet posts in the context of racial/radicalized intent @xcite , purchase intents @xcite , discussion forums @xcite and health queries @xcite . @xcite propose open intent discovery with unconstrained vocabulary as a sequence tagging task. using this framework, we present our dataset on instructional intents. in the wake of exploding visual social-media content, several image-based multi-modal intent datasets have been previously proposed. @xcite ; aprosio et al. (2020) study abusive language and hateful intent in memes and photo posts. @xcite explore intent categories derived from social psychology and use object localization to integrate visual context in task models. instagram posts are another interesting source for multimodal content @xcite . we introduce the task of video-based multimodal intent discovery, which has been unexplored. several tasks have been proposed in the recent years to probe joint video and text understanding. @xcite al. (2018); kim et al. (2017); maharaj et al. (2017); jang et al. (2017); tapaswi et al. (2016) and @xcite introduce video-based question answering datasets created from various sources of creative visual content, i.e. movies, tv shows, gifs etc. @xcite and @xcite propose the task of video-moment retrieval and next frame prediction respectively, based on query subtitles, while @xcite present the multimodal version of natural language inference. early models for performing these tasks involve combining pretrained image representations from sparsely sampled videos, and text encodings from pretrained encoders @xcite in architectures for modelling global-local interactions @xcite , temporal localization @xcite , graph-based reasoning @xcite etc. more recent attempts involve pretraining models on large video+text corpora @xcite and finetuning on downstream tasks @xcite @xcite @xcite . we explore late-fusion of video and text embeddings @xcite for intent detection in pretrained and non-pretrained multimodal settings. ## problem setting: intent discovery from livestreams in our setting, each video captures a behance livestream in which an instructor demonstrates the steps needed to accomplish various image editing or compositing tasks. we specifically focus on rich creative instructional or tutorial livestreams that teach photo editing and compositing methods using an image application such as photoshop, which consists of over 1,300 basic menu commands and subcommands, tool icons, panels, and galleries. the livestream itself consists of a screencast of the instructor's application software, a smaller video window showing the instructor, a time-coded transcript of the dialog within the session, and a tool timeline which is a time-coded log of the specific application tools used during the livestream. given the transcript of an instructional video, the video itself, and optional tool usage (tu) information from the tool timeline, fig. 1 shows examples of the creative intents (ci, shown at 01:36.16 and 01:37:09) and tool intents (ti, shown at 01:36:21, 01:36:29 and 01:36:46) we seek to discover. further, we wish to combine joint language and video knowledge to gain improvements in detecting candidate intents that are false positives such as the text at 01:37:09, which is only a parenthetical comment by the instructor. ## behance datasets dataset collection. we first obtain 2,049 videos along with their transcripts and tool timelines from the behance platform. the tool timeline contains a time-stamped record of the tools used in the software during the tutorial. the average session length is 80 minutes with an average of 587 transcribed phrases per session (see in order to prepare the intent discovery dataset, we extract candidate intent phrases from the transcripts of the behance livestream corpus. following @xcite , we define an intent as a text phrase consisting of: (i) an action word or phrase, which constitutes a definite task, goal or activity and (ii) an object, which represents those words or phrases that the action is going to act or operate upon. we generate the dependency graph of sentences, and extract the verb node as action and the direct object of the verb as the object, along with all other children nodes (see example in fig. 2 ). 3 through manual analysis, we identified two major categories of meaningful intent: tool and creative. tool intents are low-level intents that can be typically mapped to a single tool in the software. creative intents are abstract intents used to describe a high-level creative goal that consist of a complex set of actions or tool intents. for instance, in fig. 3 , "make a new layer" is a tool intent that can be mapped to the tool create layer, while "add more plants and stuff" is a creative intent. all other intents in the corpus, predominantly from chit-chat statements, are irrelevant to our task. we frame the task of intent discovery as a sequencetagging problem and tag the intent phrases within each sentence with iob (inside, outside, beginning) span annotations for the two classes: tool and creative intents. each sample consists of a timestamped sentence with span annotations and the video session it is extracted from. based on the above defined framework, we col- in our experiments, we explore varying clip duration and empirically choose a window of 10 seconds (±5s) around the sentence's timestamp (see sec. 8.2). the full video sessions are released for further research. ## methods for intent discovery intuitively, as in a lot of instructional sources like text books, the text or audio serves as the primary mode of high-level information transfer, while the video/image signal provides detailed context or demonstration. thus, we start our exploration using text models, which are built on two pretrained models: roberta @xcite and gpt2 @xcite . there are several previous works focusing on a limited set of intents @xcite , and thus, treat the problem of intent discovery as a classification problem. in our case, given the vast possibilities of potential intents in our sources, we cast the problem as a span detection problem, and design our models accordingly. ## experiments evaluation. since the transcribed phrases in behance livestreams are the result of an automatic speech recognition (asr) system, the exact span match metrics might be distorted by asr errors. hence, we use a more lenient 75% partial matchbased precision/recall/f-score metric i.e., if there is more than 75% overlap between the ground truth and predicted span, we consider it as a match. video representations. we experiment with 3d resnext-101 @xcite (f @xmath0, slowfast @xcite (clip @xmath1. models. we use the roberta large @xcite models for the unimodal experiments, as well as the multimodal experiments that are based on unimodal pretrained models. we use the pretrained hero @xcite and clipbert @xcite in the remaining experiments; their language encoders are initialized from pretrained roberta base and bert base @xcite models. each model is trained end-toend using fully-supervised training and is subjected to grid-search based hyper-parameter optimization. the best checkpoints are selected based on overall f-score. see appendix for bounds. ## results in this section, we discuss results from various models on the behance intent discovery dataset (see the text baselines. starting with the text-only baselines, we see the best performance from the roberta models, i.e., 58% and 27% partial match f-scores on the tool and creative intents, respectively (rows 2 and 3 in the naïve fusion models. the naïve fusion approach with pretrained roberta yields upto 2% improvement over the text-only baselines. in some cases, such as the 3d resnext representations, this approach degrades the performance, especially in the harder creative intent set. we attribute this to the difference in informativeness between the text and the video signal, as discussed in sec. 5.2. the late fusion models. with the late fusion approach, we see significant improvements in almost all cases. compared to the corresponding naïve fusion models, late fusion models mainly improve precision for tool intents. this result supports our hypothesis that the video signal is most useful as a gate to filter out non-intent candidates from the textual signal. the slowfast representations prove especially beneficial for creative intents, as seen in row 9 in we see similar trends from experiments on the validation set of the behance intent discovery dataset. see results in appendix. ## analysis & discussion in this section, we perform qualitative analysis of the late fusion approach and examine the effect of video clip length. we also discuss a semiautomated approach to creating annotations for intent extraction and use the data in combination with manual annotations for improved results. see appendix for more analyses. ## conclusion in this paper, we explore the novel task of videobased multimodal intent discovery. we present the unlabelled behance livestream corpus consisting of instructional videos for software tools, and the behance intent discovery dataset annotated with tool and creative intents. we propose a late-fusion approach for integration of the video signal with the text signal in a controlled manner for this task, and show significant improvements with unimodal and multimodal pretrained models.