Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on Y ou T ube (2020)

Abstract: Pretraining from unlabelled web videos has quickly become the de-facto means of achieving high performance on many video understanding tasks. Features are learned via prediction of grounded relationships between visual content and automatic speech recognition (ASR) tokens. However, prior pretraining work has been limited to only instructional videos; a priori, we expect this domain to be relatively “easy:” speakers in instructional videos will often reference the literal objects/actions being depicted. We ask: can similar models be trained on more diverse video corpora? And, if so, what types of videos are “grounded” and what types are not? We fit a representative pretraining model to the diverse YouTube8M dataset, and study its success and failure cases. We find that visual-textual grounding is indeed possible across previously unexplored video categories, and that pretraining on a more diverse set results in representations that generalize to both non-instructional and instructional domains.

## introduction self-supervised pretraining approaches have recently been adapted to web videos @xcite @xcite @xcite ; the resulting models have achieved state-of-the-art performance on a wide range of video understanding tasks, e.g., dense caption generation, action localization, etc. in general, the pretraining step requires a large, unlabelled corpus of web videos. the training objective aligns visual content (i.e., video segments) with automatic speech recognition (asr) tokens, and the resulting representations are fine-tuned for downstream tasks. the assumption underlying this family of approaches is that, in the pretraining corpus, spoken words have some consistent, grounded relationship with the temporally corresponding visual content. however, in contrast to the highly diverse corpora utilized for text-based pretraining @xcite , pretraining for web videos (so far) has been limited to instructional videos. this domain restriction is motivated by the commonly accepted notion that "procedural knowledge tends to be inherently multimodal" @xcite . we expect that the semantic information in video frames and asr tokens is readily correlated in instructional videos. but corpus diversity brings significant benefits: in the text-only case, models can effectively represent diverse real-world entities @xcite precisely because pretraining is not restricted to, e.g., only fictional stories @xcite . in search of more general representations, our main question is: does video-asr pretraining "work" for more diverse pretraining corpora? are certain categories of non-instructional videos "groundable," thus enabling diverse representation learning? or are some types too difficult, only acting as training noise? we conclude that: 1) grounding is indeed possible in a wide range of yet-to-be-computationally-exploited youtube video categories, e.g., walk-throughs, vehicles, tech reviews, etc., with some harder than others; 2) transferable representations can be successfully learned by training on a more diverse set, which may provide more versatility. ## related work asr is known to be a useful signal source in various instructional video understanding tasks @xcite @xcite , e.g., action detection/classification @xcite @xcite , segmentation/captioning @xcite , and instruction alignment @xcite . a number of multimodal instructional video datasets have been proposed @xcite @xcite . a notable recent example of work addressing a non-instructional video corpus is @xcite , who analyze grounded-ness in lifestyle vlogs. @xcite highlight the difference between keyword search vs. implicitly mining action data of interest from a broader corpus (e.g., @xcite ; @xcite ). operational grounding. our work builds upon prior operational notions of grounding: if an algorithm is able to consistently predict specific visualtextual relationships, then that relationship is said to be "grounded" @xcite @xcite @xcite . @xcite , for example, examine an image+text corpus and rank "substrings of text by how well their occurrence can be predicted from visual features." one shortcoming of any model-based operationalization of "grounding" is that only positive instances of groundedness can be identified: if one model fails to ground something, perhaps a better model could have. ## video-asr pretraining + our model recent work in designing pretraining objectives: 1) assumes that asr tokens have, on average, some correspondence to temporally co-occurring video frames within the same video; and 2) ignores clips that lack asr. we consider a model that encapsulates both of these assumptions. 1 the model is a slight simplification of @xcite , where a joint embedding for the visual content and asr tokens is learned. while more sophisticated methods based on selfattention models have since been examined (e.g., zhu and yang (2020)), joint embedding models are still performant and offer greater interpretability, thus enabling our later error analyses. model details. the similarity between clip i and asr caption j, s i,j , is estimated by computing the cosine similarity between their corresponding embeddings in the joint space. joint embedding models are parameterized using gated, multi-layer feedforward networks. the visual features we use as input are: frame-wise 2d inception-v1 pretrained for object detection @xcite and 3d cnn s3d-g features pretrained for action recognition @xcite . the language feature inputs are 300 dimensional vectors per word-type; these are fine-tuned during the training process. max pooling is used for both token embeddings and for perframe visual features to achieve a single visual and textual embedding for each clip. 2 during training, temporally corresponding (clip, asr caption) pairs are sampled ("positive" cases). for each positive case, a set of mismatched "n egative" cases is also sampled both from other videos and from the same video in equal proportion. in contrast to @xcite , we control for clip length, and sample temporally fixed-length segments. in initial experiments with variablelength segments, we found that our models were capable of "cheating" the grounding task by aligning longer (and shorter, respectively) clips with longer (and shorter) asr captions, largely ignoring content. thus, this simplifying choice makes our error analysis significantly more straightforward, and results in minimal performance change. we use 5 second segments, but results are similar with 10 or 30 second windows (see appendix e). to generate segments, we initially randomly sample 256 per video before discarding ones that have no temporally-accompanying asr. segments may overlap, though results are similar without overlaps (see appendix d). the following hinge loss is minimized for margin δ: we trained with adam (kingma and ba, 2015), a learning rate of .001, and set δ " .1, but didn't undertake significant hyperparameter optimization. we terminate training after 300k steps. ## implications for pretraining while we've thusfar shown that self-grounding is possible for a diverse set of domains, do we gain anything by training on a more diverse corpus? or do difficult-to-ground videos introduce noise and degrade representations for downstream tasks? we compare two versions of our model: one with parameters learned from training on the diverse youtube-600k corpus (m diverse ), and one with parameters learned from a domain-specific corpus of 1m instructional videos (m instructional ). first, we evaluate each model's capacity to localize instructional steps on the crosstask @xcite dataset. m diverse performs admirably, even with significant domain mismatch and fewer pretraining videos: recall drops by only 15% (32.6 ñ 27.4) when swapping from m instructional to m diverse . we next evaluate each model's performance on the same-video clip alignment task over a diverse set of videos: the sample of 6.8k humanannotated videos from the youtube8m validation set. in terms of intra-video auc, m diverse outperforms m instructional on 59% of videos. if we split the data across the "is-it-instructional" human judgements and compare the two models in each subset, m instructional "wins" in 57% of the instructional videos, whereas m diverse "wins" in 65% of non-instructional cases. in short: both models achieve reasonable performance under instructional vs. non-instructional train/test domain mismatch. taken together, this is a promising result for future pretraining work with more diverse corpora: at least for these evaluations, good performance on an instructional video grounding task is still possible under domain shift. and while the comparison of intra-video auc is not necessarily definitive, it suggests that diverse corpora may provide more versatility, and we look forward to exploring this further in future work. ## conclusion peeking through the lens of a joint embedding model, we probe into learning visual-textual grounding over a more diverse corpus of youtube videos vs. prior work. we find that learning visual-textual grounding is possible across many yet-to-be-explored categories of youtube videos, and that it's possible to learn generalizable representations from a more diverse video set.