Learning to Segment Actions from Observation and Narration (2020)

Abstract: We apply a generative segmental model of task structure, guided by narration, to action segmentation in video. We focus on unsupervised and weakly-supervised settings where no action labels are known during training. Despite its simplicity, our model performs competitively with previous work on a dataset of naturalistic instructional videos. Our model allows us to vary the sources of supervision used in training, and we find that both task structure and narrative language provide large benefits in segmentation quality.

## learning to segment actions finding boundaries in a continuous stream is a crucial process for human cognition @xcite @xcite . to understand and remember what happens in the world around us, we need to recognize the action boundaries as they unfold and also distinguish the important actions from the insignificant ones. this process, referred to as temporal action segmentation, is also an important first step in systems that ground natural language in videos @xcite . these systems must identify which frames in a video depict actions -which amounts to distinguishing these frames from background ones -and identify which actions (e.g., boiling potatoes) each frame depicts. despite recent advances @xcite , unsupervised action segmentation in videos remains a challenge. the recent availability of large datasets of naturalistic instructional videos provides an opportunity for modeling of action segmentation in a rich task context @xcite @xcite @xcite ; work begun while df was interning at deepmind. code is available at https://github.com/dpfried/action-segmentation . in these videos, a person teaches a specific highlevel task (e.g., making croquettes) while describing the lower-level steps involved in that task (e.g., boiling potatoes). however, the real-world nature of these datasets introduces many challenges. for example, more than 70% of the frames in one of the youtube instructional video datasets, crosstask @xcite , consist of background regions (e.g., the video presenter is thanking their viewers), which do not correspond to any of the steps for the video's task. these datasets are interesting because they provide (1) narrative language that roughly corresponds to the activities demonstrated in the videos and (2) structured task scripts that define a strong signal of the order in which steps in a task are typically performed. as a result, these datasets provide an opportunity to study the extent to which task structure and language can guide action segmentation. interestingly, young children can segment actions without any explicit supervision @xcite , by tapping into similar cues -action regularities and language descriptions @xcite . while previous work mostly focuses on building action segmentation models that perform well on a few metrics @xcite , we aim to provide insight into how various modeling choices impact action segmentation. how much do unsupervised models improve when given implicit supervision from task structure and language, and which types of supervision help most? are discriminative or generative models better suited for the task? does explicit structure modeling improve the quality of segmentation? to answer these questions, we compare two existing models with a generative hidden semi-markov model, varying the degree of supervision. on a challenging and naturalistic dataset of instructional videos @xcite , we find that our model and models from past work both benefit substantially from the weak supervision provided by task structure and narrative language, even on top of rich features from state-of-the-art pretrained action and object classifiers. our analysis also shows that: (1) generative models tend to do better than discriminative models of the same or similar model class at learning the full range of step types, which benefits action segmentation; ## related work typical methods @xcite @xcite @xcite @xcite for temporal action segmentation consist of assigning action classes to intervals of videos and rely on manually-annotated supervision. such annotation is difficult to obtain at scale. as a result, recent work has focused on training such models with less supervision: one line of work assumes that only the order of actions happening in the video is given and use this weak supervision to perform action segmentation @xcite @xcite @xcite . other approaches weaken this supervision and use only the set of actions that occur in each video @xcite , or are fully unsupervised @xcite . instructional videos have gained interest over the past few years @xcite @xcite @xcite since they enable weakly-supervised modeling: previous work most similar to ours consists of models that localize actions in narrated videos with minimal supervision @xcite @xcite . we present a generative model of action segmentation that incorporates duration modeling, narration and ordering constraints, and can be trained in all of the above supervision conditions by maximizing the likelihood of the data; while these past works have had these individual components, they ## the crosstask dataset we use the recent crosstask dataset @xcite of instructional videos. to our knowledge, crosstask is the only available dataset that has tasks from more than one domain, includes background regions, provides step annotations and naturalistic language. other datasets lack one of these; e.g.they focus on one domain @xcite or do not have natural language @xcite or step annotations @xcite ). an example instance from the dataset is shown in figure 1 , and we describe each aspect below. tasks each video comes from a task, e.g. make a latte, with tasks taken from the titles of selected wikihow articles, and videos curated from youtube search results for the task name. we focus on the primary section of the dataset, containing 2,700 videos from 18 different tasks. steps and canonical order each task has a set of steps: lower-level action step types, e.g., steam milk and pour milk, which are typically completed when performing the task. step names consist of a few words, typically naming an action and an object it is applied to. the dataset also provides a canonical step order for each task: an ordering, like a script @xcite , in which a task's steps are typically performed. for each task, the set of step types and their canonical order were hand-constructed by the dataset creators based on section headers in the task's wikihow article. annotations each video in the primary section of the dataset is annotated with labeled temporal segments identifying where steps occur. (in the weak supervision setting, these step segment labels are used only in evaluation, and never in training.) a given step for a task can occur multiple times, or not at all, in any of the task's videos. steps in a video also need not occur in the task's canonical ordering (although in practice our results show that this ordering is a helpful inductive bias for learning). most of the frames in videos (72% over the entire corpus) are background -not contained in any step segment. narration videos also have narration text (transcribed by youtube's automatic speech recognition system) which typically consists of a mix of the regions background pour mixture into pan flip pancake background background ## model our generative model of the video features and labeled task segments is a first-order semi-markov model. we use a semi-markov model for the action segmentation task because it explicitly models temporal regions of the video, their duration, their probable ordering, and their features. 1 it can be trained in an unsupervised way, without labeled regions, to maximize the likelihood of the features. timesteps our atomic unit is a one-second region of the video, which we refer to as a timestep. a video with t timesteps has feature vectors x 1:t . the features x t at timestep t are derived from the video, its narration, or both, and in our work (and past work on the dataset) are produced by pre-trained neural models which summarize some non-local information in the region containing each timestep, which we describe in sec. 6.3. regions our model segments a video with t timesteps into a sequence of regions, each of which consists of a consecutive number of timesteps (the region's duration). the number of regions k in a video and the duration d k of each region can vary; the only constraint is that the sum of the durations equals the video length: k @xmath0. each region has a label r k , which is either one of the task's step labels (e.g., pour milk) or a special label bkg indicating the region is background. in our most general, unconstrained model, a given task step can occur multiple times (or not at all) as a region label in any video for the task, allowing step repetitions, dropping, and reordering. structure we define a first-order markov (bigram) model over these region labels: with tabular conditional probabilities. while region labels are part of the dataset, they are primarily used for evaluation: we seek models that can be trained in the unsupervised and weaklysupervised conditions where labels are unavailable. this model structure, while simple, affords a dynamic program allowing efficient enumeration over both all possible segmentations of the video into regions and assignments of labels to the regions, allowing unsupervised training (sec. 4.1). duration our model, following past work @xcite , parameterizes region durations using poisson distributions, where each label type r has its own mean duration λ r : d k ∼ poisson(λ r k ). these durations are constrained so that they partition the video: e.g., region r 2 begins at timestep d 1 (after region r 1 ), and the final region r k ends at the final timestep t . ## experimental setting we compare models on the crosstask dataset across supervision conditions. we primarily evaluate the models on action segmentation (sec. 1). past work on the dataset @xcite has focused on a step recognition task, where models identify individual timesteps in videos that correspond to possible steps; for comparison, we also report performance for all models on this task. ## discussion we find that unsupervised action segmentation in naturalistic instructional videos is greatly aided by the inductive bias given by typical step orderings within a task, and narrative language describing the actions being done. while some results are more mixed (with the same supervision, different models are better on different metrics), we do observe that across settings and metrics, step ordering and narration increase performance. our results also illustrate the importance of strong baselines: without weak supervision from step orderings and narrative language, even state-of-the-art unsupervised action segmentation models operating on rich video features underperform feature-agnostic baselines. we hope that future work will continue to evaluate broadly. while action segmentation in videos from diverse domains remains challenging -videos contain both a large variety of types of depicted actions, and high visual variety in how the actions are portrayed -we find that structured generative models provide a strong benchmark for the task due to their abilities to capture the full diversity of action types (by directly modeling distributions over action occurrences), and to benefit from weak supervision. future work might explore methods for incorporating richer learned representations both of the diverse visual observations in videos, and the narration that describes them, into such models.