Unsupervised Task Graph Generation from Instructional Video Transcripts (2023)

Abstract: This work explores the problem of generating task graphs of real-world activities. Different from prior formulations, we consider a setting where text transcripts of instructional videos performing a real-world activity (e.g., making coffee) are provided and the goal is to identify the key steps relevant to the task as well as the dependency relationship between these key steps. We propose a novel task graph generation approach that combines the reasoning capabilities of instruction-tuned language models along with clustering and ranking components to generate accurate task graphs in a completely unsupervised manner. We show that the proposed approach generates more accurate task graphs compared to a supervised learning approach on tasks from the ProceL and CrossTask datasets.

## introduction tasks in the real-world are composed of multiple key steps with specific dependencies that dictate the order in which they can be performed (e.g., one has to check for breathing before performing cpr). exposing these dependencies between key steps has many downstream applications including assisting human users in troubleshooting and building artificial agents that efficiently learn and perform new tasks. however, information about tasks is typically available in unstructured and noisy form in the wild (e.g., 'how to' descriptions or instructional video transcripts), presenting a major challenge in extracting structured representations. there is a long history of work on reasoning about tasks, events and temporal ordering, broadly referred to as 'script understanding' @xcite of script understanding problems include generating a sequence of steps from a given task description (e.g., bake a cake) @xcite ; sancheti and rudinger, 2021; @xcite and generating flow graphs from goal and event descriptions @xcite . script generation also manifests in interactive settings such as simulated embodied environments where agents are expected to reason about subgoals in order to complete tasks @xcite . many of these prior approaches either fine-tune language models on human-annotated scripts or rely on knowledge encoded in language models to generate scripts. in contrast, we attempt to use pre-trained language models as an information extraction system to perform zero-shot script inference from noisy asr (automatic speech recognition) transcriptions of instructional videos describing a task. our focus in this work is to generate a directed graph that represents dependency relationships between the key steps relevant to a real-world task. figure 1 (a) shows a graph predicted by our approach for performing cpr. an example depen-get out two slices of bread spread peanut butter on one slice spread jelly join the slices "hello and welcome. this is episode number one of traditional school lunches. today i'm going to explain how to make a simple peanut butter and jelly sandwich. before we start making our .." log p @xmath0. so we came out here all the way today to make one of these. that's right a peanut butter and jelly sandwich. there's a lot of science behind .." get bread, peanut butter and jelly spread peanut butter and jelly eat sandwich get bread, peanut butter, and jelly spread peanut butter on one slice spread jelly on the other slice put the slices together get bread, peanut butter, and jelly spread peanut butter on one slice eat the sandwich log p @xmath1, peanut butter, and jelly spread peanut butter on one slice spread jelly on the other slice put the slices together eat the sandwich slice the sandwich in half 5) generate graph eat the sandwich slice the sandwich in half spread jelly on the other slice put the slices together get bread, peanut butter, and jelly spread peanut butter on one slice given multiple text transcripts of a task, we 1) summarize the steps described in the transcript, 2) identify the key steps, 3) re-label summary steps with key steps, 4) rank key step sequences using a language model and 5) consolidate top-k sequences to generate a task graph for the given task. more formally, consider a real-world task τ . we assume that multiple text transcripts t 1 , . . . , t n describing how this task is performed are available. 1 we assume that having access to such multiple transcripts helps robustly identify the dependencies between key steps so that an accurate task graph can be generated. for instance, if step y frequently follows step x, it is highly likely that step x needs to happen before step y (i.e., is a precondition). our goal is to generate a task graph for the given task τ which models these dependencies. in particular, this involves (i) identifying the key steps @xmath2, . . . , k m } relevant to performing the task and (ii) generating a graph with nodes k i and edges representing precondition relationships. our contributions in this work are as follows. ## approach our approach to task graph generation consists of multiple steps, illustrated in figure 2 . first, we use an instruction-tuned language model to generate a summary of steps (in free-form text) from a transcript (section 2.1). given these summary step sequences generated from multiple such transcripts for the task, we identify the key steps relevant to the task using a clustering approach (section 2.2). we then re-label summary step sequences using the identified key steps to obtain key step sequences (section 2.3) and rank them using a language model (section 2.4). finally, we generate a task graph from the key step sequences (section 2.5). ## conclusion this work presented an unsupervised approach to generate task graphs from text transcripts of instructional videos. our framework exploits multiple text transcripts which describe a task in order to robustly identify the key steps relevant to a task and the depencies between these steps. we demonstrated the effectiveness of our approach compared to supervised and unsupervised baselines on instructional video transcripts from the procel and crosstask datasets.