Retrieval-augmented Video Encoding for Instructional Captioning (2023)

Abstract: Instructional videos make learning knowledge more efficient, by providing a detailed multimodal context of each procedure in instruction.A unique challenge posed by instructional videos is key-object degeneracy, where any single modality fails to sufficiently capture the key objects referred to in the procedure. For machine systems, such degeneracy can disturb the performance of a downstream task such as dense video captioning, leading to the generation of incorrect captions omitting key objects. To repair degeneracy, we propose a retrieval-based framework to augment the model representations in the presence of such key-object degeneracy. We validate the effectiveness and generalizability of our proposed framework over baselines using modalities with key-object degeneracy.

## introduction instructions, which provide detailed information about the procedures required to achieve the desired goal, are a central part of how humans acquire procedural knowledge. instructions decompose a sequence of complex procedures into key objects and the associated actions expressed as verbs. as machine systems increasingly aim to provide realworld utility for humans, their ability to translate human goals into natural language instructions to follow becomes essential @xcite . in this light, instructional captioning, summarizing instructional videos into a set of succinct instructions, is thus an important component of enabling the distillation of human-level procedural knowledge to machines. for instructional captioning, we focus on the task of dense video captioning (dvc) @xcite which aims to produce a precise set of instructions from visual input (e.g. instructional videos). for example, to illustrate the procedure * corresponding author. ## preliminaries and related work we first introduce our target domain, namely, instruction, and its representations and previous research on their characteristics ( §2.1). our goal is to improve the encoding of video frame g ( §2.2). then we provide a concise overview of our downstream task, dvc ( §2.3). ## our approach building on preliminaries, we now describe our retrieval augmented encoding framework in detail. first, we explain how instructional scripts can contribute to repairing the degeneracy ( §3.1). our framework combines a cross-modal tdr module ( §3.2), which can aggregate the key objects across frames ( §3.3), to build robust multi-modal representations which repair key-object degeneracy. ## results we now present our experimental results, aiming to address each of the following research questions: rq1: is our cross-modal retrieval using joint image-text query more effective than standard retrieval approaches for tdr? rq2: does kofcl address key-object degeneracy in tdr, and help the retriever to build a robust understanding of key objects? rq3: does retrieval-augmentation using procedural sentences improve dvc by repairing key-object degeneracy? additionally, we compare a straightforward method to repair degeneracy, by disambiguating pronouns in transcripts. following previous work @xcite , we use a co-reference module @xcite to convert transcripts into their disambiguated versions, τ j i . interestingly, we observe a degradation of tdr in every metric. we hypothesize that the co-reference resolution introduces noise from several sources, including the module's inaccuracy itself, but also incorrect pronoun resolution using key objects belonging to other, adjacent procedures. ## conclusion we proposed retrieval-augmented encoding, to complement video frames, by repairing degeneracy and considering correlations between steps. our evaluation results validated that our proposed framework improves existing dvc systems significantly.