GRIZAL : Generative Prior-guided Zero-Shot Temporal Action Localization (2024)

Abstract: Zero-shot temporal action localization (TAL) aims to temporally localize actions in videos without prior training examples. To address the challenges of TAL, we offer GRIZAL, a model that uses multimodal embeddings and dynamic motion cues to localize actions effectively. GRIZAL achieves sample diversity by using large-scale generative models such as GPT-4 for generating textual augmentations and DALL-E for generating image augmentations. Our model integrates vision-language embeddings with optical flow insights, optimized through a blend of supervised and self-supervised loss functions. On ActivityNet, Thumos14 and Charades-STA datasets, GRIZAL greatly outperforms state-of-the-art zero-shot TAL models, demonstrating its robustness and adaptability across a wide range of video content. We will make all the models and code publicly available by open-sourcing them.

## introduction temporal action localization (tal) seeks to accurately identify specific actions occurring within extensive, unedited videos. its applications include real-time surveillance for security, improving sports training with in-depth reviews of player actions, and optimizing video content organization by enabling effective search and indexing capabilities. the deep-learning techniques have achieved a significant milestone for tal, however, these techniques necessitate training on comprehensive datasets for optimal performance. given the challenges of gathering exhaustive, annotated videos, some tal models tend to misidentify actions not encountered during training. to address this challenge, zero-shot learning seeks to identify actions without prior exposure to labeled instances of those support for this work was provided by science and engineering research board (serb) of india, under the project crg/2022/003821 actions during training. these techniques leverage semantic linkages and incorporate pre-trained visual-language (vil) models to recognize actions by comparing the semantic similarity between action descriptions and video content. recent methods like stale @xcite propose a parallel localization and classification architecture. unloc @xcite introduces an end-to-end trainable one-stage approach, starting directly from a clip two-tower model. existing self-supervised learning @xcite @xcite literature emphasizes the significance of augmentations for achieving generalized representation through diversity. both stale and unloc use learned text encoders only to get the representation whereas grizal uses multi-modal models like gafnet. stale and un-loc do not use any generative models to generate new data to use as external augmentations. hence, their performance is inferior to the methods that utilize external augmentations @xcite . prevailing tal methods predominantly rely on either retrieval-augmented techniques @xcite or stochastic-augmented training approaches @xcite @xcite . for example, @xcite incorporates traditional training augmentation with a non-parametric retrieval component, while @xcite applies transformations without explicit dependence on pre-existing samples. this limited sample diversity leads to 1) overcomplete representation stemming from semantic inconsistency. this occurs when varied visual representations of the same action impede the model's generalization. 2) undercomplete representation results from a lack of contextual understanding. here, the meaning of an action varies based on the context, leading to different interpretations. as shown in fig. 1 , both grizal (without gpt-4 text) and grizal (without dall-e image) suffer from these issues. similarly, stale suffers from over-complete representation. some methods (kalakonda et al., 2023; ju et al., 2023) utilize generative models to generate text and images related to action, which act as external augmentations. these methods pass generated content through a pre-trained model to get rich feature representations and utilize them for specific tasks. however, since these extracted features are not passed through any additional learnable layers, the model does not get fine-tuned to the specific task at hand. to address these challenges, we propose grizal. we demonstrate that incorporating diverse and contextually rich augmentations in tal results in more discriminative and controlled representations. as shown in fig. 1 , the full grizal model precisely localizes temporal action boundaries. clearly, diversity mitigates over-complete and under-complete representation issues. to achieve sample diversity, we leverage large-scale generative models such as gpt-4 for generating textual augmentations and dall-e for generating image augmentations. these generated images and texts are passed through a pre-trained multimodal model to get a rich feature map. this feature map is passed through additional learnable layers to blend these features effectively. our contributions are: ## related work there have been numerous efforts at the intersection of computer vision and natural language processing. radford et al. @xcite introduced clip, a large-scale pretrained vision-language (vil) model, trained using a contrastive frame encoder frame encoder raft mlp mlp transformer f-transformer optical video frames label @xmath0: gafnet mlp transformer mlp and @xmath1. ) z ft m( . ) m( . ) image generator text generator tg( . ) vision language embedding (vle) block main-stream block optical flow embedding(ofe) block z m weights are frozen vle block main-stream block ofe block action probability of different frames learning strategy on 400 million image-text pairs. clip demonstrated remarkable zero-shot transferability across 30 classification datasets. this motivated subsequent works to propose enhancements in training strategies, such as coop @xcite and clipadapter @xcite . a similar approach has been explored for videos @xcite . actionclip @xcite applies clip for action localization. the existing supervised learning tal networks are either two-stage @xcite or single-stage networks @xcite . effprompt @xcite introduces a two-stage sequential architecture for zero-shot action localization. it involves generating an action proposal using a pre-trained detector like bmn @xcite , followed by the proposal classification using clip features. we aim to pioneer a proposal-free framework that leverages contextual augmentations and eliminates the reliance on a proposal generation stage. using generative models to augment training data has significantly enhanced model generalization. for example, @xcite have incorporated gan-derived synthetic data into training sets. dall-e @xcite can create diverse images from textual prompts, while gpt-4 (openai, 2023) excels in language understanding. traditional clip-based models for action recognition often rely on manual augmentation or retrieval-based hard negatives, which can constrain representation quality. our approach leverages generative models to create contextually relevant augmentations tailored to specific modalities. ## grizal: our proposed method problem formulation: consider a dataset d, composed of two disjoint subsets: the training set d train and the validation set d val . each subset has a collection of (v ,al,f ), where: v is a video sequence, al denotes the action label corresponding to v , and @xmath2, f 2 , . . . , f n } represents binary annotations for each frame within v . the annotation for frame f i is defined as f @xmath3, and f @xmath4. ## experimental setup we perform experiments on activitynet-v1.3, thumos14 and charades-sta datasets and use the dataset splits proposed by @xcite . we evaluate two scenarios. in open-set scenario, we have d train ∩d @xmath5, i.e., action categories for training and validation are disjoint. here, we evaluate two splits, viz., 75%:25% and 50%:50%, of action categories in training and testing. in closed-set scenario, d @xmath6. more details are provided in the appendix. ## conclusion we introduce grizal, a novel framework designed for zero-shot action localization. grizal can synthesize information from vision-language embeddings and optical flow. this novel approach is designed to recognize and interpret actions in videos by using multimodal clues without being exposed to the action labels during training. the strategic integration of multimodal embeddings and the tailored use of loss functions contribute to the model's exceptional performance. additional modalities. currently, grizal is limited to the vision-language modality. in future versions, we will adapt our model to other modalities, such as audio, for sound localization. as shown in fig. s .5, the feature vector z rgb from the mainstream block is first subjected to a multiheaded self-attention (msa) mechanism. this self-attention mechanism allows the model to capture relationships between different positions in the sequence, enhancing its ability to understand the temporal dynamics of the video. after residual connection and layer normalization, the output is combined with the z cls vector from the vle block through pointwise addition. this helps the model by providing more context relevant to the action label. following the integration of these vectors, the fast fourier transform (fft) is applied to transform the features into the frequency domain. this elucidates the distribution of features across various frequencies. following fft, an mlp layer captures distribution patterns within the frequency domain. this allows the model to emphasize important properties while downplaying less important ones. subsequently, an inverse fft (ifft) operation is performed to map the features back to the spatial domain. the concluding msa module in our novel f-transformer utilizes the optical flow feature (z optical ) as both key and value vectors, with the preceding feature map serving as the query vector. this configuration enables the msa to synthesize attention-pooled characteristics conditioned reciprocally across modalities. in particular, the optical flow features outline the action dynamics in the video frames, highlighting areas of motion and stillness. at the same time, the visual-textual embeddings provide a visual and contextual comprehension of the action label, assisting in accurately identifying frames where the action of interest takes place. the features are then passed through additional msa and mlp layers, with layer normalization (lnorm) applied after each operation to stabilize the learning process. the final output of the f-transformer block is denoted as z f t , which incorporates both the detailed spatial features and the refined frequency domain features. doing so captures a comprehensive understanding of the video's content, both in terms of the global context provided by the self-attention and the local, detailed motion patterns highlighted by the optical flow. this multifaceted approach allows for a more accurate and robust action localization within videos. s.2 details of experimental platform s.2.1 datasets 1) the activitynet-v1.3 dataset (caba heilbron et al., 2015), comprises 19,994 videos spanning 200 action classes. to adhere to the standard evaluation protocol, we partitioned the videos into training, validation, and testing subsets, maintaining a ratio of 2:1:1. 2) the thumos14 dataset (idrees et al., 2017) includes 200 validation videos and q k v msa add + lnorm fft mlp ifft add + lnorm msa k v q add + lnorm mlp add + lnorm z ft point-wise addition z rgb z optical z cls figure s.5: f-transformer 213 testing videos distributed across 20 action categories. notably, thumos14 provides labeled temporal boundaries and action classes for each video.