Language Repository for Long Video Understanding (2025)

Abstract: Language has become a prominent modality in computer vision with the rise of LLMs. Despite supporting long context-lengths, their effectiveness in handling long-term information gradually declines with input length. This becomes critical, especially in applications such as long-form video understanding. In this paper, we introduce a Language Repository (LangRepo) for LLMs, that maintains concise and structured information as an interpretable (i.e., all-textual) representation. Our repository is updated iteratively based on multi-scale video chunks. We introduce write and read operations that focus on pruning redundancies in text, and extracting information at various temporal scales. The proposed framework is evaluated on zero-shot visual question-answering benchmarks, showing state-of-the-art performance at its scale. Our code is available at https://github.com/kkahatapitiya/LangRepo.

## introduction video data is central to learning systems that can interact and reason about the world. yet, they also associate with significant challenges such as increased compute requirements and redundant information, to name a few. this is especially critical in long-form videos. even so, recent literature on video understanding have progressed so far, enabling reasoning capabilities in hours-long video streams @xcite , in contrast to very-limited temporal spans (e.g. seconds or minutes) just a few years ago. work on efficient spatio-temporal attention mechanisms @xcite , memory management @xcite , and large-language-models (llms) @xcite @xcite have been key ingredients for such improvements. llms, or more-specifically, vision-largelanguage-models (vllms) have been outperforming pure vision models in recent years in all facets, including image-based reasoning @xcite @xcite , grounding @xcite , video understanding @xcite @xcite , and even robotics @xcite @xcite . the sheer model scale and the vast pretraining data have enabled such frameworks to capture world knowledge and semantics, beyond what is possible with visual data only. besides, the ability to process long context-lengths is also key, as it helps modeling long-term dependencies that are crucial for more-complex reasoning and interactions. however, recent studies show that despite the availability of such context-lengths, the effectiveness of models declines with longer input sequences @xcite . this promotes the search for alternate representations that can compress input language data without losing meaningful information, essentially managing the context utilization of llms. moreover, the use of text (i.e., language) in modeling has shown numerous benefits such as rich semantics @xcite @xcite , ease of information sharing between different specialized-models @xcite or modalities @xcite , and interpretability @xcite . among such, interpretability has a huge societal impact in the age of llms, to manage adversities such as bias @xcite and hallucinations @xcite . simply put, it enables human observers to understand and monitor what really happens within models. hence, interpretable representations have also been of interest to the community, in place of latent representations @xcite . motivated by the above, we introduce language repository (langrepo), an all-textual (hence, interpretable) representation for llms that updates figure 1: comparison with prior-art on egoschema (mangalam et al., 2024) subset: langrepo (ours) outperforms finetuned and zero-shot pipelines of similar scale, while being competitive with much-larger proprietary models. note the log-scale in x-axis. the write operation (write-to-repo) prunes redundant text, creating concise descriptions that keep the context-utilization of llms in-check. its iterative application with increasingly-longer chunks enables it to learn high-level semantics (e.g. long temporal dependencies). the read operation (read-from-repo) extracts such stored language information at various temporal scales, together with other optional metadata within the repository. altogether, our proposed framework is applied to long-term video reasoning tasks such as visual question-answering (vqa) on egoschema @xcite , next-qa @xcite and intentqa @xcite , showing strong performance at its scale (see fig. 1 ). ## related work long-video understanding: video models have progressed over the years, going from primitive recognition tasks @xcite to complex and fine-grained reasoning tasks @xcite over long horizons. both convolutional baselines @xcite and transformer architectures @xcite have explored research directions such as multi-scale representations @xcite , efficiency concerns associated with heavy spatio-temporal computations @xcite , and handling redundant information within video inputs @xcite . more recently, long-video understanding has made a leap forward thanks to benchmark datasets @xcite and model improvements @xcite , validating the importance of modeling complex interactions that happen over long periods of time. still, the sub-par performance of sota models on such benchmarks suggests the potential for further improvements. long-context models: even before the age of llms, models based on convolutions @xcite @xcite , recurrent blocks @xcite @xcite or transformers @xcite have exploited long-term dependencies, especially in the context of video understanding @xcite and robotics @xcite . with the rise of llms, scaling laws have revealed the importance of longer contexts even more @xcite , and, thanks to the breakthroughs such as sparse processing @xcite , caching @xcite , model-sharding @xcite , and efficient attention @xcite , such long-context llms have become a reality. even with very large context lengths, effectively reasoning over longer inputs is challenging @xcite @xcite . this motivates us to think about concise representations that can better-utilize llm context. ## observations on long-range inputs in this section, we investigate how llms perform with increasing inputs lengths (i.e., #tokens). recent llms with very-large context lengths such as gemini-pro-1.5 @xcite (1m tokens) or claude-2.1 (200k tokens), can support extremely long input sequences. yet, when feeding longer inputs, the reasoning capabilities (especially, longterm reasoning) of such models diminish. this behavior is also observed in concurrent work @xcite , and evident in benchmark results of state-of-the-art models @xcite (i.e., better performance with shorter inputs, or fewer video frames). to better investigate this in our setup, we evaluate vqa performance on standard long-term video understanding benchmarks while varying the input length (see ## language repository we present a language repository (langrepo) that iteratively updates with multi-scale descriptions from video chunks. in contrast to external feature banks @xcite or learnable latent memory representations @xcite @xcite , our proposal has a few key advantages: (1) it requires no training (i.e., zero-shot), and (2) it is compatible with both llm-based processing and human interpretation, as it is fully-textual, i.e., it exists in languagespace instead of a latent-space. langrepo consists of two main operations: (1) information writing (write-to-repo), which prunes redundancies and iteratively updates language descriptions based on increasingly-longer video chunks, and (2) information reading (read-from-repo), which extracts preserved descriptions (with any optional metadata) in multiple temporal scales. we show a detailed view of these operations in fig. 2 , and further elaborate in the following subsections. consider a long video that is split in to n nonoverlapping chunks, denoted as @xmath0, • • • , n}. assume that we already have frame or short-clip captions extracted by a vllm (e.g. llava @xcite ) corresponding to such chunks, denoted by altogether, v is represented by n × p captions which we consider as inputs to our framework. ## experiments in our experiments, we rely on captions preextracted using vllms, as given in @xcite . as for the llm, we use either mistral-7b (jiang et al., 2023) (w/ 7b parameters) or mixtral-8×7b (jiang et al., 2024) (w/ 12b active parameters) by default. as the text encoder in similaritybased pruning, we use clip-l/14 @xcite . note that all the models used in our framework are open-source and within a reasonable model-scale, making our work accessible even in academic settings. we do zero-shot inference on all datasets without any finetuning, evaluating the performance on long-form video vqa benchmarks. for evaluations, we consider three challenging long-video vqa benchmarks in our evaluations. egoschema @xcite derived from ego4d @xcite , consists of 3-minute long clips, each with a question and 5 answerchoices. its public validation subset consists of 500 videos, whereas the held-out fullset has 5k videos. next-qa @xcite ) contains videos up to 2 minutes long (at an average of 44 seconds), annotated with 52k open-ended questions and 48k close-ended questions (i.e., multiple-choice with 5 answer options). the questions belong to either temporal, causal, or descriptive categories, evaluating different reasoning capabilities of models. we consider zero-shot evaluation on the validation set. intentqa @xcite is based on the same next-qa videos, yet focuses more on intentrelated questions (e.g. why?, how? or before/after) with a total of 16k multiple-choice questions on 4.3k videos. here, we consider zero-shot setting on the test set. ## conclusion in this paper, we introduced a language repository (langrepo), which reads and writes textual information corresponding to video chunks, as a concise, multi-scale and interpretable language representation, together with additional metadata. both our write-to-repo and read-from-repo operations are text-based and implemented as calls to a backbone llm. our empirical results show the superior performance of langrepo on multiple long-video reasoning benchmarks at its respective scale, while also being (1) less-prone to performance drops due to increasing input lengths, and (2) interpretable, enabling easier human intervention as needed.