Z - GMOT : Zero-shot Generic Multiple Object Tracking (2024)

Abstract: Despite recent significant progress, Multi-Object Tracking (MOT) faces limitations such as reliance on prior knowledge and predefined categories and struggles with unseen objects. To address these issues, Generic Multiple Object Tracking (GMOT) has emerged as an alternative approach, requiring less prior information. However, current GMOT methods often rely on initial bounding boxes and struggle to handle variations in factors such as viewpoint, lighting, occlusion, and scale, among others. Our contributions commence with the introduction of the Referring GMOT dataset a collection of videos, each accompanied by detailed textual descriptions of their attributes. Subsequently, we propose Z-GMOT, a cutting-edge tracking solution capable of tracking objects from never-seen categories without the need of initial bounding boxes or predefined categories. Within our Z-GMOT framework, we introduce two novel components: (i) iGLIP, an improved Grounded language-image pretraining, for accurately detecting unseen objects with specific characteristics. (ii) MA-SORT, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking objects with high similarity. Our contributions are benchmarked through extensive experiments conducted on the Referring GMOT dataset for GMOT task. Additionally, to assess the generalizability of the proposed Z-GMOT, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task. Our dataset, code, and models are released at: https://fsoft-aic.github.io/Z-GMOT

## introduction multiple object tracking (mot) @xcite @xcite @xcite @xcite @xcite 2022a; os-gmot z-gmot (ours) human detector by yolo fully trained on coco dataset mot initial bounding boxes , track athletes in red uniform on the running track request: track only athletes in red uniform on the running track one-shot object detection figure 1 : high-level comparison between our z -gmot with conventional mot and one-shot generic mot (os-gmot) for the task of tracking athletes in red uniforms on a running track. 1 st row: mot, being a fullysupervised method, using yolox (trained on coco) and oc-sort (trained on dancetrack) attempts to detect and track all people in the scene with high false positive (fps). 2 nd row: os-gmot is based on an initial bounding box and utilizes an mot tracker (e.g. os-sort in this case). while reducing the number of fps, os-gmot heavily relies on the initial bounding box, leading to variations in results with different bounding boxes and a high number of false negatives (fns). 3 rd row: our z -gmot including: (i) iglip effectively detects objects without the need for prior training or initial bounding boxes, and (ii) ma -sort efficiently associates objects with high visual similarity. @xcite aims to recognize, localize and track dynamic objects in a scene. it has become a cornerstone of dynamic scene analysis and is essential for many important real-world applications such as surveillance, security, autonomous driving, robotics, and biology. however, current mot methods suffer from several limitations: they heavily depend on prior knowledge of tracking targets, requiring large labeled datasets; they struggle with tracking objects of unseen or specific categories; they are limited in handling objects with indistinguishable appearances. in contrast to mot, generic multiple object tracking (gmot) @xcite seeks to alleviate these challenges with reduced prior information. gmot is tailored to track multiple objects of a shared or similar generic type, offering applicability in diverse domains like annotation, video editing, and animal behavior monitoring. conventional gmot methods @xcite @xcite adhere to a one-shot paradigm @xcite and employ the initial bounding box of a single target object in the first frame to track all objects belonging to the same class. the conventional one-shot gmot, which is based on one-shot object detection (os-od), is known as os-gmot. however, this approach heavily relies on the starting bounding box and has limitations in accommodating variations in object characteristics, including pose, illumination, occlusion, scale, texture, etc. to overcome the aforementioned limitations of both mot and os-gmot, particularly in the context of tracking multiple unseen objects without the requirement for training examples, we introduce a novel tracking paradigm called zero-shot generic multiple object tracking (z -gmot, which leverages recent advancements in vision-language (vl) models. our z -gmot follows the tracking-by-detection paradigm and introduces two significant contributions aimed at enhancing both the object detection stage and object association stage. in the first stage, which involves object detection, we introduce an enhanced version of glip called iglip. while glip has shown promise in detecting objects based on textual description queries, it faces limitations when tasked with detecting multiple objects with subtle distinguishing features. specifically, our observations and empirical experiments have confirmed that it is sensitive to threshold settings, leading to high false positives (fps) at slightly lower thresholds and high false negatives (fns) at slightly higher thresholds. for instance, when asked to identify a "red ball" among multiple balls of various colors, glip may erroneously detect balls of different colors at a slightly lower threshold and miss the red ball when the threshold is increased only slightly. to address it, our proposed enhancement, iglip, incorporates two distinct pathways. one pathway is tailored to handle general object categories like "ball", while the other pathway is dedicated to capturing specific object characteristics, such as the color "red". by integrating these dual pathways, iglip aims to deliver a more accurate and precise object detection process, especially when dealing with multiple generic objects. in the second stage, which involves object association, we propose ma -sort (motion-appearance sort), an innovative tracking algorithm that seamlessly fuses visual appearance with motion-based matching. ma -sort adeptly measures appearance uniformity and dynamically balances the influence of motion and appearance during the association process. figure 1 provides a visual comparison between our z -gmot with conventional mot and os-gmot approaches with the task of tracking athletes in red uniforms on a running track as an example. in this comparison, mot is a fully-supervised learning method that employs yolox object detection @xcite trained on coco dataset @xcite and oc-sort object association @xcite trained on dancetrack dataset @xcite . being a fully-supervised method, mot attempts to detect and track all people in the scene instead of only athletes in red uniforms as requested. as a result, mot generates a high number of fps. in contrast, os-gmot relies on an initial bounding box to detect all requested objects. it also utilizes the robust oc-sort tracker @xcite for object association. while reducing the number of fps, os-gmot heavily relies on the initial bounding box, leading to variations in results with different bounding boxes and a high number of false negatives (fns). different from mot and os-gmot, our z -gmot takes the tracking request in the form of a natural language description as its input to effectively detect and track objects without prior training or initial bounding boxes. our contributions are as follows: ## proposed z -gmot our z -gmot framework follows the tracking-bydetection paradigm which includes the object detection stage and object association one. in the initial stage, we analyze the limitations of glip detector which is our motivation for proposing iglip for detecting effectively generic objects. in the subsequent stage, we introduce ma -sort to adeptly balance between motion cues and visual appearances to improve the association process. ## conclusion & discussion in this study, we present z -gmot, a novel tracking framework capable of tracking diverse objects without relying on labeled data. z -gmot adopts a tracking-by-detection paradigm and offers two key contributions: (i) zero-shot iglip for effective object detection using natural language descriptions and (ii) ma -sort for efficient tracking of visually similar objects within a broader context of generic objects. beyond proposing z -gmot, we also introduce a new refering gmot dataset. we have thoroughly assessed and demonstrated the efficacy and adaptability of z -gmot, not only in the gmot task but also in the mot task. discussion. we utilize glip as our preferred vlm for developing iglip. however, it is important to recognize the rich diversity of vlms available in the field, which opens up exciting avenues for deeper exploration. moreover, in our current study, we have implemented z -gmot exclusively using only textual description object and attributes. nevertheless, our referring gmot dataset offers additional information, such as object_synonyms and other_attributes, which hold great potential for further research, particularly in the context of prompt tuning or prompt engineering. exploring these additional aspects of our referring gmot dataset could lead to enhanced object tracking capabilities as well as other fields such as surveillance, robotics, and animal welfare. we expect our work to inspire future research in the unexplored realm of unseen mot/gmot paradigms, potentially leading to extensions in other tracking scenarios, e.g., open-vocabulary mot/gmot.