D et GPT : Detect What You Need via Reasoning (2023)

Abstract: In recent years, the field of computer vision has seen significant advancements thanks to the development of large language models (LLMs). These models have enabled more effective and sophisticated interactions between humans and machines, paving the way for novel techniques that blur the lines between human and machine intelligence. In this paper, we introduce a new paradigm for object detection that we call reasoning-based object detection. Unlike conventional object detection methods that rely on specific object names, our approach enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. Our proposed method, called DetGPT, leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user’s instructions and the visual scene. This enables DetGPT to automatically locate the object of interest based on the user’s expressed desires, even if the object is not explicitly mentioned. For instance, if a user expresses a desire for a cold beverage, DetGPT can analyze the image, identify a fridge, and use its knowledge of typical fridge contents to locate the beverage. This flexibility makes our system applicable across a wide range of fields, from robotics and automation to autonomous driving. Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines. We hope that our proposed paradigm and approach will provide inspiration to the community and open the door to more interactive and versatile object detection systems.

## introduction in recent years, the natural language processing field has seen remarkable advancements in the development of increasingly large language models (llms). llms such as gpt-3 @xcite , @xcite , palm @xcite , megatron-turing-530b @xcite , chinchilla @xcite , and others have expanded the horizons of language understanding and generation. these neural networks, with hundreds of billions of parameters, exhibit human-like proficiency in complex reasoning @xcite @xcite @xcite . simultaneously, breakthroughs in the image and multimodal processing, as exemplified by vision-language models (vlms) like gpt4 (openai, 2023), llava @xcite and @xcite , have empowered the llms with ability to understand image inputs. these cutting-edge innovations are highly promising for a diverse array of applications across numerous fields. as highlighted by recent studies @xcite @xcite , since intelligent robot heavily relies on interactions with humans, the field of embodied ai / robotics is set to experience a significant transformation. with the emergence of human-like intelligence of llms and vlms, robots will be able to interpret human instructions and reason over visual scenes, enabling them to execute corresponding actions. this breakthrough will lead to the creation of intelligent robots that are more helpful to humans, and opens up possibilities for various fields. however, it is important to note that while vlms have made remarkable progress in generating highquality image descriptions, this alone is insufficient for robots to interact with the physical world effectively. to achieve this, robots must be able to accurately identify and localize objects within visual scenes, which is a vital prerequisite for performing actions such as "moving" and "grasping" objects. this goal of "localizing objects" is closely linked to the field of object detection, which is one of the most fundamental and extensively studied research areas in computer vision. conventional object detection systems, such as faster- @xcite , retina-net @xcite , and @xcite can only detect a fixed number of object categories, which restricts their practicality. recently, a series of open-vocabulary detection (ovd) systems have emerged as the new trend @xcite @xcite . specifically, those models adopt the contrastive learning approach to align the object-level visual features with the textual class embeddings extracted from a pretrained text encoder (e.g., bert @xcite ). in this way, those models are able to detect a much wider range of objects during inference. despite the success achieved by ovd systems, they still require humans to provide specific object names, which is neither user-friendly nor realistic. firstly, human users tend to provide high-level instructions, which may not explicitly contain the object of interest. secondly, the constraints of human knowledge often hinders the users to provide the object names. for example, the user may wish to identify fruits with a high vitamin k content but lack the necessary expertise to determine which fruits fulfill this requirement. finally, the range of object categories that humans can supply is intrinsically finite and non-exhaustive. as an illustration, when attempting to detect "objects posing hazards to autonomous vehicles," humans may only be able to enumerate a limited number of scenarios, such as compromised visibility or intricate pedestrian traffic patterns. in summary, it would be desirable if the detection model is able to interpret human instruction, employ its own knowledge to identify all objects of interest via reasoning, and finally localize them. to this end, we propose a new research task: reasoning-based object detection. in essence, humans provide abstract queries via natural language, then the model discerns and reasons which objects in the image may fulfill the query, subsequently detecting them. we made preliminary explorations in this direction. specifically, we fine-tune a vlm (e.g., minigpt-4 @xcite ) built on llms (e.g., vicuna @xcite ) to perform reasoning and predict objects of interest based on user queries (instructions) and input images. we then provide the object names to an open-vocabulary detector for specific location prediction. to facilitate future research in the direction of reasoning-based object detection, we curate a benchmark named rd-bench containing 20000 images and around 120000 query-answer pairs, which will be opensourced for the research community. ## problem statement recent vision language models (vlms) backed with llms have shown promising results in visual understanding based on the visual scene and natural language input. however, they lack the ability for fine-grained visual understanding and precise localization, which makes them difficult to be applied to real-world scenarios involving embodied ai, such as robotics ad autonomous driving. on the other hand, object detection is a crucial task in computer vision, which enables models to analyze images in a fine-grained manner and predict precise object locations. unfortunately, existing detection methods either can only predict fixed number of object classes, or need exact object names for detection. to address these limitations, we propose a new task termed reasoning-based object detection. in this task, users provide abstract queries using natural language, and the model analyzes both the image and user input, reasons about which objects in the image may fulfill the user's goal, and finally detects their location in the image. for example, as shown in figure 2 , when a user requests "i want to have a cold beverage," the model first analyzes the image of a kitchen and determines that there is no "cold beverage" available. then, it identifies a refrigerator in the image and, based on the common sense knowledge stored in the llm, infers that the find what's interesting about the image. the image shows a bird perched on a window sill, looking out onto a busy city street. the bird appears to be a pigeon, which is a common sight in urban areas. the bird's presence in the image is interesting because it provides a glimpse into the life of a wild animal living in an urban environment. find what's interesting about the image. in the image, there is an interesting scene where a bird is flying past a window, and a cat is sitting on the windowsill, watching the bird intently. the bird is located closer to the left side of the image, while the cat is on the right side, occupying the edge of the window. this captivating moment captures the interaction between the two animals, showcasing their curiosity and natural instincts towards each other. find what's interesting about the image. the image shows a view of a cityscape from a window. the window is made of glass and has a white frame. there are buildings and cars visible in the distance. the sky is cloudy and there are some trees on the sidewalk. there is a red cup on the windowsill. ## multi-modal query-answer instruction data generation the traditional way for labelling a dataset requires a lot of human labor. recently, llms such as chatgpt possess superior generation capability, which can be used to replace human labeling with automatically generated annotations @xcite @xcite @xcite . however, such text-only llms are not able to interpret visual inputs, which hinders their practicality for data generation based on images. motivated by llava @xcite , we leverage the images of pre-existing datasets @xcite , and employ two types of textual annotations to bridge the gap between visual and textual representations: (1) image captions, which depict the visual content from different viewpoints. (2) objects categories, which are the objects present in the image. specifically, we adopt coco @xcite and ob-jects365 @xcite datasets for constructing rd-bench. based on the given captions and objects, we design query-answer prompts to instruct chatgpt to generate the following: (1) a more detailed description of the scene, which gives chatgpt itself a better sense of the visual scene; we reorganize the annotations such that each image is associated with the corresponding queryanswer pairs. the detailed system prompt for our cross-modal object detection task is shown in ## demonstration we present the visualization results of detgpt in figure 5 and evaluate its capabilities. interestingly, detgpt exhibits the following appealing features: 1) it is proficient in common-sense reasoning based on the user's abstract query and the image; 2) it can utilize the rich knowledge stored in llms that are beyond human common sense; 3) thanks to the abundant knowledge stored in the vlm, detgpt generalizes to a broad range of objects that do not appear during the instruction tuning. ## experiments we conduct first stage training on paired image-text data to achieve vision-language alignment. afterwards, we conduct instruction tuning and evaluation on our curated rd-bench. specifically, we randomly sample a subset of 5000 images that are originally from coco dataset and the corresponding query-answer pairs, which accounts for around 30000 query-answer pairs for instruction tuning. for evaluation, we sample (1) 1000 images that are originally from coco dataset to evaluate det-gpt's in domain (id) performance, and (2) 1000 images from object365 dataset, which are not seen by the model during training, to test its out of domain (ood) performance. training details. for first stage training, the learning rate is set to 1 × 10 -4 , the batch size is set to 128, and the model is trained for 40000 steps. for instruction tuning, the learning rate is 3.5×10 -5 , the batch size is set to 32, and the model is trained for 40000 steps. we use adamw as the optimizer, with cosine learning rate scheduler. we use 8 a40 gpus to conduct all experiments. evaluation. we conduct evaluation using the conventional metric for object detection, i.e., mean average precision (map), which quantifies how well the predicted bounding boxes overlap with the ground truth ones. specifically, the precision and recall are defined as follows: @xmath0, fp represents false positives (incorrectly detected objects), fn represents the number of false negatives (missed detections). then, average precision (ap) and mean average precision (map) are defined as: there exist two major issues during evaluation: (1) the language model's output can not be guaranteed to exactly match the object names in the benchmark, even though they share the same meaning; ## conclusion we propose a new task termed reasoning-based object detection, in which the model needs to interpret high-level human instructions, reason over the visual scene, and finally localize the objects of interest. to facilitate future research in this task, we curate rd-bench, a dataset that can be used for training and evaluation. then, we design a twostage detection pipeline, named detgpt, which demonstrates strong ability in open-ended tasks and achieves promising performance on our proposed benchmark. we hope our work will pave the way for a more interactive and user-friendly object detection system, which will inspire later works on embodied ai, autonomous driving, and robotics. find the elderly person. the image depicts an elderly person wearing a face mask while walking down the street with a cane. a young man is helping the elderly person. find the toy plane. figure 7 : demonstration of failure cases. top: even though the multi-modal is able to understand the visual scene and find the elderly lady, the object detector localizes both the young man and the elderly person, and label them both as elderly person. this may be due to the detector is not able to distinguish "young man" from "elderly person". bottom: there is only one toy plane in the image, but the multi-modal model recognizes "a pile of toy planes". this may be caused by the multi-modal model's lack of fine-grained visual recognition capability.