Bridging Language and Scenes through Explicit 3- D Model Construction (2025)

Abstract: We introduce the methodology of explicit model construction to bridge linguistic descriptions and scene perception and demonstrate that in Visual Question-Answering (VQA) using MC4VQA (Model Construction for Visual Question-Answering), a method developed by us. Given a question about a scene, our MC4VQA first recognizes objects utilizing pre-trained deep learning systems. Then, it constructs an explicit 3-D layout by repeatedly reducing the difference between the input scene image and the image rendered from the current 3-D spatial environment. This novel “iterative rendering” process endows MC4VQA the capability of acquiring spatial attributes without training data. MC4VQA outperforms NS-VQA (the SOTA system) by reaching 99.94% accuracy on the benchmark CLEVR datasets, and is more robust than NS-VQA on new testing datasets. With newly created testing data, NS-VQA’s performance dropped to 97.60%, while MC4VQA still kept the 99.0% accuracy. This work sets a new SOTA performance of VQA on the benchmark CLEVR datasets, and shapes a new method that may solve the out-of-distribution problem.

## introduction the success of llms is witnessed by its capability of human-like questionanswering @xcite , but, they remain as black-box systems, data hungry, and do not work well for out-of-distribution data in real application @xcite . spatial semantics bridges spatial descriptions and visual perception and is the first semantics that human babies acquire. it is used as a reference for the understanding of other semantics @xcite . it plays a fundamental role in computational linguistics and cognitive modelling @xcite . visual question answering (vqa) is a challenging task that involves answering questions about an image in natural language @xcite . for example, given an image of a dice and the question "what is the shape of the object?", a vqa system should be able to generate the answer "cube". vqa is a challenging task because it requires the model to understand both the visual and spatial content of the image and the meaning of the question @xcite . a vqa system must be able to reason about spatial relations, such as the distance between objects, the relative positions of objects, and the orientation of objects. the state-of-the-art (sota) vqa system is neural-symbolic vqa (ns-vqa) @xcite . ns-vqa achieves a near-perfect accuracy of 99.8% on the clevr dataset @xcite , which is a challenging dataset of images and questions that test a vqa system's ability to reason about spatial relations. ns-vqa combines deep representation learning for visual recognition and language understanding with symbolic program execution for reasoning. ns-vqa generates executable programs as the meaning of the question, and apply for the learned visual and spatial attributes to produce the answer. ns-vqa learns spatial attributes about an input image by supervised deep learning. therefore, it does not have an explicit 3-d spatial layout of the input image. this weakens the explainability and reliability, makes the system data-hungry and performs well only when training and testing data share the same or very similar distribution @xcite . on the other hand, sufficient empirical experiments in psychological research advocates the model theory for spatial reasoning @xcite @xcite @xcite , whose standard process is a sequence of model construction, model inspection, and model variation @xcite . the preferred mental model theory argues that people construct a preferred and simplified model in mind, in a deterministic manner, while ignoring other possible models (ragni and @xcite -the construction of the first model shall not be a stochastic process that produces one model this time and another the next time (ragni and @xcite , the next model will be revised following the principle of minimal changes from the current one @xcite @xcite , and generated by a local transformation of the current model. inspired from the model theory, here, we move one step ahead of ns-vqa, by replacing its supervised learning component of spatial attribute with a 3-d spatial reconstruction component, and developed the process of "model construction by iteration render" (mcir). as illustrated in figure 1 , the mcir process first initialises a 3-d spatial layout for all recognised objects, figure 1 (i), followed by the loop of render-and-update, figure 1 (ii,iv). the render operation projects a 3-d layout into a 2-d image, figure 1(c ); the update operation is carried out to reduce the difference between the original input image and the current rendered image. the result of the comparison operation is always greater than or equal to zero. we compare mc4vqa with ns-vqa in two experiments. the first experiment is performed using the original clevr dataset. mc4vqa achieved an accuracy of 99.94%. this outperforms all state-of-the-art methods, including ns-vqa. the aim of the second experiment is to examine whether traditional supervised learning endows neural-networks the ability to acquire 3-d spatial attributes from 2d images. we developed a new testing dataset, which contains 4000 images, generated by the clevr image generator from four different camera perspectives. each scene is generated using a randomly selected camera configuration. ns-vqa had an overall accuracy of 98.39%. in contrast, our proposed method maintained another near-perfect accuracy at 99.8%. the success of mc4vqa not only demonstrates the power of the method of model construction and inspection for the acquisition of spatial knowledge (advocated in the psychological literature), but also shows the limitation of supervised deep learning -lacking the ability of generalisation of training patterns @xcite . the contributions of mc4vqa are listed as follows: (1) it is the first vqa system that explicitly reconstructs 3-d spatial layout to bridge spatial linguistic descriptions and visual perception; (2) mc4vqa can be further developed by integrating more features of mental model theory in psychology, or used in psychological experiments; (3) source code and new datasets are publicly accessible. the rest of the paper is structured as follows: section 2 reviews a number of related works; section 3 formalises the task of vqa by explicitly re-constructing 3-d spatial layout; section 4 presents the detail of mc4vqa; section 5 reports experiment results of mc4vqa, which grealy outperforms the sota performance, and demonstrates the power of the model construction method in new testing data; section 5 concludes the paper, and lists a number of future research topics. ## related work a convergent opinion from linguistics, neuroscience, and psychology is that the spatial domain is the first domain that human babies understand, and is the reference domain for the understanding of other domains @xcite @xcite . the next generation of language system shall be a brain-and ai-inspired understanding system that explicitly represents situations @xcite . our work focuses on the ns-vqa model, and promises a novel method to explicitly represent scene images by constructing 3-d geometric spatial models. ns-vqa uses an older object detection model based on detectron @xcite and mask r-cnn @xcite . since then, newer models with improved accuracy and speed have been released, such as yolo @xcite , which produces impressive results and can be used for real-time video processing. yolo yolo (you only look once) is a powerful object detection model which is known for its speed and accuracy @xcite . the current version of yolo (v8) @xcite is the state-of-theart object detection model that utilizes cross stage partial (csp) @xcite architecture, which was introduced in yolov4 @xcite . our mc4vqa uses yolov8 as its object detection model. yolo offers several pretrained models, of which we chose "yolov8x-seg" which has great segmentation accuracy. ## motivation of vqa through model construction and inspection ever since tolman's rats experiments @xcite in the 1940s, sufficient evidence has been collected to show that animals and humans can construct comprehensive spatial models in mind of their environments through sensorimotor interaction (spelke and @xcite and that this spatial model in mind structures our language @xcite @xcite . this motivates us to move one step ahead of ns-vqa by replacing its supervised resnet34 compo-nent with a novel component that explicitly constructs 3d spatial layout, thus mc4vqa (model construction for vqa). this allows the symbolic program execution engine to more accurately identify objects and their spatial relationships in the scene. as being unsupervised, our method may improve the overall generalization of the scene construction, allowing to function on unknown camera configurations. ## formalising the task in this section, we define the task of vqa through model construction and inspection. the input of mc4vqa consists of an image i and a question q asking the content of this image, whose content can be described as a set of objects i o1 . . . i on and a set of 2d locations l oi of i oi , line 1 in algorithm 1. the process of model construction p will construct a 3d spatial layout s for i. s consists of a set of 3d objects o i with their size and their 3d location information. let s 0 be an initial 3d layout, line 2 in algorithm 1, the construction process p will update s i to s i+1 , with the following procedure: p will trigger an inspection function i to take a photo of s i , so called "rendering", let i(s i ) = i (i) . then, a function m will measure the difference between i (i) and the original image i. finally, a function g will apply a set of geometric operations on objects in s i . this transforms s i into s i+1 , so that a photo of s i+1 will be more similar to the original image, that is, m(i (i+1) , i) < m(i (i) , i). the construction process will stop, if m(i (i+1) , i) is less than a predefined threshold value . the final 3d layout s n will be inspected to answer the question q (algorithm 1). ## mc4vqa mc4vqa has four components: an object detector (yolov8), a 3d model constructor (mcir), a question parser (bi-lstm encoder), and a program executor. object detection the yolov8 object detector is trained on the same 4000 clevr images used by ns-vqa. the input image is first passed to the object detector to generate object proposals. the object proposals are composed of the predicted object masks and the object bounding boxes, along with their class names. object proposals with a score of less than 0.9 are discarded. the predicted class names are composed of the discrete attributes of the objects, e.g., the object size, colour, material, and ## conclusions and outlooks understanding surrounding environment is a fundamental ability for the survival of animals and humans, e.g., to escape from dangerous predators. it is a challenging research task in nlu and ai, and has various downstream applications, e.g., autonomous driving, service robots. vqa with the benchmark clevr dataset is a micro-world to explore this field, in which images are about layouts of synthesised geometric objects. supervised neural networks to learn spatial attributes are very successful, with two conditions: (1) it needs a huge amount of training data; our experiment results show that our new method is very promising -it does not need training data for acquiring spatial regions and achieves higher accuracy in answering questions about out-of-distribution scenes. in this work, we implemented mcir using a simple object-level loop to optimize object locations and used ns-vqa's question parser and executor with the clevr validation questions. in the future, we will adopt a dual-camera configuration to figure out the locations of 3d objects precisely and will use the constructed 3d layout construction as the spatial semantics to interpret linguistic descriptions.