Learning to Refer: How Scene Complexity Affects Emergent Communication in Neural Agents (2025)

Abstract: We explore how neural network-based agents learn to map continuous sensory input to discrete linguistic symbols through interactive language games. One agent describes objects in 3D scenes using invented vocabulary; the other interprets references based on attributes like shape, color, and size. Learning is guided by feedback from successful interactions. We extend the CLEVR dataset with more complex scenes to study how increased referential complexity impacts language acquisition and symbol grounding in artificial agents.

## introduction and background how do cognitive systems bridge the gap between rich, continuous sensory experiences and the sparse, discrete symbols used in communication? while perception operates through continuous signals, linguistic communication relies on finite vocabularies that must ground meaning about the perceived world @xcite @xcite . this representational challenge, known as the symbol grounding problem @xcite , becomes particularly acute in artificial systems where discrete symbols must acquire meaning through interaction rather than pre-programmed associations. referring expressions require systems to map visual attributes onto linguistic descriptions that uniquely identify target objects and thus can be used to study symbol grounding. @xcite formalized this process through an incremental generation algorithm that constructs descriptions by systematically adding distinguishing properties in order of salience until achieving unique identification. by this, referring expression only contain attributes that are necessary to discriminate the target from the surroundings. research in this area investigates how artificial agents can develop referential abilities through lan-guage games -interactive scenarios where communication protocols emerge from repeated coordination attempts @xcite @xcite @xcite . modern implementations use deep neural networks as agents that exchange discrete messages to solve visual discrimination tasks, allowing systematic study of how symbol meaning emerges from interaction. this paper examines emergent referential communication in neural agents tasked with identifying objects in 3d visual scenes. using a highly controlled extension of the clevr dataset @xcite , we are able to manipulate the bias the neural agents are able to use in the emergent communication. we are able to vary the complexity of referential scenarios to understand the constraints governing successful symbol grounding. our work is a study of how increasing the complexity of the scene (and therefore the space of potential referential expressions to be learned) affects learning through interaction of particular configurations of neural networks. ## dataset we extend the original clevr framework @xcite to have more control over the generated scenes. 1 by this, the objects in the generated images are controlled to have different humanrecognizable attributes, namely the shape, size and color. these attributes also correspond to referring expressions in natural language such as english which effectively biases the agents to learn a language that is comparable to a human language. the objects in the scene are separated into two categories: one target object and a controlled num-(a) 'clevr color', small brown cylinder (b) 'dale-2', small green cylinder (c) 'dale-5', large purple cylinder ber of distractor objects. the target object is the main object in the scene and the models are trained to identify and communicate it between each other. this object is unique in the scene in respect to the attributes. the distractor group contains objects that can share a maximum of two attributes per object. distractors are not required to be unique. using these rules, we generate three datasets with the following constraints: (i) the size of the generated images is 480×320 pixels; (ii) 10.000 images are created for each of the datasets; (iii) each image contains a maximum of 10 objects, that are not intersecting, have the same minimum distance between objects and are at least partially visible from the camera. ## method 3.1 image processing to extract the features and process the images of the datasets, we build upon the proposed architecture in @xcite which was used to train baseline models on the original clevr dataset. hereby, the image is first passed through a frozen resnet-101 model @xcite . two convolutional layers with subsequent relu nonlinearities condense the important information from the output of the feature extractor. the convolutional layers reduce the channels to 128 channels, using a kernel size of 3 and a stride and padding of 1. this matrix represents the encoded image with its extracted features. ## discussion and conclusion we demonstrate a method for conducting focused experiments on artificial data through which we gain valuable insights what particular models are capable of learning from data and their dependence on the structure and representations in the data in the context of linguistic coordination and learning over a visual scene. this knowledge can be transferred to the design of larger systems that are trained on real data to gain insights about learning architectures, representations of features and datasets. they can also be used as a diagnostic probes for systems trained on real data. our language games revealed that agents can successfully develop communication protocols, achieving substantial performance gains over baselines. however, emergent communication faces constraints: medium-sized vocabularies and message lengths proved most effective. scene complexity significantly impacts learning, with simpler scenes enabling near-perfect communication while complex scenes challenged most configurations. the natural language experiments provided crucial insights into these limitations. when generating referring expressions, models achieved high accuracy on simple scenes but struggled with complex discriminations. critically, the size attributes proved most difficult to learn across all tasks, followed by the color, while the shape was consistently well-identified. this indicates that humans and artificial neural networks have quite different learning biases that facilitate learning for humans (e.g. pragmatic referring described in the dale-reiter algorithm) is difficult to learn for systems. the experiments demonstrate that once we add such learning biases (e.g. modelling focused attention) learning becomes more successful. overall, the results indicate that to be successful, learning language and vision models needs to go beyond mere observation of pixels and words. future work should investigate the linguistic properties of the emergent languages to better understand how agents encode visual attributes in their communicative protocols. detailed analysis of message patterns could reveal whether emergent languages develop similar structures seen in natural languages.