A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions (2023)

Abstract: We present a large, multilingual study into how vision constrains linguistic choice, covering four languages and five linguistic properties, such as verb transitivity or use of numerals. We propose a novel method that leverages existing corpora of images with captions written by native speakers, and apply it to nine corpora, comprising 600k images and 3M captions. We study the relation between visual input and linguistic choices by training classifiers to predict the probability of expressing a property from raw images, and find evidence supporting the claim that linguistic properties are constrained by visual context across languages. We complement this investigation with a corpus study, taking the test case of numerals. Specifically, we use existing annotations (number or type of objects) to investigate the effect of different visual conditions on the use of numeral expressions in captions, and show that similar patterns emerge across languages. Our methods and findings both confirm and extend existing research in the cognitive literature. We additionally discuss possible applications for language generation.

## introduction in recent years, vision and language models have been shown to outperform models trained on a single modality in a variety of domains, such as language modeling @xcite , document quality assessment @xcite , and visual classification and segmentation @xcite @xcite . while empirical evidence exists, including from the studies cited above, that each modality can benefit the other for task performance, less attention has been devoted to the broader question of how the two modalities influence and constrain one another. in this study, we focus on one aspect of this question: how does vision constrain language? we study the relation between the semantic content of an image and the language used to describe it. as an example, consider figure 1 . captions of the right image, which crops the agent, use passive voice more frequently than those of the left (taken from flickr30k, @xcite . we aim to study such trends by examining the influence of visual features of the image on the linguistic choices taken when describing it. a number of psycholinguistic studies have aimed to answer this question by systematically varying visual conditions (such as image cropping) and analyzing verbal descriptions of the scenes by human participants for recurring differences (e.g., @xcite . although such controlled studies allow for precise measurement, the visual stimuli are synthetic (rather than depicting natural scenes), the manual annotation of descriptions limits the size of the dataset, and typically only one linguistic property is investigated in a single language. we address this gap by proposing a scalable methodology that uses existing image-caption corpora in multiple languages. we measure the correlation of visual features and linguistic properties of the caption by training visual classifiers to predict, for a given raw image, whether a linguistic property is expressed in its captions. we compare the impact of different training sets (single vs. multiple languages) and different types of pre-training (none vs. object categories vs. visual vs. textual pre-training objectives). we use 9 large-scale image-caption datasets (overall 2.9m captions for 604k images), covering four languages (english, german, chinese, japanese), to study lexical properties (use of numerals and negation words) and structural properties (use of passive voice, transitivity of the main verb, choice of verbal vs. nominal constructions), which we automatically annotate in the captions. to the best of our knowledge, this is the first largescale, multilingual study of the impact of visual input on linguistic choice. we find evidence showing that the visual input imposes constraints on linguistic properties, and that such trends are detectable using the proposed methodology. in a complementary corpus study, we link the prevalence of linguistic properties to existing, highlevel visual annotations (number and type of objects), and find that these properties can be linked to the use of numeral expressions in similar patterns across languages, and in accordance with smallscale, highly-controlled psycholinguistic studies. our findings have both cognitive and computational implications. on the cognitive side, this study confirms findings from small-scale cognitive studies at scale: for naturalistic scenes, typologically diverse languages, and descriptions from thousands of native speakers. the magnitude of the data that can be studied with our method also allows the derivation of new insights, which can motivate additional controlled studies, making the proposed practice an effective exploration method. on the computational side captioning models have been shown to generalize better when first predicting the syntactic structure of the generated caption @xcite . this research direction may benefit from the signal provided by our classifiers of linguistic properties. ## background the notion that grammatical and lexical phenomena can be characterized semantically, at least in their prototypical instances, has a long tradition in linguistics @xcite @xcite , among many others). for example, transitive clauses are often characterized as corresponding to actions instigated by volitional agents over passive objects, that are affected by the action. however, formally defining these semantic features in a nonlinguistic way and showing empirically that the presence of these features indeed entails the presence of the corresponding linguistic feature, has proven to be methodologically challenging. one possible direction to address this is the use of images that implicitly define a type of non-linguistic semantics. this section briefly reviews different approaches for studying visual constraints on our set of five phenomena from a cognitive and computational perspective (omitting phenomena that have not been previously covered). ## approach we draw inspiration from the cognitive studies presented in section 2.1. these studies carefully design visual scenes that differ only in terms of the visual feature of interest (e.g., visibility of the agent), ask participants to describe the scenes, and compare the linguistic properties of the descriptions across different conditions. if a linguistic property is significantly more prevalent in one condition, it is assumed that this visual feature constrains that linguistic property. while such setups allow careful control over the experimental design, they are also less ecologically valid in that they impose (1) synthetic visual stimuli and (2) limitations on the number and diversity of participants, phenomena and languages to include. we address both shortcomings by (1) using large existing image caption datasets as a corpus of diverse language descriptions of naturalistic scenes, and (2) annotating the captions automatically, yet accurately, with linguistic properties. using a large amount of data instead of a controlled experiment raises an issue. unlike in controlled cognitive studies, the sets of images we use are not arranged into 'minimal pairs', which are identical except for a visual feature of interest. to overcome this limitation, we exploit the large amount of data available via the automatic annotation of linguistic properties. we train visual classifiers to predict if a linguistic property is expressed in image captions when only the image is provided. if the classifiers achieve high accuracy on a held out test set, it is an indication that the visual features are informative enough to predict the linguistic property. 2 figure 2 gives a high-level depiction of our approach. to complement our analysis, we also conduct a corpus study. first, we use semantic annotations (object classes and bounding boxes) already available in existing datasets to group images by high-level properties and analyze the prevalence of linguistic properties in each group. second, we compare the linguistic properties of captions for the same image in different languages. if a property is salient in the captions of all languages for a given image, it is likely that its visual content constrains descriptions that use that property. we present a corpus analysis using both approaches in section 5.2. ## experimental setup in this section we describe the languages (4.1), linguistic properties (4.2) and datasets (4.3) used in our experiments. ## experiments we now describe our experiments and analyses. in section 5.1 we train visual classifiers to predict linguistic properties, section 5.2 presents a complementary corpus analysis, and section 5.3 presents additional insights that may lead to future research. ## conclusion the synergistic relation between vision and language has been shown in the cognitive literature and leveraged in computational models, but how the two modalities inform each other has not been sufficiently studied at scale. we present a large scale study of the correlation of visual properties with linguistic phenomena, using naturalistic images described by a large crowd of native speakers of four languages. in addition to confirming results of previous cognitive studies, we present new findings, e.g., the effect of object type on the use of numerals in visual scene description and the cross-lingual correlation of the use of numerals. considering the effort needed to execute a controlled study, our proposed method can be used as an effective exploration technique for finding hypotheses for future controlled studies. in addition, our framework is general, and extends naturally to more languages and properties. beyond the cognitive contribution, our work can inform nlp models. recent work suggests that in captioning models, training the model to predict a structured representation of the caption (e.g., based on pos prediction) before the text improves compositional generalization @xcite . in future work, we will study the utility of predicting our proposed linguistic properties for improving captioning models.