Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction (2022)

Abstract: Neural language models encode rich knowledge about entities and their relationships which can be extracted from their representations using probing. Common properties of nouns (e.g., red strawberries, small ant) are, however, more challenging to extract compared to other types of knowledge because they are rarely explicitly stated in texts.We hypothesize this to mainly be the case for perceptual properties which are obvious to the participants in the communication. We propose to extract these properties from images and use them in an ensemble model, in order to complement the information that is extracted from language models. We consider perceptual properties to be more concrete than abstract properties (e.g., interesting, flawless). We propose to use the adjectives’ concreteness score as a lever to calibrate the contribution of each source (text vs. images). We evaluate our ensemble model in a ranking task where the actual properties of a noun need to be ranked higher than other non-relevant properties. Our results show that the proposed combination of text and images greatly improves noun property prediction compared to powerful text-based language models.

## introduction common properties of concepts or entities (e.g., "these strawberries are red") are rarely explicitly stated in texts, contrary to more specific properties which bring new information in the communication (e.g., "these strawberries are delicious"). this phenomenon, known as "reporting bias" @xcite ), makes it difficult to learn, or retrieve, perceptual properties from text. however, noun property identification is an important task which may allow ai applications to perform commonsense reasoning in a way that matches people's psychological or cognitive predispositions and can improve agent task: retrieve relevant properties of nouns (a) language model (b) clip (c) cem 1.male 2.friendly 3.harmless 4.female 5.available ... ... lm a cat is generally [mask]. candidate properties concreteness score lm predictions clip predictions cem 1.feral 2.cute 3.domesticated 4.affectionate 5.friendly ... ... an object with the property of [mask]. candidate properties clip 1.whiskered 2.adop we frame property identification as a ranking task, where relevant properties for a noun need to be retrieved from a set of candidate properties found in association norm datasets @xcite @xcite . we experiment with text-based language models @xcite @xcite and with clip @xcite which we query using a slot filling task, as shown in @xcite and (b). our ensemble model (figure 1(c )) combines the strengths of the language and vision models, by specifically privileging the former or latter type of representation depending on the concreteness of the processed properties @xcite . given that concrete properties are characterized by a higher degree of imageability @xcite , our model trusts the visual model for perceptual and highly concrete properties (e.g., color adjectives: red, green), and the language model for abstract properties (e.g., free, infinite). our results confirm that clip can identify nouns' perceptual properties better than language models, which contain higher-quality information about abstract properties. our ensemble model, which combines the two sources of knowledge, outperforms the individual models on the property ranking task by a significant margin. ## related work probing has been widely used in previous work for exploring the semantic knowledge that is encoded in language models. a common approach has been to convert the facts, properties, and relations found in external knowledge sources into "fillin-the-blank" cloze statements, and to use them to query language models. apidianaki and garí soler (2021) do so for nouns' semantic properties and highlight how challenging it is to retrieve this kind of information from bert representations @xcite . furthermore, slightly different prompts tend to retrieve different semantic information @xcite , compromising the robustness of semantic probing tasks. we propose to mitigate these problems by also relying on images. features extracted from different modalities can complement the information found in texts. multimodal distributional models, for example, have been shown to outperform text-based approaches on semantic benchmarks @xcite @xcite . similarly, ensemble models that integrate multimodal and text-based models outperform models that only rely on one modality in tasks such as visual question answering @xcite @xcite , visual entailment @xcite , reading comprehension, natural language inference @xcite ), text generation @xcite , word sense disambiguation @xcite , and video retrieval @xcite . we extend this investigation to noun property prediction. we propose a novel noun property retrieval model which combines information from language and vision models, and tunes their respective contributions based on property concreteness @xcite . concreteness is a graded notion that strongly correlates with the degree of imageability @xcite ; concrete words generally tend to refer to tangible objects that the senses can easily perceive @xcite . we extend this idea to noun properties and hypothesize that vision models would have better knowledge of perceptual, and more concrete, properties (e.g., red, flat, round) than text-based language models, which would better capture abstract properties (e.g., free, inspiring, promising). we evaluate our ensemble model using concreteness scores automatically predicted by a regression model @xcite . we compare these results to the performance of the ensemble model with manual (gold) concreteness ratings @xcite . in previous work, concreteness was measured based on the idea that abstract concepts relate to varied and composite situations @xcite . consequently, visually grounded representations of abstract concepts (e.g., freedom) should be more complex and diverse than those of concrete words (e.g., dog) @xcite . @xcite specifically measure the entropy of the vectors induced by multimodal models which serve as an expression of how varied the information they encode is. they demonstrate that the entropy of multimodal vectors strongly correlates with the degree of abstractness of words. ## conclusion we propose a new ensemble model for noun property prediction which leverages the strengths of 22 details on the frequency of the properties retrieved by each model are reported in appendix e.1. we provide more randomly sampled qualitative examples in appendix e.5. 23 refer to e.3 for the number of nouns with exact same top-k predictions for different values of k. 24 apple, plum, grapefruit, tangerine, orange, lime, lemon, grape, rhubarb, cherry, cap, cape, blueberry, strawberry, pine, pineapple, prune, raspberry, nectarine, cranberry language models and multimodal (vision) models. our model, cem, calibrates the contribution of the two types of models in a property ranking task by relying on the properties' concreteness level. the results show that the cem model, which combines roberta and clip outperforms powerful text-based language models (such as gpt-3) with significant margins in three evaluation datasets. additionally, our methodology yields better performance than alternative ensembling techniques, confirming our hypothesis that concrete properties are more accessible through images, and abstract properties through text. the accuracy scores obtained on the larger datasets show that there is still room for improvement for this challenging task.