Visual Objects As Context: Exploiting Visual Objects for Lexical Entailment (2020)

Abstract: We propose a new word representation method derived from visual objects in associated images to tackle the lexical entailment task. Although it has been shown that the Distributional Informativeness Hypothesis (DIH) holds on text, in which the DIH assumes that a context surrounding a hyponym is more informative than that of a hypernym, it has never been tested on visual objects. Since our perception is tightly associated with language, it is meaningful to explore whether the DIH holds on visual objects. To this end, we consider visual objects as the context of a word and represent a word as a bag of visual objects found in images associated with the word. This allows us to test the feasibility of the visual DIH. To better distinguish word pairs in a hypernym relation from other relations such as co-hypernyms, we also propose a new measurable function that takes into account both the difference in the generality of meaning and similarity of meaning between words. Our experimental results show that the DIH holds on visual objects and that the proposed method combined with the proposed function outperforms existing unsupervised representation methods.

## introduction recognizing lexical entailment (le) is a fundamental component in natural language processing (nlp), helping with many tasks such as textual entailment recognition @xcite , taxonomy creation @xcite , and metaphor detection @xcite . lexical entailment defines an asymmetric relation between two terms, where one term can be inferred by the other, but not vice versa. for example, dog entails animal but not vice versa because animal does not always mean dog. to recognize le, it is required (1) to construct a good representation that captures the generality of the meaning of a term, and (2) to define a measure to jointly calculate the asymmetric difference in the generality of meaning and the similarity of meaning between two given terms. an increasing number of representation methods and measures to compute hypernymy have been proposed to date @xcite @xcite . especially, @xcite and @xcite proposed unsupervised methods that follow the distributional informativeness hypothesis (dih). however, they have not used visual information in their methods and instead, required a large amount of textual data to construct the representations. in the field of computer vision, @xcite have shown that terms at higher levels in the hierarchy of wordnet @xcite tend to correspond to a greater variety of images than terms at lower levels. @xcite have focused on this tendency and used a set of images obtained through image search to construct a word representation for the le task, where image features were extracted from a convolutional neural network (cnn) @xcite . however, no work has directly studied whether the dih holds on visual objects. to this end, we propose an unsupervised method to construct word representations for the le task by using a set of images associated with each word. more specifically, we define a representation of a word as a bag of visual objects (labels) found in the associated images. thus, our method allows us to directly evaluate the feasibility of the dih on visual objects. moreover, our method has two advantages over the previous methods. firstly, unlike previous text-based approaches, our method does not require a huge amount of text corpora to construct representations. secondly, due to the discrete nature of object labels, our representation is expected to be more discriminative than others constructed from a (middle layer of) cnn. it is a desirable property to distinguish the generality of the meaning of a word in the le task. in addition to our representation method, we also propose a new function to jointly measure the difference in the generality of meaning and similarity of meaning between two terms to distinguish word pairs in a hypernym relation from others. we evaluate our representation method and function on different types of le datasets. we experimentally show that the combination of our representation and function outperforms the dih-based method @xcite , word embeddings trained on a large text corpus, and the cnn-derived visual representation method @xcite , revealing that the dih holds on visual objects. we also analyze the number of images as well as the number of unique objects to study how they affect the quality of our representations. in summary, our contributions are three-fold: ## methodology we construct our representations from visual objects. we illustrate an overview of our representation construction in figure 1 . based on the representations, we introduce hypernymy measures to measure the generality of word meanings. we then explain how the le task is solved. ## experiment we conducted lexical entailment experiments by using different types of datasets to evaluate the capability of our object-based representation method. ## conclusion we proposed a new word representation method based on discrete visual objects in images associated with each word for the le task. our method outperformed both traditional unsupervised cnnbased representations and text-based dih representations on different types of lexical entailment datasets. we also experimentally confirmed that the distributional informativeness hypothesis holds on visual objects. in addition, we revealed that our method got rapidly saturated at around 10-20 images and 200 dimensions (i.e., the context size). this suggests that our representations can achieve sufficient informativeness even with a smaller number of images and contexts. one of our future research directions is to examine the capability of our object-based representations in other tasks, such as lexical induction or word similarity tasks.