Comparative Probing of Lexical Semantics Theories for Cognitive Plausibility and Technological Usefulness (2020)

Abstract: Lexical semantics theories differ in advocating that the meaning of words is represented as an inference graph, a feature mapping or a cooccurrence vector, thus raising the question: is it the case that one of these approaches is superior to the others in representing lexical semantics appropriately? Or in its non antagonistic counterpart: could there be a unified account of lexical semantics where these approaches seamlessly emerge as (partial) renderings of (different) aspects of a core semantic knowledge base? In this paper, we contribute to these research questions with a number of experiments that systematically probe different lexical semantics theories for their levels of cognitive plausibility and of technological usefulness. The empirical findings obtained from these experiments advance our insight on lexical semantics as the feature-based approach emerges as superior to the other ones, and arguably also move us closer to finding answers to the research questions above.

## introduction lexical semantics is at the core of language science and technology as the meaning of expressions results from the meaning of their lexical units and the way these are combined. how to represent the meaning of words is a central topic of inquiry and three broad families advocate that lexical semantics is represented as a semantic network @xcite , a feature-based mapping @xcite , or a semantic space @xcite @xcite . in a semantic network approach, the meaning of a lexical unit is represented as a node in a graph whose edges categorically encode different types of semantic relations holding among the units (e.g. hypernymy, meronymy, etc.). in a feature-based model, the semantics of a lexicon is represented by a map where a key is the lexical unit of interest and the respective value is a set of other units denoting characteristics prototypically associated with the denotation of the unit in the key (e.g. color, usage, etc.). under a semantic space perspective, the meaning of a lexical unit is represented by a vector in a high-dimensional space (aka word embedding), whose components are ultimately based on the frequency of co-occurrence with other units, i.e. on their linguistic contexts of usage. the motivation for these theories is to be found in their different suitability and success in explaining a range of empirical phenomena in terms of how these are manifest in ordinary language usage and also how they are elicited in laboratory experimentation. these phenomena are related to the acquisition, storage and retrieval of lexical knowledge (e.g. the spread activation effect @xcite , the fan effect @xcite among many others) and to the interaction with other cognitive faculties or tasks, as categorization @xcite , reasoning @xcite , problem solving @xcite , learning @xcite , etc. these different approaches have inspired a number of initiatives to build repositories of lexical knowledge. popular representatives are, for semantic networks, wordnet @xcite , for feature-based models, swow @xcite , and for semantic spaces, word2vec @xcite a.o. different knowledge bases (kbs) are rooted in different empirical sources: wordnet is based on lexical intuitions of human experts; the information in swow is evoked from laypersons cued with lexical units; and word2vec reflects the co-occurrence frequency of words in texts. figure 1 : fmri patterns in participant 1 for word eye: predicted (left) with wordnet 60k embeddings via matrix factorization, cf. §3.2.3 below; observed (right), as in the data set of @xcite . against this background, a fundamental research question is: could there be a unified account of lexical semantics such that the above approaches seamlessly emerge as (partial) renderings of (different) aspects of the same core semantic knowledge base? or in its antagonistic counterpart: is it the case that one of the above approaches is superior to the others in representing lexical semantics appropriately? in this paper we contribute to these research questions with experiments consisting of two phases. first, different lexical semantic kbs, instantiating different lexical semantic theories, are converted to a common representation. the map-based dataset (swow) is converted to a graph-based representation and the two graph-represented datasets (swow and wordnet) are then converted to a vector-based representation. second, to assess the appropriateness of these knowledge bases, the respective word embeddings are evaluated by means of the performance in language processing tasks where they are integrated and where lexical knowledge plays a crucial role. we resort to the task of predicting brain activation patterns from semantic representations of words, illustrated in figure 1 , to assess the lexical kbs. if a kb k1 is more successful than another k2 in supporting these tasks, this indicates that k1 has higher cognitive plausibility, likely being superior at encoding the meaning of words. second, we resort also to the task of determining the semantic similarity between words from their formal semantic representations. though it may arguably be less well grounded on actual cognitive representation of lexical meaning given the empirical frailty of the similarity scores @xcite , this has been a popular task for the intrinsic evaluation of word embeddings. third, for extrinsic evaluation, we resort to downstream natural language processing (nlp) tasks. as reported in the present paper, the findings from these experiments indicate that the feature-based approach emerges as superior to the other approaches to the representation of lexical meaning. ## related work in @xcite , the meaning of each word w was represented by semantic features given by the normalized co-occurrence counts in a big corpus of w with a set of 25 verbs related to basic sensory and motor activities. for each word, the respective fmri activation level at every voxel is calculated as a weighted sum of each of the 25 semantic features, where the weights are learned by regression to maximum likelihood estimates given the observed fmri data. mean accuracy of 0.77 was reported for the 60 words and 9 subjects in the task of predicting brain activation originated by the exposure to words. in an initial period, different authors focused on different ways to set up the features supporting this task. jelodar et al. (2010) used the same 25 features but resorted to relatedness measures based on wordnet. as features, @xcite used instead 5 sensory-motor experience-based attributes, and the relatedness scores between the stimulus word and the attributes were based on human ratings. @xcite , in turn, used 65 attributes with relatedness scores crowdsourced from over 1,700 participants. as embeddings became popular, authors moved from features to word embeddings. @xcite found their best results with dependency-based embeddings. @xcite used word2vec together with a visual model built with a cnn on the google images dataset. recently, @xcite evaluated 8 embeddings in predicting fmri patterns: the experiential embeddings of @xcite ; the non-distributional feature-based embeddings of @xcite ; and 5 different distributional embeddings, namely word2vec @xcite , fasttext this work is licensed under a creative commons attribution 4.0 international licence, @xcite , dependency-based word2vec @xcite , glove @xcite and lexvec @xcite ; as well as the vectors from @xcite . the dependency word2vec achieved the best performance among the embeddings, while the @xcite seminal approach with 25 features "is doing slightly better on average" than the other approaches. in contrast to these papers, the goal here is not to beat the state of the art in brain activation prediction but to probe lexical semantic theories with the help of this task. ## common representations to proceed with such comparative probing, a first step consists in the conversion of the different lexical kbs into the common representation format of semantic vector spaces, which we describe in this section. ## similarity and relatedness the tasks considered in the third experiment consist of predicting the semantic similarity or relatedness between words in pairs and in seeking to match the gold scores assigned by humans to such test pairswith the cosine between the predicted vectors mapped into the scale used for gold scores. the goal in this paper is not to beat the state of the art in similarity prediction tasks but to understand whether when testing embeddings from different empirical sources, these tasks deliver results similar to those delivered by brain activity tasks -our interest is on gaining insight about whether the similarity tasks, based on simpler and cheaper data, can be used for the same practical purposes of the brain activity tasks, based in much more expensive and hard to get data @xcite . for semantic similarity, we used simlex-999 (with 999 pairs) @xcite , wordsim-353-similarity (203 pairs) @xcite and rg1965 (65) @xcite . for semantic relatedness, wordsim-353-relatedness (252) @xcite , men (3000) @xcite and mturk-771 (771) @xcite were used. the results for wordnet-, text-and swow-based embeddings are in tables 3 , 4 and 5 ( @xcite , respectively. ## extrinsic evaluation results from intrinsic and extrinsic evaluation in nlp are not necessarily aligned with each other as performance increments in intrinsic results may have or not an incremental impact on the intricacies and performance of the larger systems where components happen to be embedded. as intrinsic vs. extrinsic congruence is thus something that has to be determined empirically, it is important to proceed with probing lexical semantics theories with downstream tasks, foot_5 for their extrinsic evaluation. for multi-task benchmark, we resort to glue platform @xcite , which contains 9 tasks of 3 types, from which we used a subset of 5 tasks with affordable computational footprint: the 2 singlesentence tasks, cola and sst-2; 1 of the 3 similarity and paraphrase tasks, namely mrpc; and 2 of the 4 inference tasks, namely rte and wnli. sst-2 is a task-based on the stanford sentiment treebank @xcite consisting of sentences extracted from movie reviews and human annotation of their sentiment. the model trained on these data has to determine the sentiment expressed in input sentences. the mrpc task is based on the microsoft research paraphrase corpus @xcite with sentence pairs automatically extracted from online news and manually annotated as to whether the sentences in each pair are semantically equivalent. the rte task resorts to the recognizing textual entailment datasets from online news, which result from annual challenges for the task of textual entailment, and gather the data from rte1 @xcite , rte2 @xcite , rte3 @xcite , and rte5 @xcite . this task consists of predicting whether the premise entails the hypothesis in each test item. wnli is a task-based on the winograd schema challenge @xcite where each example contains a sentence with a pronoun and a list of admissible antecedents. this task consists of picking the right antecedent. finally, the cola task is based on the corpus of linguistic acceptability. ## conclusions contributions a first major contribution of this paper is the design of an experimental setup to comparatively probe lexical semantics theories of all kinds -feature-, inference-and cooccurrence-based -for both their cognitive plausibility and their usefulness in nlp. this setup consists of converting representative repositories of these theories to a common format and integrating them, under such a common format, in different models for different probing tasks: • to be converted, exemplar techniques from each major graph embedding family (edge reconstruction, matrix factorization, random walk) were used. another major contribution is the empirical results obtained with a systematic application of this probing setup, including the central finding that the feature-based lexical knowledge base is superior to knowledge bases complying with other lexical semantic theories in consistently supporting models with top performance across the different probing tasks. concerning the other, incidental empirical findings: • as to graph embedding techniques, it was possible to understand that graph embedding methods taking into account the affinity between nodes at the global level of the graph tend to better serve downstream tasks.