C ogni V al in Action: An Interface for Customizable Cognitive Word Embedding Evaluation (2020)

Abstract: We demonstrate the functionalities of the new user interface for CogniVal. CogniVal is a framework for the cognitive evaluation of English word embeddings, which evaluates the quality of the embeddings based on their performance to predict human lexical representations from cognitive language processing signals from various sources. In this paper, we present an easy-to-use command line interface for CogniVal with multiple improvements over the original work, including the possibility to evaluate custom embeddings against custom cognitive data sources.

## introduction & background the system presented in this work is based on the cognival framework presented by @xcite . we present the first encompassing framework for cognitive word embedding evaluation. we improve and extend the original features of cognival and provide a simple command line interface for scalable and customized experiments. cognival is openly available at https://github.com/ ds3lab/cognival-cli and can be easily installed with pip. language models and word representations are the corner stones of state-of-the-art nlp models. evaluating and comparing the quality of different word representations is a well-known, largely open challenge. while word representations and language models have proven very useful for nlp applications, their interpretability is inherently challenging. interpretability is key for many nlp applications to be able to understand the algorithms' decisions. for a truly intrinsic evaluation of word embeddings more research about the cognitive plausibility of current language models is required @xcite . moreover, one of the challenges for computational linguistics is to build cognitively plausible models of language processing, i.e., models that integrate multiple aspects of human language processing at the syntactic and semantic level @xcite . currently, word embeddings are evaluated with extrinsic or intrinsic methods. extrinsic evaluation is the process of assessing the quality of the embeddings based on their performance on downstream nlp tasks, e.g., sentiment analysis. however, embeddings can be trained and fine-tuned for specific tasks, but this does not mean that they accurately reflect the meaning of words. one the other hand, intrinsic evaluation methods, such as word similarity and word analogy tasks, merely test single linguistic aspects. these tasks are based on conscious human judgements, which can be biased by subjective factors @xcite . it has been noted that current intrinsic evaluation methods do not capture the cognitive plausibility of the word embeddings and language models @xcite . both intrinsic and extrinsic evaluation types often lack statistical significance testing and do not provide a global quality score. cognival addresses these issues and proposes an evaluation method based on cognitive lexical semantics. cognitive lexical semantics proposes that words are defined by how they are organized in the brain @xcite . recordings of brain activity play a central role in furthering our understanding of how human language works. @xcite showed in a neuroscientific study how words are represented in semantic maps across the brain. moreover, language representations tuned on brain activity show improved performance on nlp tasks @xcite , and word representations trained on brain activity are more generalizable to unseen words @xcite . hence, it seems natural to evaluate language models against human language processing data, as we have proposed with cognival @xcite . to accurately encode the semantics of words, we believe that embeddings, and full language models, should reflect this mental lexical representation. this allows us to evaluate word embeddings by quantifying their cognitive plausibility. there was a need for an extensive approach showing the utility of human cognitive data for language model evaluation and its correlation in predicting downstream task performance @xcite . evaluating word embeddings with cognitive language processing data has been proposed previously. for instance, @xcite and @xcite evaluated different embeddings by predicting the neuronal activity of nouns. @xcite showed preliminary results in evaluating embeddings against continuous text stimuli in eye-tracking and functional magnetic resonance imaging (fmri) data. moreover, @xcite recently presented an extensive set of language-brain encoding experiments. electroencephalography (eeg) data has been used for similar purposes. @xcite and @xcite show that components of event-related potentials can successfully be predicted with neural network models and word embeddings. however, these approaches mostly focus on one modality of brain activity data from small individual cognitive datasets. the presence of only few and small data sources has been one reason why this type of evaluation has not been too popular until now @xcite . hence, for cognival we collected a wide range of cognitive data sources ranging from eye-tracking to eeg and fmri to ensure coverage of a large vocabulary and of different features of the cognitive processes during language comprehension. the cognival command line interface (cli) is the first tool to unify a diverse range of cognitive data sources of multiple recording modalities of cognitive processing signals and to provide a generic user interface. in this paper, we provide a user interface for cognival, which evaluates english word embeddings against the lexical representations of words in the human brain, recorded when passively understanding language. the cognival command line interface (cli) makes large-scale cognitive word embedding evaluation accessible to nlp practitioners. it offers pre-processed cognitive data sources, readily provided for evaluation in a user-friendly interaction. it supports and complements other intrinsic and extrinsic evaluation methods for word embeddings. the cognival cli is a unified framework, allowing the evaluation of a large set of existing pre-trained embeddings but also of custom word representations and language models on a large range of cognitive sources. ## system overview the cognival cli is implemented in python (version 3.7.4) and provides an interactive shell using python-nubia foot_1 . for the purpose of cognitive embedding evaluation, @xcite collected and prepared 15 cognitive data sources and evaluated 6 pre-trained embedding types, including glove, word2vec, wordnet2vec, fasttext, elmo and bert. the command line interface provides these preprocessed data types. for details about the format of the cognitive data sources please refer to @xcite . the evaluation process is automatized in the cognival cli and works as as depicted in figure 1 . first, the user defines the general evaluation configuration, including path specifications and training parameters (command: config). if required, the user can then import custom word representations as well as custom cognitive data sources, using the import function. second, the user specifies the embedding/cognitive-data combinations to be evaluated, as well as the hyper-parameter ranges for the neural regression models. moreover, if requested, cognival generates random vectors of the same dimension of the embeddings to be evaluated. the embeddings can also be evaluated against this random baseline. as an improvement from @xcite , the cognival cli automatically generates 10 sets of different random embeddings and averages over the results for a fairer comparison to a more robust baseline. the tuning (implemented through a grid search) and training of all models (n embeddings x m cognitive data sources) is fully automatized within the command run. thereafter, the user can either use the saved results as they are (i.e., mean squared errors for each word in the vocabulary), or they can run the significance testing (command: significance), which consists of a wilcoxon signed-rank test for each hypothesis (i.e., for each embedding/cognitive-data evaluation combination), applying the bonferroni correction for the multiple hypotheses problem, as described by @xcite . finally, the automatized generation of the report also includes significance testing by default and compares the results to the baseline of random embeddings before aggregating them. the dynamic html or pdf reports include all detailed results for the individual combinations, as well as aggregated over the modalities (see figure 2 ). ## use cases the target audience for the cognival cli are nlp and machine learning practitioners and researchers developing word embeddings and in need of an evaluation benchmark. in this cognival demonstration paper, we describe the following two possible use case scenarios. ## conclusion & future work in this demonstration paper, we presented the new command line interface for cognival. the cognival cli builds upon the work by @xcite and extends it with various new features, especially the ability to evaluate custom embeddings against custom cognitive data sources. we described the functionalities of the tool as well as various use cases and an application scenario. the cognivsl cli aims at improving the accessibility and usability of cognitive embedding evaluation for nlp practitioners. cognival is still under active research and will be extended to additionally support the evaluation of sentence embeddings and further languages.