FAST : A carefully sampled and cognitively motivated dataset for distributional semantic evaluation (2021)

Abstract: What is the first word that comes to your mind when you hear giraffe, or damsel, or freedom? Such free associations contain a huge amount of information on the mental representations of the corresponding concepts, and are thus an extremely valuable testbed for the evaluation of semantic representations extracted from corpora. In this paper, we present FAST (Free ASsociation Tasks), a free association dataset for English rigorously sampled from two standard free association norms collections (the Edinburgh Associative Thesaurus and the University of South Florida Free Association Norms), discuss two evaluation tasks, and provide baseline results. In parallel, we discuss methodological considerations concerning the desiderata for a proper evaluation of semantic representations.

## introduction assessing the performance of a distributional semantic model (dsm), be it a count model or one of the numerous and popular neural embeddings, has never been a straightforward endeavour. it is becoming an increasingly pressing issue due to the 'black box' nature of word embeddings and their aleatoric, often irreproducible training. we witness a proliferation of tasks and a growing dissatisfaction within the community: what are we modeling, really? are models learning to handle a specific dataset, or a task, or do they really abstract semantic knowledge from text? this has lead to a shift towards evaluation methods capable of capturing human-like generalization @xcite . dsm evaluation with standard datasets is affected by two major problems. the first issue is the possibility that dsms may just exploit contingent properties of a task. for example, they could be learning to identify random controls instead of capturing the intended semantic relation @xcite . even with carefully chosen distractors, models may just be picking up contingent features of the experimental items: @xcite observe that many hypernym classifiers merely learn to recognize words that are 'typical' hypernyms, independently of the stimulus items (i.e. corresponding hyponyms). the second issue is dataset size, since small test sets inevitably lead to overfitting if many different model architectures or parameter settings are evaluated. take the toefl synonym task, where a single item is worth 0.8 accuracy points, as an extreme example: @xcite achieve 100% accuracy after testing a wide range of dsm parameters. it is also true, however, that large datasets often lack in design quality (e.g. by using random controls, cf. above). in this paper, we present free association tasks (fast), a new evaluation dataset for english lexical semantics designed to satisfy the desiderata highlighted above: (a) compatibility with humanlike generalization, (b) quality of distractors, and (c) dataset size. we address (a) by focussing on a cognitive modeling task, namely free associations (which word comes to your mind when you hear giraffe?). free association norms are cognitively motivated (hence semantically plausible) and usually very large, covering thousands of stimuli. this allows for a better selection of distractor items (which makes the task more difficult but also more enlightening), addressing both (b) and (c). as a bonus, we do not collect yet another dataset; instead, we carefully sample from existing collections and define two different evaluation tasks, providing extensive baseline results for both. the first task we propose is a multiple-choice task:given a stimulus (e.g., giraffe) decide which of three candidates (neck, apple, wine) is its most frequent associate. it is in this task that we exploit the size of the underlying collection to collect better distractors: we don't sample them randomly, but rely on frequency criteria instead. distractors are selected to match the frequency band of the correct candidate (neck), in order to prevent the models from relying on frequency effects. the second task we propose is an open-vocabulary lexical access task: given a stimulus word (e.g., giraffe), select its most frequent associate from a large vocabulary of candidate words (consisting of the top associates for all stimuli in the data set). advantages of the multiple-choice task are that it is easier to implement, less computationally expensive, lends itself to the application of machine-learning techniques as a straightforward classification problem, and has an obvious evaluation metric (accuracy). the open vocabulary task is cognitively more realistic and matches the experimental setting underlying free association norms, but it is computationally expensive (because of the larger number of candidates to be considered), poses a harder machine-learning problem (learning to rank), and suitable evaluation metrics are less obvious (see sec. 4.1). this paper makes the following contributions: we release fast and propose two new evaluation tasks based on it; we report preliminary modelling results as a baseline for further work. ## related work evaluation in terms of cognitive modeling is challenging from many points of view. first, the modeled data are typically continuous (reaction times or eeg signal, see @xcite and @xcite for an overview) or, when categorical (free association norms), much less constrained compared to the data used in standard dsm evaluation tasks because the models have to handle a large and diverse vocabulary. second, a speaker's behaviour results from the interplay of many factors, some of which we expect to be conflated in distributional meaning representations and intertwined with 'true' lexical semantics (e.g, ambiguity, culture-specific information), others to be completely absent (e.g., individual differences between speakers). third, cognitive modeling datasets typically have not been designed to evaluate corpusbased models and thus need to be further manipulated (annotated, adapted, sampled), bringing in additional design assumptions. to sum up, while quantifying dsm performance is straightforward in standard tasks, it is less so in the case of cognitive modeling tasks. ## dataset the starting point for fast are the two largest collections of free association norms available: the edinburgh associative thesaurus (eat, 8210 stimuli, 100 subjects; @xcite and the university of south florida free association norms (usf, 5019 stimuli, 6000 subjects; @xcite . pre-processing all items in eat and usf were pos-tagged using frequency information from the large, publicly available web corpus encow 2014 2 to guess the most probable pos tag for each word form out of context. pos tags are not employed in the evaluation experiments conducted in this paper, but we believe that guessing pos based on frequency is cognitively plausible for out-ofcontext tasks and distribute pos annotation with the official release, to support further evaluation with pos-disambiguated dsm representations (e.g. @xcite . we chose encow because it is publicly available, large (10 bln words) and thus suitable for the extraction of reliable cooccurrence estimates and vector representations. fast contains lemmatized items. this choice has practical reasons -many dsms and other evaluation datasets are also lemmatized -and is supported by the literature. bullinaria and levy (2012) compare dsms built from raw, stemmed and lemmatized data. in a selection of semantic similarity tasks, a slight advantage is found for the lemmatized and stemmed models over the raw ones (with variations due to the different tasks, and to the in-2 corporafromtheweb.org/encow14/ teraction with other parameters). for a different selection of tasks, @xcite report best performance for stemmed data vs. inflected, lemmatized, pos-and ccg-tagged. 3 lapesa and evert (2013) obtain similar results for modeling semantic priming. we carried out lemmatization with morpha, a robust morphological analyzer. 4 unknown words were lemmatized based on their pos tag. all items in usf and eat were annotated with their encow lemma frequency (using the same lemmatization), as a basis for our sampling procedure. ## results tables 1 and 2 report evaluation results on the multiple-choice and lexical access task, respecto discarding the first svd dimensions, which @xcite found beneficial for modeling paradigmatic relations. combining first-order (collocations) and secondorder (dsms) information results in a further improvement, and a mixed span yelds the best performance in the multiple choice task on eat. for the lexical access task, the combined model with a mixed span consistenly outperforms both the 2word and 10-word combinations, even though the improvement over the 10-word span is small, showing that our assumptions regarding the complementarity of the two models may not entirely be justified. to further explore the complementarity between first-order (collocations) and second-order (dsms), we visualize the correlation between neighbour ranks of the two individual models in figures 1 (usf) and 2 (eat). each point corresponds to a test item, i.e. a pair of stimulus and first response, with dsm (2-word window, @xmath0. note that both axes are logarithmic and jitter has been added to visualize the large number of items where both ranks are 1 (etc.). the plots show that the two models are often in agreement, i.e. both the dsm and the first-order model put the first response among the top 5 ranks. indeed, there seems less complementarity to be exploited than we would have assumed, i.e. points in the bottom right or top left quadrants of the plot, for which the two models disagree substantially. ## conclusion this paper introduced the fast dataset and discussed extensive baseline results in two carefully designed tasks. the results on the multiple-choice task showed that first-order co-occurrence models outperform count-based dsms, due to the known prevalence of syntagmatic relations in free associations. combining the predictions of co-occurrence models and dsms turned out to be particularly advantageous, with results competitive to those achieved by the neural embeddings in particular for eat. in the lexical access task, dsms are slightly superior to co-occurrence models, and the combination of first-order and second-order data again leads to a substantial improvement. here, fast- besides the integration of more recent datasets @xcite , future work should target more sophisticated methods for the combination of different models, e.g., using machine learning techniques rather than just taking the harmonic mean as in our baseline experiments, and should also explore type-level representations derived from contextualized embeddings. as far as the interpretation of model predictions is concerned, fast lends itself to a multifactorial meta-analysis based on, e.g., abstractness ratings, semantic classes and ortographic properties of the stimulus. such rich information sources could also be integrated in the prediction step (e.g., constrain the system to predict associates at a similar level of abstractness to the stimulus).