WAX : A New Dataset for Word Association e X planations (2022)

Abstract: Word associations are among the most common paradigms to study the human mental lexicon. While their structure and types of associations have been well studied, surprisingly little attention has been given to the question of why participants produce the observed associations. Answering this question would not only advance understanding of human cognition, but could also aid machines in learning and representing basic commonsense knowledge. This paper introduces a large, crowd-sourced data set of English word associations with explanations, labeled with high-level relation types. We present an analysis of the provided explanations, and design several tasks to probe to what extent current pre-trained language models capture the underlying relations. Our experiments show that models struggle to capture the diversity of human associations, suggesting WAX is a rich benchmark for commonsense modeling and generation.

## introduction word associations @xcite are a prevalent paradigm in cognitive science for probing the human mental lexicon @xcite . they reflect spontaneous human associations between concepts. in a typical study, a participant is presented with a cue word (e.g., bagpipe) and asked to spontaneously produce the words that come to mind in response (music, . . . ). through large-scale crowd-sourcing studies covering over 12k cues, 3m responses and thousands of participants, a large word association graph @xcite ) has been constructed, as a resource of basic human conceptual knowledge. this repository of shared associations can serve as a source of commonsense knowledge as shown recently by incorporating swow as knowledge resource into commonsense reasoning models @xcite . however, existing word association data sets like swow only provide cue-association pairs, but do not further distinguish between different types of associations. to fill this gap, we constructed a novel data set to recover the underlying reasons by collecting associations together with free-text explanations from participants, and distill high-level relation types from them. our data set can enhance our understanding of the reasons and types for conceptual associations in humans, and can serve as an explicit knowledge resource for reasoning models. our data set wax (word association explanations) encodes english word associations with diverse explanations and high-level relation types and is illustrated in figure 1 . in a large crowd-sourcing study, we (a) collected human word associations by presenting participants with a cue word (bagpipe) and collecting the association words that spontaneously came to mind (music, kilt, . . . ) (figure 1 , circles); (b) asked the same participants to explain the link between the cue and their corresponding associations in a short sentence (figure 1 , speech bubbles); and (c) labeled explanations with a relation type adpated from a predefined set @xcite (e.g., function, edge labels in figure 1 ). we ensure data quality through several layers of careful annotator training and data filtering. compared to existing work on categorizing word associations @xcite , wax is larger in size, grounds associations in explanations, and will be released to the research community, supporting future research on understanding and modeling conceptual knowledge. wax complements existing commonsense knowledge graphs, which either involved decades of manual work (conceptnet; @xcite ), rely on highly templated responses, limiting their ability to reflect the natural diversity in human associations (atomic; @xcite ); or only indirectly link concepts via a shared scene (com-mongen; lin et al. ( 2020) ). wax results from a new, scalable method of collecting general commonsense knowledge, while maintaining both quality and diversity of associations and explanations, and can be cheaply extended to other languages. we annotated a subset of wax with high-level, discrete relation labels, enabling us to quantify the diversity of human mental relations, and to evaluate machine learning models in their ability to (a) distinguish different relations; and (b) generate plausible association explanations. our experiments using pre-trained language models demonstrate the value of wax as a rich and challenging data set for a variety of commonsense modeling and generation tasks. in sum, our main contributions are: ## background our work relates to several research lines, including word associations, commonsense knowledge graphs, and explainability. word associations word associations, as reflections of human mental lexica, have been studied extensively in psychology @xcite . in early studies, word associations were predominantly collected on a small scale from homogeneous participants @xcite . recently, crowd-sourcing has proved effective for collecting large-scale word association data sets in several languages, i.e., english @xcite , dutch @xcite and japanese @xcite . among them, @xcite is the largest multi-lingual word association graph, covering 14 languages. 2 however, the graphs only include directed associations between words pairs, rendering the underlying reasons for association unknown. types of mental associations were previously studied in cognitive psychology @xcite @xcite @xcite . previous work @xcite showed that relations of word associations can be recovered by (1) asking subjects to explain (in words or in writing) the reasons for the produced association, then (2) inferring a relation based on the explanations. we follow the methodology from the above works both to recover the association reasons (see our method description in §3) and to label a subset of our word associations with relation types. in contrast with previous work, where collected data sets were small (e.g., 100 cues) and were not made available to the research community, we provide a large-scale data set by gathering explicit explanations and relation types, to encourage future work on automatic association inference and relation labeling. several relation type ontologies have been proposed @xcite @xcite @xcite , which typically distinguish four broad relation categories: taxonomic (apple, pear), situational (airplane, travel), properties (sweater, comfortable), and linguistic/form (hobby, lobby). al. (2012) build on the broad categories above, and refine them into a total of 28 subtypes, which we used as the basis for our own association labeling scheme ( §3.2). ## the wax corpus we present our two-stage framework for collecting word association relations between pairs of concepts (words) by crowd-sourcing explicit explanations of the relations (figure 2 ). in phase 1, we collect associations and free-text explanations to elicit the underlying reasoning. in phase 2, we label a subset of (cue, association, explanation)-tuples (c, a, e) foot_1 with relation types r to characterize the inventory of common relation types. appendix a contains details on annotator instructions and payment, as well as quality control. ## conclusion word associations have been used as a lens into human conceptual representations for a long time, however, the types and reasons of these associations have not been studied at scale. we presented wax, a large data set of word associations with explanations and relation labels. wax is both an opportunity better understand the human mental lexicon, and a repository of relational commonsense knowledge both structured as (c, a, r) tuples, and free-text through the associated explanations. we demonstrated the utility of wax for supervised relation classification and explanation generation; and presented a detailed data set analysis including association diversity and data-driven relation types. in future work, we plan to use wax in tasks such as automatically labelling edges in commonsense knowledge graphs, commonsense question answering, and natural language inference.