U NDERSTANDING M ENTAL R EPRESENTATIONS O F O BJECTS T HROUGH V ERBS A PPLIED T O T HEM
Anonymous authors
Paper under double-blind review
A BSTRACT
In order to interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and the extent to which they rely or have an effect on the properties of the object.
This knowledge is called the object “affordance”.
We propose an approach for creating an embedding of objects in an affordance space, in which each dimension corresponds to an aspect of meaning shared by many actions, using text corpora.
This embedding makes it possible to predict which verbs will be applicable to a given object, as captured in human judgments of affordance, better than a variety of alternative approaches.
Furthermore, we show that the dimensions learned are interpretable, and that they correspond to typical patterns of interaction with objects.
Finally, we show that the dimensions can be used to predict a state-of-the-art mental representation of objects, derived purely from human judgements of object similarity.
I NTRODUCTION
In order to interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and their dependence (or effect) on properties of the object.
Gibson (1979) coined the term “affordance” to describe what the environment “provides or furnishes the animal”.
Norman (2013) developed the term to focus on the properties of objects that determine the action possibilities.
The notion of “affordance” emerges from the relationship between the properties of objects and human actions.
If we consider “object” as meaning anything concrete that one might interact with in the environment, there will be thousands of possibilities, both animate and inanimate (see WordNet (Miller, 1998)).
The same is true if we consider “action” as meaning any verb that might be applied to the noun naming an object (see VerbNet (Schuler, 2005)).
Intuitively, only a relatively small fraction of all possible combinations of object and action will be plausible.
Of those, many will also be trivial, e.g. “see” or “have” may apply to almost every object.
Finally, different actions might reﬂect a similar mode of interaction, depending on the type of object they are applied to (e.g. ”chop” and ”slice” are distinct actions, but they are both used in food preparation).
Mental representations of objects encompass many aspects beyond function.
Several studies (McRae et al., 2005; Devereux et al., 2014; Hovhannisyan et al., 2020) have asked human subjects to list binary properties for hundreds of objects, yielding thousands of answers.
Properties could be taxonomic (category), functional (purpose), encyclopedic (attributes), or visual-perceptual (appearance), among other groups.
While some properties were affordances in themselves (e.g. “edible”), others reﬂected many affordances at once (e.g. “is a vegetable” means that it could be planted, cooked, sliced, etc).
More recently, Zheng et al.
; Hebart et al.
(2020) introduced SPoSE, a model of the mental representations of objects.
The model was derived from a dataset of 1.5M Amazon Mechanical
Turk (AMT) judgments of object similarity, where subjects were asked which of a random triplet of objects was the odd one out.
The model was an embedding for objects where each dimension was constrained to be sparse and positive, and where triplet judgments were predicted as a function of the similarity between embedding vectors of the three objects considered.
The authors showed that these dimensions were predictable as a combination of elementary properties in the Devereux et al.
(2014) norm that often co-occur across many objects.
Hebart et al.
(2020) further showed that 1) human subjects could coherently label what the dimensions were “about”, ranging from categorical (e.g. is animate, food, drink, building) to functional (e.g. container, tool) or structural (e.g. made of metal or wood, has inner structure).
Subjects could also predict what dimension values new objects would have, based on knowing the dimension value for a few other objects.
SPoSE is unusual in its wide coverage – 1,854 objects – and in having been validated in independent behavioral data.
Our ﬁrst goal is to produce an analogous affordance embedding space for objects, where each dimension groups together actions corresponding to a particular “mode of interaction”affordance mining.
Our second goal is to understand the degree to which affordance knowledge underlies the mental representation of objects, as instantiated in SPoSE.
In this paper, we will introduce and evaluate an approach for achieving both of these goals.
Our approach is based on the hypothesis that, if a set of verbs apply to the same objects, they apply for similar reasons.
We start by identifying applications of action verbs to nouns naming objects, in large text corpora.
We then use the resulting dataset to produce an embedding that represents each object as a vector in a low-dimensional space, where each dimension groups verbs that tend to be applied to similar objects.
We do this for larger lists of objects and action verbs than previous studies (thousands in each case).
Combining the weights on each verb assigned by various dimensions yields a ranking over verbs for each concept.
We show that this allows us to predict which verbs will be applicable to a given object, as captured in human judgments of affordance.
Further, we show that the dimensions learned are interpretable, and they group together verbs that would all typically occur during certain complex interactions with objects.
Finally, we show that they can be used to predict most dimensions of the SPoSE representation, in particular those that are categorical or functional.
This suggests that affordance knowledge underlies much of the mental representation of objects, in particular semantic categorization.
R ELATED W ORK
The problem of determining, given an action and an object, whether the action can apply to the object was deﬁned as “affordance mining” in Chao et al.
The authors proposed complementary methods for solving the affordance mining problem by predicting a plausibility score for each combination of object and action.
The best method used word co-occurrences in two ways: ngram counts of verb-noun pairs, or similarity between verb and noun vectors in Latent Semantic
Analysis (Deerwester et al., 1990) or Word2Vec (Mikolov et al., 2013) word embeddings.
For evaluation, they collected AMT judgements of plausibility (“is it possible to < verb > a < object > ”) for every combination of 91 objects and 957 action verbs.
The authors found they could retrieve a small number of affordances for each item, but precision dropped quickly with a higher recall.
Subsequent work (Rubinstein et al., 2015; Lucy & Gauthier, 2017; Utsumi, 2020) predicted properties of objects in the norms above from word embeddings (Mikolov et al., 2013; Pennington et al., 2014), albeit without a focus on affordances.
Forbes et al.
(2019) extracted 50 properties (some were affordances) from Devereux et al.
(2014), for a set of 514 objects, to generate positive and negative examples for 25,700 combinations.
They used this data to train a small neural network to predict these properties.
The input to the network was either the product of the vectors for object and property, if using word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Levy & Goldberg, 2014), or the representation of a synthesized sentence combining them, if using contextualized embeddings (Peters et al., 2018; Devlin et al., 2018).
They found that the latter outperformed the former for property prediction, but none allowed reliable affordance prediction.
In addition to object/action plausibility prediction, Ji et al.
(2020) addressed the problem of determining whether a object1/action/object (target of the action with object1) was feasible.
They selected a set of 20 actions from Chao et al.
(2015) and combined them with the 70 most frequent objects in ConceptNet (Speer & Havasi, 2012) into 1400 object/action pairs, which were then labelled as plausible or not; given rater disagreements, this yielded 330 positive pairs and 1070 negative ones.
They then combined the positive pairs with other objects as ”tails” (recipients of the action), yielding 3900 triplets.
They reached F1 scores of 0.81 and 0.52 on the two problems, respectively.
Other papers focus on understanding the relevant visual features in objects that predict affordances, e.g. (Myers et al., 2015; Sawatzky et al., 2017;
Wang & Tarr, 2020).
The latter collected affordance judgments on AMT (“what can you do with < object > ”) for 500 objects and harmonized them with WordNet synsets for 334 action verbs.
For validation of the rankings of verb applicability predicted by our model, we will use the datasets from
Chao et al.
and Wang & Tarr (2020), as they are the largest available human rated datasets.
In robotics research, affordance refer to relation between agent, action and the environment, under the constraints of motor and sensing capabilities of the agent (Lopes et al., 2007).
Affordance modeling for robotics have been studied extensively, see recent surveys (Jamone et al., 2016; Zech et al., 2017; Hassanin et al., 2018).
Due to the restriction in action possibilities and the complexity of real world scenarios, poor semantic generalization of affordance inference is observed in visual and speech-based robotic systems.
Attempts to use semantic role labelling or text corpora, in tandem with visual features, have been proposed (e.g Persiani & Hellstr¨om (2019); Chen et al.
In general, though, this work focuses on a restricted set of objects and manipulation actions.
In computational linguistics, (Resnik, 1993) introduced computational approaches to determining selectional preference, the degree to which a particular semantic class tends to be used as an argument to a given predicate.
From this perspective, the method we introduce in this paper provides a way of scoring verbs for how well they would apply to objects.
All methods proposed to do this take advantage of some grouping of verbs and objects into classes (e.g. WordNet in Resnik (1996), or co-occurrence statistics of words in a corpus (Erk, 2007;
Pad´o et al., 2007; S´eaghdha, 2010; Van
De Cruys, 2014; Zhang et al., 2020).
Erk (2007) used similarities of co-occurrence patterns for words to compute selectional preferences for semantic roles in FrameNet.
Pad´o et al.
(2007) used the same similarity function to predict the plausibility of verb/relation/argument triples.
S´eaghdha (2010) learned a custom topic model with separate latent variables (and dictionaries) for verbs and nouns.
Van De Cruys (2014) trained a neural network to predict preference scores for combinations of verbs or objects, represented via embedding vectors.
Zhang et al.
(2020) learns embeddings for individual words together with modiﬁcations for when the word is used in a certain relation.
It scores combinations of words by similarity of the modiﬁed embedding vectors.
All of these methods could be used to score verbs by how applicable they are to a given noun, the ancillary task we use to make sure our proposed embedding carries the relevant information.
That said, they would require substantial implementation effort, even where the code is publicly available.
This said, making that prediction is not our main goal.
Our proposed embedding space is a latent variable model for verb-noun applications.
While this is also the case for the dual topic model in (S ´ eaghdha, 2010) the internal representation in Van De Cruys (2014), or the embeddings of nouns/verbs in Zhang et al.
(2020), they would all require extensive modiﬁcation to add sparsity assumptions – important for interpretability – and to produce combined verb rankings for embedding vectors.
D ATA AND M ETHODS 3. O BJECTS AND A CTIONS CONSIDERED
In this paper, we use the list of 1854 object concepts introduced in (Hebart et al., 2019) and for which SPoSE embeddings are available.
This list sampled from concrete, picturable, and nameable nouns in American English, and was further expanded by a crowdsourced Amazon Mechanical Turk study to elicit category membership ratings.
The following 27 categories accounted for most of the objects: food, animal, clothing, tool, drink, vehicle, fruit, vegetable, body part, toy, container, bird, furniture, sports equipment, musical instrument, dessert, part of car, weapon, plant, insect, kitchen tool, ofﬁce supply, clothing accessory, kitchen appliance, home decor, medical equipment, and electronic device.
As we are not doing sense disambiguation for each noun that names an object, we will use noun or object interchangeably throughout the paper.
We created our own verb list by having three annotators go through all verb categories on VerbNet (Schuler, 2005), and selecting those that included verbs that corresponded to an action performed by a human on an object.
We kept all the verbs in each selected category, and selected only those categories where all annotators agreed.
Those VerbNet categories contained ∼ 10 − 50 verbs sharing thematic roles and selectional preferences (e.g. ﬁll-9.8, amalgamate-22.2, manner-speaking-37.3, build-26.1, remove-10.1, cooking-45.3, create-26.4, destroy-44, mix-22.1, vehicle-51.4.1, dress-41.1.1).
The resulting list has 2541 verbs.
3. E XTRACTION OF V ERB A PPLICATIONS TO N OUNS FROM T EXT C ORPORA
We used the UKWaC and Wackypedia corpora (Ferraresi et al.)
containing, approximately, 2B and 1B tokens, and 88M and 43M sentences, respectively.
The former is the result of a crawl of British web pages, while the latter is a subset of Wikipedia.
Both have been extensively cleaned up and have clearly demarcated sentences, which makes them ideal for dependency parsing.
We replaced all common bigrams in Brysbaert et al.
(2014) by a single token.
We identiﬁed all sentences containing both verbs and nouns in our list, and we used Stanza (Qi et al., 2020) to produce dependency parses for them.
We extracted all the noun-verb pairs in which the verb was a syntactic head of a noun having obj (object) or nsubj:pass (passive nominal subject) dependency relations.
We compiled raw counts of how often each verb was used on each noun, producing a count matrix M .
Note that this is very different from normal co-occurrence counts - those would register a count whenever verb and noun were both present within a short window (e.g. up to 5 words away from each other), regardless of whether the verb applied to the noun, or they were simply in the same sentence.
Out of the 1854 nouns considered, there were 1755 with at least one associated action, and this is the list we will use in the remainder of this paper.
Note also that the counts pertain to every possible meaning of the noun, given that no word sense disambiguation was performed.
Finally, we converted the matrix M into a Positive Pointwise Mutual Information (PPMI (Turney &
Pantel, 2010))
matrix P where, for each object i and verb k :
[Equation]
where P ( M i ∗ ) and P ( M ∗ k ) are respectively the marginal probability of i and k .
P can be viewed as a pair-pattern matrix (Lin & Pantel, 2001), where the PPMI helps in separating frequency from informativess of the co-occurrence of nouns and verbs (Turney & Pantel, 2010; Turney & Littman, 2003).
However, PPMI is also biased and may be large for rare co-occurrences, e.g. for ( o i , v k ) that co-occur only once in M .
This is addressed in the process described in the next section.
3. O BJECT EMBEDDING IN A VERB USAGE SPACE
Object embedding via matrix factorization Our embedding is based on a factorization of the PPMI matrix P ( m objects by n verbs) into the product of low-rank matrices O ( m objects by d dimensions) and V ( n verbs by d dimensions), yielding (cid:101) P :=
OV T ≈ P .
O is the object embedding in d -dimensional space, and V is the weighting placed on each verb by each dimension.
Intuitively, if two verbs occur often with the same objects, they will both have high loadings on one of the d -dimensions; conversely, the objects they occur with will share high loadings on that dimension.
The idea of factoring a count matrix (or a transformation of it) dates back to Latent Semantic Analysis (Landauer & Dumais, 1997), and was investigated by many others ((Turney & Pantel, 2010) is a good review).
Given that PPMI is biased towards rare pairs of noun/verb, the matrix P is not necessarily very sparse.
If factorized into a product of two low-rank matrices, however, the structure of the matrix can be approximated while excluding noise or rare events (Bullinaria & Levy, 2012).
Optimization problem Given that PPMI is positive, the matrices O
and V are as well.
This allows us to obtain them through a non-negative matrix factorization (NMF) problem
[Equation]
which can be solved through an iterative minimization procedure.
For the regularization R ( O, V ) , we chose the sparsity control R ( O, V ) ≡ (cid:80) ij O ij + (cid:80) ij V ij .
In our experiments, we use d = 70 and β = 0 .
These values were found using the two-dimensional hold-out cross validation (Kanagal & Sindhwani, 2010), due to its scalability and natural ﬁt to the multiplicative update algorithm for solving (2).
Speciﬁcally, denoting M t and M v to be the mask matrices representing the held-in and held-out entries, we optimize for
[Equation]
and obtain the reconstruction error E = (cid:107) M v (cid:12) ( P − O
[Equation]
V
[Equation]
) T ) (cid:107) 2 F + β R ( O ∗ , V ∗ ) .
For more details, please refer to Appendix A.
The optimization problem (2) is NP-hard and all state-of-the-art algorithms may converge only to a local minimum (Gillis, 2014); choosing a proper initialization of O and V is crucial.
We used the NNDSVD initialization (Boutsidis & Gallopoulos, 2008), a
SVD-based initialization which favours sparsity on O and V and approximation error reduction.
Estimating the verb usage pattern for each object Each column V : ,k of matrix V contains a pattern of verb usage for dimension k , which captures verb co-occurrence across all objects.
Deriving a similar pattern for each object i , given its embedding vector O i, : =
[ o i 1 , o i 2 , . . .
o
i d ] , requires combining these patterns based on the weights given to each dimension.
The ﬁrst step in doing so requires computing the cosine similarity between each embedding dimension O : ,h and the PPMI values ˜ P : ,k for each verb k in the approximated PPMI matrix ˜ P = OV T , which is
[Equation]
Given the embedding vector for object i , O i, : =
[ o i 1 , o i 2 , . . .
o
i d ] , we compute the pattern of verb usage for the object as O i, : S .
Thus, this is a weighted sum of the similarity between every O : ,h and ˜
P : ,k .
We will refer to the ordering of verbs by this score as the verb ranking for object i .
E XPERIMENTS AND R ESULTS 4. P REDICTION OF AFFORDANCE PLAUSIBILITY
Affordance ranking task The ﬁrst quantitative evaluation of our embedding focuses on the ranking of verbs as possible affordances for each object.
We will use the Affordance Area Under The Curve (AAUC) relative to datasets that provide, for each object, a set of verbs known (or likely) to be affordances.
Intuitively, the verb ranking for object i is good if it places these verbs close to the top of the ranking, yielding an AAUC of 1.
Conversely, a random verb ranking would have an AAUC of 0.5, on average.
Note that this is a conservative measure, given that a perfect ranking would still penalize every true affordance not at the top.
Hence, this is useful as a relative measure for comparing between our and competing approaches for producing rankings.
More formally, given the K ground truth verb affordances { g k } Kk =1 of object i , and its verb ranking { v i } ni =1 , we denote (cid:96) k
to be the index such that v (cid:96) k
= g k ∀ k .
We then deﬁne AUCC for object i as AUCC =
1 K (cid:80) Kk
=1 (cid:0) 1 −
[Table]
Datasets We used the two largest publicly available object affordance datasets as ground truth.
In the ﬁrst dataset, WTAction (Wang & Tarr, 2020), objects are associated with the top 5 actions label provided by human annotators in response to “What can you do with this object?”.
Out of 1,046 objects and 445 actions in this dataset, there are 971 objects and 433 verbs that overlap with those in our lists ( ∼ 3.12 action labels per object) .
The second dataset, MSCOCO (Chao et al., 2015), scores every candidate action for an object ranging from 5.0 (“deﬁnitely an affordance”) to 1.0 (“deﬁnitely not an affordance”) marked by 5 different workers.
We consider only a 5.0 score as being an affordance, whereas Chao et al.
(2015) used both 4.0 and 5.0.
Out of 91 objects andactions
, there are 78 objects and 558 verbs that overlap with ours ( ∼ 34 action labels per object).
Baseline methods We compared the ranking of verbs produced by our algorithm with an alternative proposed in (Chao et al., 2015): ranking by the cosine similarity between word embedding vectors for each noun and those for all possible verbs in the dataset.
We considered several off-the-shelf embedding alternatives, namely Word2Vec (Mikolov et al.
(2013), 6B token corpus), GloVe (Pennington et al.
(2014), 6B and 840B token corpora, and our 2B corpus), Dependency-Based Word
Embedding (DBWE, Levy & Goldberg (2014), 6B corpus), and Non-negative Sparse Embedding (NNSE,Murphy et al.
(2012), 16B corpus).
Finally, we included the other two methods in (Chao et al., 2015), LSA (Deerwester et al.
(1990), trained on our corpus), and ranking by frequency of verb/noun pair in Google N-grams (Lin et al.
The embeddings are 300-D in all cases, except for NNSE (1000-D, results are similar for 2500-D).
We contrasted Word2Vec and GloVe because they are based on two different embedding approaches (negative sampling and decomposition of a word co-occurrence matrix), developed on corpora twice as large as ours.
We contrasted 6B and 840B versions of GloVe to see the effect of increasing dataset size, and 2B to show the effect of our corpus.
In these methods, the co-occurrence considered is simply proximity within a window of a few tokens, rather than application of the verb to the noun.
We included DBWE because it uses dependency parse information (albeit to deﬁne the word co-occurrence window, rather than select verb applications to nouns as we do).
We included NNSE because it is based on a sparse, non-negative factorization of a co-occurrence matrix of features derived from words combined with speciﬁc dependency relations.
Finally, we ranked the verbs by their values in the row of the PPMI matrix P corresponding to each probed object, to see the effect of using a low-rank approximation in extracting information.
Results For each dataset, we reduced our embeddings O and V according to the sets of objects and verbs available.
We then obtained the verb ranking for each object, as described in Section 3.3, [Table] as well as rankings predicted with the different baseline methods in the previous section.
Tableshows the AAUC results (in terms of the mean µ and the standard error S E of AAUCs) obtained with these verb rankings on the two datasets.
Our ranking is better than those of all the baseline methods, as determined from a paired two-sided t -test, in both WTAction ( p -values of 7 . 36e − 23 , 4 .
19e − 14 , 6 . 82e − 174 , 1 .
34e − 53 , 1 .
44e − 89 , 5 .
21e − 47 , 8 .
70e − 25 and 8 . 14e − 39 ) and MSCOCO ( p -values of 1 .
54e − 23 , 3 .
37e − 21 , 9 .
32e − 36 , 2 . 15e − 27 , 2 . 82e − 30 , 9 , 64e − 20 , 7 . 93e − 17 and 2 .
13e − 29 ).
The overall distributions of AAUCs for the two datasets are shown in Appendix B. Our procedure yields AAUC closer to 1.0 for many more items than the other methods.
This suggests that the cooccurrence of verb and noun within a window of a few tokens, the basis of the word embeddings that we compare against, carries some information about affordances, but also includes other relationships and noise (e.g. if the verb is in one clause and the noun in another).
Results are better in the embedding trained in a corpus 280X larger, but still statistically worse than those of our procedure.
Ranking based on the PPMI matrix P performs at the level of the 6B token embeddings.
This suggests that our procedure is effective at removing extraneous information in the matrix P .
4. P REDICTION OF SP O SE OBJECT REPRESENTATIONS FROM OUR AFFORDANCE EMBEDDING
The SPoSE representation and dataset The dimensions in the SPoSE representation (Hebart et al., 2020) are interpretable, in that human subjects coherently label what those dimensions are “about”, ranging from the categorical (e.g. animate, building) to the functional (e.g. can tie, can contain, ﬂammable) or structural (e.g. made of metal or wood, has inner structure).
Furthermore, subjects could also predict what dimension values new objects would have, based on knowing the dimension value for a few other objects.
The SPoSE vectors for objects are derived solely from behaviour in a “which of a random triplet of objects is the odd one out” task.
The authors propose a hypothesis for why there is enough information in this data to allow this: when given any two objects to consider, subjects mentally sample the contexts where they might be found or used.
The resulting dimensions reﬂect the aspects of the mental representation of an object that come up in that sampling process.
The natural question is, then, which of these dimensions reﬂect affordance or interaction information, and that is what our experiments aim to answer.
For our experiments, we used the 49-D SPoSE embedding published with (Hebart et al., 2020).
Out of these, we excluded objects named by nouns that had no verb co-occurrences in our dataset and, conversely, verbs that had no interaction with any objects.
We averaged the vectors for each set of objects named by the same polysemous noun (e.g. “bat”).
This resulted in a dataset ofobjects/nouns, with their respective SPoSE embedding vectors, and 2462 verbs.
Prediction of SPoSE meaning dimensions from affordance embedding The ﬁrst experiment we carried out was to predict the SPoSE dimensions for an object from its representation in our affordance embedding.
The ability to do this tells us which SPoSE dimensions – and, crucially, which kinds of SPoSE dimensions – can be explained in terms of affordances, to some extent.
Denoting the SPoSE embeddings for m objects as a m × 49 matrix Y, we solved the following Lasso regression problem for each column Y : ,i
[Equation]
where λ was chosen based on a 2-Fold cross-validation, with λ in [10e − 7 , 10e 3 ] with log-scale spacing.
Since both Y : ,i and our embedding O represent object features by positive values, we restricted w ≥ 0 .
Intuitively, this means that we try to explain every SPoSE dimension by combination of the presence of certain affordance dimensions, not by trading them off. Figure 1 shows the predictions ˜ Y : ,i = Ow ∗ i ,
[Figure]
f oo d a n i m a l c o n t a
i n e r t oo l c l o t h i n g e l e c t r o n i c
d e v
i c e v e h
i c
l e s p
o r t s e q u i p m e n t p l a n t h
o m e
d e c o r w e a p o n t o y v e g e t a b l e d e ss e r t f u r n i t u
r e f r u i t
b o d y p a r t c l o t h
i n g a cc e ss
o r y m u
s i c a l i n s t r u m e n t p a r t o f c a r o ff i
c e s u pp l y m e
d i c a l e q u
i p m e n t b i r d k i t c h e n t oo l d e c o r a t i o n k
i t c
h e n a pp li
a n c e d r i n k i n s e c t
[Figure]
plotted for categories with size ≥ 15 .
The visual resemblance of the patterns, and the range of correlations between true and predicted dimensions (top: 0.84, mean: 0.46), see Figure 2b for distribution), indicate that the affordance embedding contains enough information to predict most SPoSE dimensions.
Relationship between SPoSE and affordance dimensions We ﬁrst considered the question of whether affordance dimensions correspond directly to SPoSE dimensions, by looking for the best match for each of the latter in terms of correlation.
As shown in Figure 2(a), most SPoSE dimensions have one or more affordance dimensions that are similar to them.
However, when we consider the cross-validated regression models to predict SPoSE dimensions from affordance dimensions, every model places non-zero weight on several of the latter.
Furthermore, those predictions are much more similar to SPoSE dimensions than almost any individual affordance dimension.
Figure 2(b) plots, for every SPoSE dimension, the correlation with its closest match (x-axis) versus the correlation with the cross-validated prediction of the regression model (y-axis).
The best predicted dimensions are categorical, e.g. “animal”, “plant”, or “tool”, or functional, e.g. “can tie” or “ﬂammable”.
Structural dimensions are also predictable, e.g. “made of metal”, “made of wood”, or “paper”, but appearance-related dimensions, e.g. “colorful pattern”, “craft”, or “degree of red”, less so.
What can explain this pattern of predictability?
Most SPoSE dimensions can be expressed as a linear combination of affordance dimensions, where both the dimensions and regression weights are nonnegative .
This leads to a sparse regression model – since dependent variables cannot be subtracted to improve the ﬁt – where, on average, 5 affordance dimensions have 80% of the regression weight.
Each affordance dimension, in turn, corresponds to a ranking over verbs.
Figure 3a shows the topverbs in the 5 most important affordance dimensions for predicting the “animal” SPoSE dimension.
As each affordance dimension loads on verbs that correspond to coherent modes of interaction (e.g. observation, killing, husbandry), the model is not only predictive but also interpretable.
Whereas we could also use dense embeddings to predict SPoSE dimensions, they do not work as well (in either
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
x=y
[Figure]
accuracy or interpretability, see Figure 3b for GloVe). If we consider the top 5 verbs from affordance
[Figure]
dimensions that are most used in predicting each SPoSE dimension, we see that “tool” has “sharpen, blunt, wield, plunge, thrust” (D50); “animal” has “spot, observe, sight, hunt, watch” (D3), “hunt, kill, cull, exterminate, chase” (D37), or “pasture, herd, slaughter, milk, tether” (D18); “food” has “serve, eat, cook, prepare, order” (D15), or “bake, leaven, ice, eat, serve” (D64); “plant” shares D with “food”, but also has “cultivate, grow, plant, prune, propagate” (D57).
The full list of affordance dimensions most relevant for predicting each SPoSE dimension is in Appendix F (Figure 8–13).
These results suggest that SPoSE dimensions are predictable insofar as they can be expressed as combinations of modes of interaction with objects.
Using the approach in Section 3.3, we produced a combined ranking over verbs for each SPoSE dimension.
We replaced the embedding O in (4) with the SPoSE prediction (cid:101) Y and we ranked the verbs for dimension h according to S ( ˜ Y : ,h , ˜ P k ) .
Tableshows, for every SPoSE dimension, ranked by predictability, the top 10 verbs in its ranking.
Testing against a null hypothesis of 0 correlation, the p-values obtained range from 0.0 to the maximum of 7.61e-7 for the SPoSE dimension “furry”, indicating the statistical signiﬁcance of correlation with our regression outcomes.
Examining this table provides empirical conﬁrmation that, ﬁrst and foremost, highly predictable categorical dimensions correspond to very clean affordances.
The same is true for functional dimensions, e.g. “can tie” or “container” or “ﬂammable”; even though they are not “classic” categories, subjects group items belonging to them based on their being suitable for a purpose (e.g. “fasten”, “ﬁll”, or “burn”).
But why would this hold for structural dimensions?
One possible reason is that objects having that dimension overlap substantially with a known category (e.g. “made of metal” and “tool”).
Another is that the structure drives manual or mechanical affordance (e.g. “elongated” or “granulated”).
Finally, what are the affordances for appearance dimensions that can be predicted?
Primarily, actions on items in categories that share that appearance, e.g. “textured” is shared by fabric items, “round” is shared by many fruits or vegetables.
Prediction is worse when the items sharing the dimension come from many different semantic categories ((Hebart et al., 2019) lists the pictures of items shown to subjects)
[Table]
C ONCLUSIONS
In this paper, we introduced an approach to embed objects in a space where every dimension corresponds to a pattern of verb applicability to those objects.
We view such a pattern as a very broad extension of the classical notion of ”affordance”, obtained by considering verbs that go well beyond motor actions, and objects that encompass many different categories beyond tools or household objects.
We showed that this embedding can be learned from a text corpus and used to rank verbs by how applicable they would be to a given object.
We evaluated this prediction against two separate human judgment ground truth datasets, and veriﬁed that our method outperforms general-purpose word embeddings trained on much larger text corpora.
This gave us with conﬁdence that the patterns of applicability we identiﬁed captured information relevant to human judgments of affordance.
We used our embedding to predict SPoSE dimensions for objects.
SPoSE is an embedding that was derived from human behavioural judgements of object similarity, and has interpretable dimensions.
We used the resulting prediction models to probe the relationship between the dimensions of our embedding and the various types of SPoSE dimensions.
This allowed us to conclude that our ”affordance” embedding knowledge predicts 1) category information, 2) purpose, and 3) some structural aspects of the object.
SPoSE dimensions to do with visual appearance were poorly predicted.
Our embedding is thus sufﬁcient for predicting most SPoSE dimensions.
To be able to go further this, and conclude that our embedding is a valid model of the mental representations of objects – insofar as our uses for them go – would require additional experiments.
One possibility would be to run an to explicitly ask human subjects ”given objects that load highly on this embedding dimension, what can you do with them”, and consider the typicality of verb answers against the weight given to those verbs by the dimension.
Given that our embedding is based on language data about which verbs apply to which objects, we would expect these experiments to give verb loadings coherent with ours.
To increase prediction quality in future work, one approach will be to enrich and reﬁne the cooccurrence matrix in larger corpora, now that the basic approach has been shown to be feasible.
Another interesting future direction will be to understand how affordance could be driven by more ﬁne-grained visual appearance properties, by considering other semantic dependencies, or jointly using text and image features such as (Wang & Tarr, 2020).
A A PPENDIX : H YPER - PARAMETER S ELECTION FOR N ON - NEGATIVE M ATRIX F ACTORIZATION
[Figure]
Denote M t , M v ∈ { 0 , 1 } n × m to be the mask matrices for indicating held-in and held-out entries of the input PPMI matrix P in CV procedure, we then optimize for O ∗ and V ∗ :
[Equation] [Equation]
To apply the multiplicative update scheme as proposed in (Lee & Seung, 2001), we ﬁrst consider the partial derivatives with respect to O and V .
Denote F ( O, V ) ≡ (cid:107) M t
[Equation]
( P − OV T )
[Equation]
2 F +
[Equation]
O
[Equation]
V ) , we have
[Equation]
We then have the following update rules that is guaranteed to be non-increasing:
[Equation]
where the fraction here represents elementary-wise division.
For the choice of M t and M v , we follow the same approach as proposed in (Kanagal & Sindhwani, 2010).
We ﬁrst split the matrix into K blocks, where the rows and columns are randomly shufﬂed.
Denote r ( k ) and c ( k ) to be the index vectors for rows and columns respectively, where r ( k ) i
= 1 if row i contains in block k , or c ( k ) j = 1 if column j contains in block k .
The mask for k -th block can then be expressed as M ( k ) =
r ( k ) ⊗ c ( k ) .
We then randomly select q out of K blocks as holdout blocks, which gives
[Equation]
where k s is the index of selected block.
The reconstruction error E can thus be computed:
[Equation]
Figure 4 shows a zoom-in plot of the reconstruction error under different combinations of d and β .
For every ( d , β ) setting, we perform multiple optimization since NMF is sensitive to initialization.
We then choose d = 70 and β = 0 .
3 accordingly.
Empirically, we observe that the rank selection is quite robust to over-ﬁtting when there is a sufﬁcient sparsity control, for instance, β > 0 .
1 in our dataset.
We also observe that whenever d ∈ [50 , 150] and β ∈ [0 . 05 , 0 . 5] , the results are similar.
B A PPENDIX : D ISTRIBUTION OF AAUC S ON WT ACTION AND MSCOCO D ATASET
The following ﬁgures show the AAUC distribution of the top 5 embeddings on the WTaction and
MSCOCO dataset respectively
[Figure]
[Figure]
C A PPENDIX : P REDICTION OF SP O SE DIMENSIONS
[Figure]
f oo d a n i m a l c o n t a
i n e r t oo l c l o t h i n g e l e c t r o n i c
d e v
i c e v e h
i c
l e s p
o r t s e q u i p m e n t p l a n t h
o m e
d e c o r w e a p o n t o y v e g e t a b l e d e ss e r t f u r n i t u
r e f r u i t
b o d y p a r t c l o t h
i n g a cc e ss
o r y m u
s i c a l i n s t r u m e n t p a r t o f c a r o ff i
c e s u pp l y m e
d i c a l e q u
i p m e n t b i r d k i t c h e n t oo l d e c o r a t i o n k
i t c
h e n a pp li
a n c e d r i n k i n s e c t
[Figure]
D A PPENDIX : A FFORDANCES A SSIGNMENT FOR EACH SP O SE D IMENSION
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
E A PPENDIX : F ULL DESCRIPTIONS FOR EACH SP O SE DIMENSION
The abbreviation and the full description of every SPoSE dimension.
[Table] [Table]
F A PPENDIX : C OMPONENTS OF SP O SE DIMENSION APPROXIMATION
The following consecutive Figures 8–13 show the component information of each SPoSE dimension.
The percentage is calculated based on the portion w i · (cid:107) O : ,i (cid:107) , where w i is the regression coefﬁcient corresponding to our i -th embedding dimension.
The right ticks show the dimension affordances.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]