Paper:
Verb Physics: Relative Physical Knowledge of Actions and Objects
Learning commonsense knowledge from natural language text is nontrivial due to reporting bias: people rarely state the obvious, e.g., “my house is bigger than me”. However, while rarely stated explicitly, this trivial everyday knowledge does influence the way people talk about the world, which provides indirect clues to reason about the world. For example, a statement like “John entered his house” implies that his house is bigger than John. In this paper, we present an approach to infer relative physical knowledge of actions and objects along six dimensions (e.g., size, weight, and strength) from unstructured natural language text. We frame knowledge acquisition as joint inference over two closely related problems: learning (1) relative physical knowledge of object pairs and (2) physical implications of actions when applied to those object pairs. Empirical results demonstrate that it is possible to extract knowledge of actions and objects from language and that joint inference over different knowledge types improves performance.
1 000
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
089
090
091
092
093
094
095
096
097
098
099
In this paper, we present an approach to infer relative physical knowledge of actions and objects along six dimensions (e.g., size, weight, and strength) from unstructured natural language text. We frame knowledge acquisition as joint inference over two closely related problems: learning (1) relative physical knowledge of object pairs and (2) physical implications of actions when applied to those object pairs. Empirical results demonstrate that it is possible to extract knowledge of actions and objects from language and that joint inference over different knowledge types improves performance.
1 Introduction
Reading and reasoning about natural language text often requires trivial knowledge about everyday physical actions and objects. For example, given a sentence “Martin could fit the trophy into the suitcase”, we can trivially infer that the trophy must be smaller than the suitcase, even though it’s not stated explicitly. This reasoning requires knowledge about the action “fit”, in particular, typical preconditions that need to be satisfied in order to perform the action. In addition, reasoning about
Natural language clues
Relative physical knowledge about objects
Physical implications of actions
“She barged into the stable.”
HUMAN STABLE
size: smaller
weight: lighter speed: faster
strength: n/a
rigidness: less rigid
x barged into y
⇒ x is smaller than y
⇒ x is lighter than y
⇒ x is faster than y
⇒ x is less rigid than y
the applicability of various physical actions in a given situation often requires background knowledge about objects in the world, for example, that people are usually smaller than houses, that cars generally move faster than humans walk, or that a brick probably is heavier than a feather.
In fact, the potential use of such knowledge about everyday actions and objects can go beyond language understanding and reasoning. Many open challenges in computer vision and robotics may also benefit from such knowledge, as shown
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
in recent work that requires visual reasoning and entailment (Izadinia et al., 2015; Zhu et al., 2014). Ideally, an AI system should acquire such knowledge through direct physical interactions with the world. However, such a physically interactive system does not seem feasible in a foreseeable future.
In this paper, we present an approach to acquire trivial physical knowledge from unstructured natural language text as an alternative knowledge source. In particular, we focus on acquiring relative physical knowledge of actions and objects organized along five dimensions: size, weight, strength, rigidness, and speed. Figure 1 illustrates example knowledge of (1) relative physical relations on object pairs and (2) physical implications of actions when applied to those object pairs.
While natural language text is a rich source to obtain broad knowledge about the world, compiling trivial commonsense knowledge from unstructured text is a nontrivial feat. The central challenge lies in reporting bias: people rarely states the obvious (Gordon and Van Durme, 2013; Sorower et al., 2011), since it goes against Grice’s conversational maxim on the quantity (Grice, 1975) of information.
In this work, we demonstrate that it is possible to overcome reporting bias and still extract the unspoken knowledge from language. The key insight is this: there is consistency in the way people describe how they interact with the world, which provides vital clues to reverse engineer the common knowledge shared among people. More concretely, we frame knowledge acquisition as joint inference over two closely related puzzles: inferring relative physical knowledge about object pairs while simultaneously reasoning about physical implications of actions.
Importantly, four of five dimensions of knowledge in our study — weight, strength, rigidness, speed — are either not visual or not easily recognizable from image recognition using currently available computer vision techniques. Thus, our work provides unique values to complement recent attempts to acquire commonsense knowledge from web images (Izadinia et al., 2015; Bagherinezhad et al., 2016; Sadeghi et al., 2015).
In sum, our contributions are threefold:
• We introduce a new task of commonsense knowledge extraction from language, focusing on physical implications of actions and relative physical relations among objects, or-
ganized along five dimensions. • We propose a model that can infer relations
over grounded object pairs together with first order relations implied by physical verbs. • We develop a new dataset VERBPHYSICS
that compiles crowd-sourced knowledge of actions and objects.1
The rest of the paper is organized as follows. We first provide the formal definition of knowledge we aim to learn in Section 2. We then describe our data collection in Section 3 and present our inference model in Section 4. Empirical results and related work follow in Sections 5 and Section 6. We conclude at Section 7.
2 Representation of Relative Physical Knowledge
2.1 Knowledge Dimensions
We consider five dimensions of relative physical knowledge in this work: size, weight, strength, rigidness, and speed. “Strength” in our work refers to the physical durability of an object (e.g., “diamond” is stronger than “glass”), while “rigidness” refers to the physical flexibility of an object (e.g., “glass” is more rigid than a “wire”). When considered in verb implications, size, weight, strength, rigidness generally concerns pre-conditions of the action, while speed concerns the post-condition of the action.
2.2 Relative physical knowledge
Let us first consider the problem of representing relative physical knowledge between two objects. We can write a single piece of knowledge like “A person is larger than a basketball” as
person >size basketball
Any propositional statement can have exceptions and counterexamples. Moreover, we need to cope with uncertainties involved in knowledge acquisition. Therefore, we assume each knowledge piece is associated with a probability distribution. More formally, given objects x and y, we define a random variable Oax,y whose range is {>, <,'} with respect to a knowledge dimension a ∈ {SIZE,WEIGHT,STRENGTH,RIGIDNESS,SPEED} so that:
P (Oax,y = r), r ∈ {>, <,'}. 1Will be publicly shared at Anonymized.URL.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
approx max width
action
theme Iagent,theme agent goal Itheme,goal
Iagent,goal
“He threw the ball”
Iagent,theme
x threw y
⇒ x is larger than y ⇒ x is heavier than y ⇒ x is slower than y
“We walked into the house”
x walked into y
⇒ x is smaller than y ⇒ x is lighter than y ⇒ x is faster than y
Iagent,goal “I squashed the bug with my boot”
squashed x with y
⇒ x is smaller than y ⇒ x is lighter than y
⇒ x is weaker than y
Itheme,goal
⇒ x is less rigid than y
⇒ x is slower than y
Figure 2: Example physical implications represented as frame relations between a pair of arguments.
This immediately provides two simple properties:
P (Ox,y = >) = P (Oy,x = <) P (Ox,x = ') = 1
2.3 Physical Implications of Verbs
Next we consider representing relative physical implications of actions applied over two objects. For example, consider an action frame “x threw y.” In general, following implications are likely to be true:
“x threw y” =⇒ x >size y “x threw y” =⇒ x >weight y
Again, in order to cope with exceptions and uncertainties, we assume a probability distribution associated with each implication. More formally, we define a random variable F av to denote the implication of the action verb v when applied over its arguments x and y with respect to a knowledge dimension a so that:
P (F sizethrew = >) := P (“x threw y”⇒ x >size y) P (Fwgtthrew = >) := P (“x threw y”⇒ x >wgt y)
where the range of F sizethrew is {>, <,'}. Intuitively, F sizethrew represents the likely first order relation implied by “throw” over ungrounded (i.e., variable) object pairs.
Above definition assumes that there is only a single implication relation for any given verb with
respect to a specific knowledge dimension. This is generally not true, since a verb, especially a common action verb, can often invoke a number of different frames according to frame semantics (Fillmore, 1976). Thus, given a number of different frame relations v1...vT associated with a verb v, we define random variables F with respect to a specific frame relation vt, i.e., F avt . We use this notation going forward.
Frame Perspective on Verb Implications: Figure 2 illustrates the frame centric view on physical implication knowledge we aim to learn. Importantly, the key insight of our work is inspired by Fillmore’s original manuscript on frame semantics (Fillmore, 1976). Fillmore has argued that “frames”—the contexts in which utterances are situated—should be considered as a third primitive of describing a language, along with a grammar and lexicon. While existing frame annotations such as FrameNet (Baker et al., 1998), PropBank (Palmer et al., 2005), and VerbNet (Schuler, 2005) provide a rich frame knowledge associated with a predicate, none of them provide the exact kind of physical implications we consider in our paper. Thus, our work can potentially contribute to these resources by investigating new approaches to automatically recover richer frame knowledge from language.
3 Data and Crowd-sourced Knowledge
Action Verbs: We pick 50 classes of Levin verbs from both “alternation classes” and “verb
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
classes” (Levin, 1993), which corresponds to about 1100 unique verbs. We sort this list by frequency of occurrence in our frame patterns in the Google Syntax Ngrams corpus (Goldberg and Orwant, 2013) and pick the top 100 verbs.
Action Frames: Figure 2 illustrates examples of action frame relations. Because we consider implications over pairwise argument relations for each frame, there are sometimes multiple frame relations we consider for a single frame. To enumerate action frame relations for each verb, we use syntactic patterns based on dependency parse by extracting the core components (subject, verb, direct object, prepositional object) of an action, then map the subject to an agent, the direct object to a theme, and the prepositional object to a goal.2 For those frames that involve an argument in a prepositional phrase, we create a separate frame for each preposition based on the statistics observed in the Google Syntax Ngram corpus.
Because the syntax ngram corpus provides only tree snippets without context, this way of enumerating potential frame patterns tend to overgenerate. Thus we refine our prepositions for each frame by taking either the intersection or union with the top 5 Google Surface Ngrams (Michel et al., 2011), depending on whether the frame was under- or over-generating. We also add an additional crowdsourcing step where we ask crowd workers to judge whether a frame pattern with a particular verb and preposition could plausibly be found in a sentence. This process results in 813 frame templates, an average of 8.13 per verb.
Object Pairs: To provide a source of ground truth relations between objects, we select the object pairs that occur in the 813 frame templates with positive pointwise mutual information (PMI) across the Google Syntax Ngram corpus. After replacing a small set of “human” nouns with a generic HUMAN object, filtering out nouns labeled as abstract by WordNet (Miller, 1995), and distilling all surface forms to their lemmas (also with WordNet), the result is 3656 object pairs.
3.1 Crowdsourcing Knowledge
We collect human judgements of the frame knowledge implications to use as a small set of seed
2Future research could use an SRL parser instead. We use dependency parse to benefit from the Google Syntax Ngram dataset that provides language statistics over an extremely large corpus, which does not exist for SRL.
knowledge (5%), a development set (45%), and a test set (50%). Crowd workers are given with a frame template such as “x threw y,” and then asked to list a few plausible objects (including people and animals) for the missing slots (e.g., x and y).3 We then ask them to rate general relationship that the arguments of the frame exhibit with respect to the knowledge dimension we choose (size, weight, etc.). For each knowledge dimension, or attribute, a, workers select an answer from (1) x >a y, (2) x <a y, (3) x 'a y, or (4) no general relation.
We conduct a similar crowdsourcing step for the set of object pairs. We ask crowd workers to compare each of the 3656 object pairs along the five knowledge dimensions we consider, selecting an answer from the same options above as with frames. We reserve 50% of the data as a test set, and split the remainder up either 5% / 45% or 20% / 30% (seed / development) to investigate the effects of different seed knowledge sizes on the model.
Statistics for the dataset are provided in Table 1. About 90% of the frames as well as object pairs had 2/3 agreement between workers. After removing frame/attribute combinations and object pairs that received less than 2/3 agreement, or were selected by at least 2/3 workers to have no relation,
3This step is to prime them for thinking about the particular template; we do not use the objects they provided.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
we end up with roughly 400–600 usable frames and 2100–2500 usable object pairs per attribute.
4 Model
We model knowledge acquisition as probabilistic inference using a factor graph. As shown in Figure 3, the graph consists of multiple substrates (page-wide boxes) corresponding to different knowledge dimensions (shown only three of them —strength, size, weight—for brevity). Each substrate consists of two types of sub-graphs: verb subgraphs and object subgraphs, which are connected through factors that quantify action–object compatibilities. Connecting across substrates are factors that model inter-dependencies across different knowledge dimensions. In what follows, we describe each graph component.
4.1 Nodes
In the factor graph, we have two types of nodes in order to capture both classes of knowledge. The object first type of nodes are object pair nodes. Each object pair node is a random variable Oax,y which captures the relative strength of attribute a between objects x and y.
The second type of nodes are frame nodes. Each frame node is a random variable F avt . This corresponds to the verb v used in a particular type of frame t, and captures the implied knowledge the frame vt holds along an attribute a.
All random variables take on the values {>, <,'}. For an object pair node Oax,y, the value represents the belief about the relation between x and y along attribute a. For a frame node F avt , the value represents the belief about the relation along attribute a between any two objects that might be used in the frame vt.
We denote the sets of all object pair and frame random variables O and F , respectively.
4.2 Action–Object Compatibility
The key aspect of our work is to reason about two types of knowledge simultaneously: relative knowledge on grounded object pairs, and implications of actions related to those objects. Thus we connect the verb subgraphs and object subgraphs through selectional preference factors ψs between two such nodes Oax,y and F a vt if we find evidence from text that suggests objects x and y are used in the frame vt. These factors encourage both random variables to agree on the same value.
As an example, consider a node Osizep,b which represents the relative size of a person and a basketball, and a node F sizethrewdobj which represents the relative size implied by an “x threw y” frame. If we find significant evidence in text that “[person] threw [basketball]” occurs, we would add a selectional preference factor to connect Osizep,b with F sizethrewdobj and encourage them towards the same value. This means that if it is discovered that people are larger than basketballs (>), then we would expect the frame “x threw y” to entail x >size y (also >).
4.3 Semantic Similarities
Some frames have relatively sparse text evidences to support their corresponding knowledge acquisition. Thus, we include several types of factors based on semantic similarities as described below.
Cross-Verb Frame Similarity We add a group of factors ψv between two verbs v and u (to connect a specific frame of v with a corresponding frame of u) based on the verb-level similarities.
Within-Verb Frame Similarity Within each verb v, which consists of a set of frame relations v1, ...vT , we also include frame-level similarity factors ψf between vi and vj . This gives us more evidence over a broader range of frames when textual evidence might be sparse.
Object Similarity As with verbs, we add factors ψo that encourage similar pairs of objects to take the same value. Given that each node represents a pair of objects, finding that x and y are similar yields two main cases in how to add factors (aside from the trivial case where the variable Oax,y is given a unary factor to encourage the value ').
1. If nodes Ox,z and Oy,z exist, we expect objects x and y to both have a similar relation to z. We add a factor that encourages Ox,z and Oy,z to take the same value. The same is true if nodes Oz,x and Oz,y exist.
2. On the other hand, if nodesOx,z andOz,y exist, we expect these two nodes to reach the opposite decision. In this case, we add a factor that encourages one node to take the value > if the other prefers the value <, and vice versa. (For the case of ', if one prefers that value, then both should.)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
vsizesquish
vsizethrow
vsizewalk
vweightthrow
vweightwalk … … … F sizethrow1 F sizethrow2 F sizethrow3 F sizethrow4
f a o v
s
f
v v
a
s
s v
a
a
frames for vsizethrow
o
o
s
h a rd
n e ss
random variable (RV)
group of RVs
factor connects RV
factors connect subset of RVs
verb similarity
frame similarity
object similarity
attribute similarity
selectional preference
f
a
o
v
s attribute subgraphs
subgraphs object
verb subgraphs
si z e
w e ig
h t
Osizes,t
Osizep,q
Osizeq,r
Osizep,s
st re
n g th
vstrengthsquish
Ostrengthp,t Ostrengthp,q
Figure 3: High level view of the factor graph model. Performance on both learning relative knowledge about objects (right), as well as entailed knowledge from verbs (center) via realized frames (left), is improved by modeling their interplay (orange). Unary seed (ψseed) and embedding (ψemb) factors are omitted for clarity.
4.4 Cross-Knowledge Correlation
Some knowledge dimensions, such as size and weight, have a significant correlation in their implied relations. For two such attributes a and b, if the same frame F avi and F b vi exists in both graphs, we add a factor ψa between them to push them towards taking the same value.
4.5 Seed Knowledge
In order to kick off learning, we provide a small set of seed knowledge among the random variables in {O,F} with seed factors ψseed. These unary seed factors push the belief for its associated random variable strongly towards the seed label.
4.6 Potential Functions
Unary Factors For all frame and object pair random variables in the training set, we train a maximum entropy classifier to predict the value of the variable. We then use the probabilities of the classifier as potentials for seed factors given to all random variables in their class (frame or object pair). Each log-linear classifier is trained separately per attribute on a featurized vector of the variable:
P (r|Xa) ∝ ewa·f(Xa)
The feature function is defined differently according to the node type:
f(Oap,q) := 〈g(p), g(q)〉 f(F avt) := 〈h(t), g(v), g(t)〉
Here g(x) is the GloVe word embedding for the word x (t is the frame relation’s preposition, and g(t) is simply set to the zero vector if there is no preposition) and h(t) is a one-hot vector of the frame relation type. We use GloVe vectors of 100 dimensions for verbs and 50 dimensions for objects and prepositions (the dimensions picked based on development set).
Binary Factors In the case of all other factors, we use a “soft 1” agreement matrix with strong signal down the diagonals: > ' <> 0.7 0.1 0.2
' 0.15 0.7 0.15 < 0.2 0.1 0.7 
4.7 Inference
After our full graph is constructed, we use belief propagation to infer the assignments of frames and object pairs not in our training data. Each message µ is a vector x with probabilities for each value
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
x ∈ {>, <,'}. A message passed from a random variable v to a neighboring factor f is the product of the messages from its other neighboring factors:
µv→f (x) ∝ ∏
f ′∈N(v)\{f}
µf ′→v(x)
A message passed from a factor f with potential ψ to a random variable v is a marginalized belief about the value of v of the other neighboring random variables combined with its potential:
µf→v(x) ∝ ∑
x′∈x′\x
ψ(x′) ∏
v′∈N(f)\{v}
µv′→f (x ′ v′)
5 Experimental Results
Factor Graph Construction We first need to pick a set of frames and objects to determine our set of random variables. The frames are simply the subset of the frames that were crowdsourced in the given configuration (e.g., seed + dev), with “soft 1” unary seed factors (the gold label indexed row of the binary factor matrix) given only to those in the seed set. The same selection criteria and seed factors are applied to the crowdsourced object pairs.
For lexical similarity factors (ψv, ψo), we pick connections based on the cosine similarity scores of GloVe vectors (Pennington et al., 2014) thresholded above a value (0.4) chosen based on development set performance. Attribute similarity factors (ψa) are chosen based on sets of frames that reach largely the same decisions on the seed data (95%). Frame similarity factors (ψf ) are added to pairs of frames with linguistically similar constructions. (Both attribute and frame similarity factors did not help on development set performance and thus were omitted from the final models; their effects are shown in the ablations in Table 3.)
Finally, selectional preference factors (ψs) are picked by using positive pointwise mutual information (PMI) between the frames and the object pairs’ occurrences in the Google Syntax Ngram corpus.
Baselines Baselines include making a RANDOM choice, picking between >, <, and '), picking the MAJORITY label, and a maximum entropy classifier based on the embedding representation (EMB-MAXENT).
5.1 Inferring Knowledge on Actions
Our first experiment is to predict knowledge implied by new frames. We experiment with two different seed knowledge sets: OUR MODEL (A) is based on more skimp seed knowledge, taking only 5% of the object pair data as seed. In contrast, OUR MODEL (B) is based on 20% of the object pairs as seed knowledge.
The full results for the baseline methods and our model are given in Table 4.7. Though the speed attribute has a skewed label distribution, giving the majority baseline high performance, our model outperforms the baselines on other attributes as well as overall.
Metaphorical Language: While our frame patterns are intended to capture action verbs, our templates also match senses of those verbs that can be used with abstract or metaphorical arguments, rather than directly physical ones. One example from the development set is “x contained y.” While x and y can be real objects, more abstract senses of “contained” could involve y as a “forest fire” or even a “revolution.” In these instances, x >size y is plausible as an abstract notion: if some entity can contain a revolution, we might think that entity as “larger” or “stronger” than the revolution.
5.2 Inferring Knowledge on Objects
Our second experiment is to predict the correct relations of new object pairs. Results for this task are also given in Table 4.7.
6 Related work
Several works straddle the gap between IE, knowledge base completion, and learning commonsense knowledge from text. Earlier works in these areas use large amounts of text to try to extract general statements like “A THING CAN BE READABLE” (Gordon et al., 2010) and frequencies of events (Gordon and Schubert, 2012). Our work focuses on specific domains of knowledge rather than general statements or occurrence statistics, and develops a frame- centric approach to circumvent reporting bias. Other work uses a knowledge base and scores unseen tuples based on similarity to existing ones (Angeli and Manning, 2013; Li et al., 2016). Relatedly, previous work uses natural language inference to infer new facts from a dataset of commonsense facts that can be ex-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Development Test Algorithm size weight stren rigid speed overall size weight stren rigid speed overall RANDOM 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33
MAJORITY 0.38 0.41 0.42 0.18 0.83 0.43 0.35 0.35 0.43 0.20 0.88 0.44 EMB-MAXENT 0.62 0.64 0.60 0.83 0.83 0.69 0.55 0.55 0.59 0.79 0.88 0.66 OUR MODEL (A) 0.71 0.63 0.61 0.82 0.83 0.71 0.55 0.55 0.55 0.79 0.89 0.65 OUR MODEL (B) 0.74 0.69 0.67 0.82 0.78 0.74 0.76 0.59 0.66 0.80 0.87 0.73
Development Test Algorithm size weight stren rigid speed overall size weight stren rigid speed overall RANDOM 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33 0.33
MAJORITY 0.50 0.54 0.51 0.50 0.53 0.51 0.51 0.55 0.52 0.49 0.50 0.51 EMB-MAXENT 0.68 0.66 0.64 0.67 0.65 0.66 0.71 0.67 0.64 0.65 0.63 0.66 OUR MODEL (A) 0.74 0.69 0.67 0.68 0.66 0.69 0.68 0.70 0.66 0.66 0.60 0.66
Table 2: Accuracy of baselines and our model on both tasks. Top: frame prediction task, OUR MODEL (A) uses 5% object pairs as seed, OUR MODEL (B) uses 20% object pairs as seed. Bottom: object pair prediction task (5% object pairs as seed).
Ablated (or added) component Accuracy – Verb similarity 0.69 + Frame similarity 0.62 – Action-object compatibility 0.62 – Object similarity 0.70 + Attribute similarity 0.62 – Frame embeddings 0.63 – Frame seeds 0.62 – Object embeddings 0.62 – Object seeds 0.62 OUR MODEL (A) 0.71
Table 3: Ablation results on size attribute for the frame prediction task on the development dataset for OUR MODEL (A) (5% of the object pairs as seed data). Our final model (OUR MODEL (B), using 20% object pairs as seed data), employs only unary embeddings and selctional preference factors.
tracted from unstructured text (Angeli and Manning, 2014). While our work also focuses on commonsense knowledge, we attempt to directly learn a small number of specific types of knowledge from text without reasoning from an existing database or dataset of facts.
A handful of works have attempted to learn the types of knowledge we address in this work. One recent work tried to directly predict several binary attributes (such “is large” and “is yellow”) from on off-the-shelf word embeddings, noting that accuracy was very low (Rubinstein et al., 2015). Another line of work addressed grounding verbs in the context of robotic tasks. One paper in this line acquires verb meanings by observing state changes in the environment (She and Chai, 2016). While they have data on 165 verb frames, they only have enough to report statistical signifi-
cance on four of them. Another work in this line does a deep investigation of eleven verbs, modeling their physical effect via annotated images along eighteen attributes (Gao et al., 2016). These works are encouraging investigations into multimodal groundings of a small set of verbs. Our work instead grounds into a fixed set of attributes but leverages language on a broader scale to learn about more verbs in more diverse set of frames.
7 Conclusion
We presented a novel take on verb-centric frame semantics to learn implied physical knowledge latent in verbs. We showed that by modeling changes in physical attributes entailed by verbs together with objects that exhibit these properties, we are able to better solve both tasks. Experiments on our novel dataset confirm that a model which takes advantage of physical relations as they arise from verbs and between objects outperforms baselines lacking such information. Our dataset and code will be made publicly available upon publication.

Review:
Thank you for the author response. It addresses some my concerns, though much of it are promises ( we will... ) -- necessarily so, given space constraints, but then, this is precisely the problem: I would like to see the revision to the paper to be able to check that the drawbacks have been fixed. The changes needed are quite substantial, and the new experimental results that they promise to include will not have undergone review if the paper is accepted at this stage. I'm still not sure that we can simply leave it to the authors to make the necessary changes without a further reviewing round. I upgrade my score to a 3 to express this ambivalence (I do like the research in the paper, but it's extremely messy in its presentation).  --------------  - Strengths:  The topic of the paper is very creative and the purpose of the research really worthwhile: the paper aims at extracting common knowledge from text, overcoming the well-known problem of reporting bias (the fact that people will not state the obvious, such as the fact that a person is usually bigger than a ball), by doing joint inference on information that is possible to extract from text.  - Weaknesses:  1) Many aspects of the approach need to be clarified (see detailed comments below). What worries me the most is that I did not understand how the approach makes knowledge about objects interact with knowledge about verbs such that it allows us to overcome reporting bias. The paper gets very quickly into highly technical details, without clearly explaining the overall approach and why it is a good idea.  2) The experiments and the discussion need to be finished. In particular, there is no discussion of the results of one of the two tasks tackled (lower half of Table 2), and there is one obvious experiment missing: Variant B of the authors' model gives much better results on the first task than Variant A, but for the second task only Variant A is tested -- and indeed it doesn't improve over the baseline.   - General Discussion:  The paper needs quite a bit of work before it is ready for publication.   - Detailed comments:  026 five dimensions, not six  Figure 1, caption:  implies physical relations : how do you know which physical relations it implies?  Figure 1 and 113-114: what you are trying to do, it looks to me, is essentially to extract lexical entailments (as defined in formal semantics; see e.g. Dowty 1991) for verbs. Could you please explicit link to that literature?  Dowty, David.  Thematic proto-roles and argument selection.  Language (1991): 547-619.  135 around here you should explain the key insight of your approach: why and how does doing joint inference over these two pieces of information help overcome reporting bias?  141  values  ==>  value ?  143 please also consider work on multimodal distributional semantics, here and or in the related work section. The following two papers are particularly related to your goals:  Bruni, Elia, et al.  Distributional semantics in technicolor.  Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 2012.  Silberer, Carina, Vittorio Ferrari, and Mirella Lapata.  Models of Semantic Representation with Visual Attributes.  ACL (1). 2013.  146 please clarify that your contribution is the specific task and approach -- commonsense knowledge extraction from language is long-standing task.  152 it is not clear what  grounded  means at this point  Section 2.1: why these dimensions, and how did you choose them?  177 explain terms  pre-condition  and  post-condition , and how they are relevant here  197-198 an example of the full distribution for an item (obtained by the model, or crowd-sourced, or  ideal ) would help.  Figure 2. I don't really see the  x is slower than y  part: it seems to me like this is related to the distinction, in formal semantics, between stage-level vs. individual-level predicates: when a person throws a ball, the ball is faster than the person (stage-level) but it's not true in general that balls are faster than people (individual-level). I guess this is related to the pre-condition vs. post-condition issue. Please spell out the type of information that you want to extract.  248  Above definition : determiner missing  Section 3   Action verbs : Which 50 classes do you pick, and you do you choose them? Are the verbs that you pick all explicitly tagged as action verbs by Levin?   306ff What are  action frames ? How do you pick them?  326 How do you know whether the frame is under- or over-generating?  Table 1: are the partitions made by frame, by verb, or how? That is, do you reuse verbs or frames across partitions? Also, proportions are given for 2 cases (2 3 and 3 3 agreement), whereas counts are only given for one case; which?  336  with... PMI : something missing (threshold?)  371 did you do this partitions randomly?  376  rate *the* general relationship   378  knowledge dimension we choose : ? (how do you choose which dimensions you will annotate for each frame?)  Section 4  What is a factor graph? Please give enough background on factor graphs for a CL audience to be able to follow your approach. What are substrates, and what is the role of factors? How is the factor graph different from a standard graph? More generally, at the beginning of section 4 you should give a higher level description of how your model works and why it is a good idea.  420  both classes of knowledge : antecedent missing.  421  object first type   445 so far you have been only talking about object pairs and verbs, and suddenly selectional preference factors pop in. They seem to be a crucial part of your model -- introduce earlier? In any case, I didn't understand their role.  461  also ?  471 where do you get verb-level similarities from?  Figure 3: I find the figure totally unintelligible. Maybe if the text was clearer it would be interpretable, but maybe you can think whether you can find a way to convey your model a bit more intuitively. Also, make sure that it is readable in black-and-white, as per ACL submission instructions.  598 define term  message  and its role in the factor graph.  621 why do you need a  soft 1  instead of a hard 1?  647ff you need to provide more details about the EMB-MAXENT classifier (how did you train it, what was the input data, how was it encoded), and also explain why it is an appropriate baseline.  654  more skimp seed knowledge : ?  659 here and in 681, problem with table reference (should be Table 2).   664ff I like the thought but I'm not sure the example is the right one: in what sense is the entity larger than the revolution? Also,  larger  is not the same as  stronger .  681 as mentioned above, you should discuss the results for the task of inferring knowledge on objects, and also include results for model (B) (incidentally, it would be better if you used the same terminology for the model in Tables 1 and 2)  778  latent in verbs : why don't you mention objects here?  781  both tasks : antecedent missing  The references should be checked for format, e.g. Grice, Sorower et al for capitalization, the verbnet reference for bibliographic details.