Original: B EYOND O BJECT R ECOGNITION : A N EW B ENCHMARK TOWARDS O BJECT C ONCEPT L EARNING
Revised: B EYOND O BJECT R ECOGNITION : A N EW B ENCHMARK TOWARDS O BJECT C ONCEPT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Understanding objects is a central building block of artiﬁcial intelligence, especially for embodied AI.
Revised: Understanding objects is a central building block of artiﬁcial intelligence, especially for embodied AI.

Original: Even though object recognition excels with deep learning, current machines still struggle to learn higher-level knowledge, e.g., what attributes does an object have, what can we do with an object.
Revised: Even though object recognition excels with deep learning, current machines still struggle to learn higher-level knowledge, e.g., what attributes does an object have, what can we do with an object.

Original: In this work, we propose a challenging Object Concept Learning (OCL) task to push the envelope of object understanding.
Revised: In this work, we propose a challenging Object Concept Learning (OCL) task to push the envelope of object understanding.

Original: It requires machines to reason out object affordances and simultaneously give the reason: what attributes make an object possesses these affordances .
Revised: It requires machines to reason out object affordances and simultaneously give the reason: what attributes make an object possesses these affordances .

Original: To support OCL, we build a densely annotated knowledge base including extensive labels for three levels of object concept: categories, attributes, and affordances, together with their causal relations.
Revised: To support OCL, we build a densely annotated knowledge base including extensive labels for three levels of object concept: categories, attributes, and affordances, and their causal relations.

Original: By analyzing the causal structure of OCL, we present astrong baseline, Object Concept Reasoning Network (OCRN).
Revised: By analyzing the causal structure of OCL, we present a baseline, Object Concept Reasoning Network (OCRN).

Original: It leverages causal intervention and concept instantiation to infer the three levels following their causal relations.
Revised: It leverages causal intervention and concept instantiation to infer the three levels following their causal relations.

Original: Inextensive experiments, OCRN effectively infers the object knowledge while follows the causalities well.
Revised: In experiments, OCRN effectively infers the object knowledge while follows the causalities well.

Original: Our data and code will be publicly available.
Revised: Data and code will be publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Object understanding is essential for intelligent machines.
Revised: Object understanding is essential for intelligent machines.

Original: Recently, beneﬁted by deep learning and large-scale datasets (Deng et al., 2009;
Revised: Recently, beneﬁted by deep learning and large-scale datasets (Deng et al., 2009;

Original: Lin et al., 2014), category recognition (Krizhevsky et al., 2012; Ren et al., 2015) has made tremendous leaps of progress.
Revised: Lin et al., 2014), category recognition (Krizhevsky et al., 2012; Ren et al., 2015) has made tremendous leaps of progress.

Original: But to close the gap between human and machine perception, machines need to pursue deeper object understanding, e.g., recognizing higher-level attributes (Isola et al., 2015) and affordances (Gibson, 2014), which may help them establish the concept of objects (Martin, 2007) when interacting with complex contexts.
Revised: But to close the gap between human and machine perception, machines need to pursue deeper object understanding, e.g., recognizing higher-level attributes (Isola et al., 2015) and affordances (Gibson, 2014), which may help them establish the concept of objects (Martin, 2007) when interacting with complex contexts.

Original: Category apple is a symbol indicating its referent (real apple).
Revised: Category apple is a symbol indicating its referent (real apple).

Original: In line with symbol grounding (Harnad, 1990), machines should learn knowledge beyond category to approach concept understanding.
Revised: In line with symbol grounding (Harnad, 1990), machines should learn knowledge beyond category to approach concept understanding.

Original: According to cognition studies (Ross, 2008; Martin, 2007), attributes depicting objects from the physical/visual side play an important role in object understanding.
Revised: According to cognition studies (Ross, 2008; Martin, 2007), attributes depicting objects from the physical/visual side play an important role in object understanding.

Original: Thus, many works (Lampert et al., 2009; Xiao et al., 2010; Farhadi et al., 2009) studied to ground objects with attributes, e.g., a hammer consists of a long handle and a heavy head.
Revised: Thus, many works (Lampert et al., 2009; Xiao et al., 2010; Farhadi et al., 2009) studied to ground objects with attributes, e.g., a hammer consists of a long handle and a heavy head.

Original: Moreover, attributes can depict object state (Isola et al., 2015).
Revised: Moreover, attributes can depict object state (Isola et al., 2015).

Original: An elegant characteristic of attributes is cross-category: objects of the same category can have various states ( big or fresh apple ), whilst various objects can have the same state ( sliced orange or carrot ).
Revised: An elegant characteristic of attributes is cross-category: objects of the same category can have various states ( big or fresh apple ), whilst various objects can have the same state ( sliced orange or carrot ).

Original: If the category is the ﬁrst level of object concept, the attribute can be seen as the second level closer to the physical fact.
Revised: If the category is the ﬁrst level of object concept, the attribute can be seen as the second level closer to the physical fact.

Original: However, recognizing attributes is still far away from concept understanding.
Revised: However, recognizing attributes is still far away from concept understanding.

Original: Given a hammer , we need to know it can be held to hit nails.
Revised: Given a hammer , we need to know it can be held to hit nails.

Original: That is, requiring machines to infer affordance (Gibson, 2014) indicating what actions humans can perform upon objects.
Revised: That is, requiring machines to infer affordance (Gibson, 2014) indicating what actions humans can perform upon objects.

Original: Here, we refer to affordance as the third level, which is closely related to common sense and causal inference (Gibson, 2014).
Revised: Here, we refer to affordance as the third level, which is closely related to common sense and causal inference (Gibson, 2014).

Original: Although affordance learning has been studied in robotics (Do et al., 2018; Hermans et al., 2011) and vision (Chuang et al., 2018; Zhu et al., 2014) communities in decades, it is still challenging.
Revised: Although affordance learning has been studied in robotics (Do et al., 2018; Hermans et al., 2011) and vision (Chuang et al., 2018; Zhu et al., 2014) communities in decades, it is still challenging.

Original: First, previous works (Nguyen et al., 2017; Fouhey et al., 2015) often focus on recognizing affordance solely.
Revised: First, previous works (Nguyen et al., 2017; Fouhey et al., 2015) often focus on recognizing affordance solely.

Original: But we usually infer affordance based on attribute observation.
Revised: But we usually infer affordance based on attribute observation.

Original: When we need to knock in a nail without a hammer at hand, we may ﬁnd other hard or heavy objects instead, e.g., a thick book.
Revised: When we need to knock in a nail without a hammer at hand, we may ﬁnd other hard or heavy objects instead, e.g., a thick book.

Original: This profoundly reveals the causality between attribute and affordance.
Revised: This profoundly reveals the causality between attribute and affordance.

Original: Second, previous works are designed for category/scale/scene-limited tasks, e.g., in Zhu et al.
Revised: Second, previous works are designed for category/scale/scene-limited tasks, e.g., in Zhu et al.

Original: (2014), 40 objects 
Revised: (2014), 40 objects and 14 affordances are included; Hermans et al. (2011) collect 375 indoor images of 6 objects,

Original: [Figure]
Revised: [Figure]

Original: and 14 affordances are included; Hermans et al. (2011) collect 375 indoor images of 6 objects,attributes, and 7 affordances; a recent dataset (Nguyen et al., 2017) contains 10 indoor objects andaffordances.
Revised: attributes, and 7 affordances; a recent dataset (Nguyen et al., 2017) contains 10 indoor objects andaffordances.

Original: Hence, they cannot support general affordance reasoning for large-scale applications.
Revised: Hence, they cannot support general affordance reasoning for large-scale applications.

Original: To reshape object learning, we believe it is essential to look at the above three levels in a uniﬁed and causal way based on an extensive knowledge base.
Revised: To reshape object learning, we believe it is essential to look at the above three levels in a uniﬁed and causal way based on an extensive knowledge base.

Original: Hence, we move a step forward to propose the object concept learning (OCL) task: given an object, machines need to infer its category, attributes, and further answer “ what can we do upon it and why ”.
Revised: Hence, we move a step forward to propose the object concept learning (OCL) task: given an object, machines need to infer its category, attributes, and further answer “ what can we do upon it and why ”.

Original: In a nutshell, machines need to reason affordance based on object appearance, category, and attributes.
Revised: In a nutshell, machines need to reason affordance based on object appearance, category, and attributes.

Original: To this end, we build a largescale and dense dataset consisting of 381 object categories, 114 attributes, and 170 affordances.
Revised: To this end, we build a largescale and dense dataset consisting of 381 object categories, 114 attributes, and 170 affordances.

Original: It contains 80,463 images of diverse scenes, 185,941 object instances in different states.
Revised: It contains 80,463 images of diverse scenes, 185,941 object instances in different states.

Original: Different from previous works (Chao et al., 2015; Hermans et al., 2011; Zhu et al., 2014), OCL offers a more nuanced angle.
Revised: Different from previous works (Chao et al., 2015; Hermans et al., 2011; Zhu et al., 2014), OCL offers a more nuanced angle.

Original: It includes: (1) category -level attribute ( A ) and affordance ( B ) labels; (2) instancelevel attribute ( α ) and affordance ( β ) labels.
Revised: It includes: (1) category -level attribute ( A ) and affordance ( B ) labels; (2) instancelevel attribute ( α ) and affordance ( β ) labels.

Original: Besides, we consider and annotate the causal relations between three levels to evaluate the reasoning ability of models and keep the follow-up methods from ﬁtting data only.
Revised: Besides, we consider and annotate the causal relations between three levels to evaluate the reasoning ability of models and keep the follow-up methods from ﬁtting data only.

Original: Accordingly, based on the causal structure of OCL, we also propose a neurocausal method, O bject C oncept R easoning N etwork ( OCRN ), as the future baseline.
Revised: Accordingly, based on the causal structure of OCL, we also propose a neurocausal method, O bject C oncept R easoning N etwork ( OCRN ), as the future baseline.

Original: It operates the instantiation from category-level to instance-level, and leverages causal intervention (Pearl et al., 2016) to alleviate the bias from categories and infer attributes and affordances.
Revised: It operates the instantiation from category-level to instance-level, and leverages causal intervention (Pearl et al., 2016) to alleviate the bias from categories and infer attributes and affordances.

Original: OCRN outperforms a host of baselines and shows impressive performance while follows the causal relations well.
Revised: OCRN outperforms a host of baselines and shows impressive performance while follows the causal relations well.

Original: In summary, our contributions are threefold: (1) Introducing the object concept learning (OCL) task poses challenges and opportunities for deeper object understanding and knowledge-based reasoning.
Revised: In summary, our contributions are threefold: (1) Introducing the object concept learning (OCL) task poses challenges and opportunities for deeper object understanding and knowledge-based reasoning.

Original: (2) Building a dataset consists of diverse objects, elaborate attributes and affordances, together with their causal relations to benchmark OCL.
Revised: (2) Building a dataset consists of diverse objects, elaborate attributes and affordances, together with their causal relations to benchmark OCL.

Original: An object concept reasoning network (OCRN) is introduced to reason three levels with concept instantiation and debiasing, which performs well on OCL.
Revised: An object concept reasoning network (OCRN) is introduced to reason three levels with concept instantiation and debiasing, which performs well.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Object Attribute.
Revised: Object Attribute.

Original: Attribute depicts the visual/physical properties like color, size, shape, etc.
Revised: Attribute depicts the visual/physical properties like color, size, shape, etc.

Original: It usually plays the role of intermedia between pixels and higher-level concepts, e.g., prompting object recognition (Farhadi et al., 2009), affordance learning (Hermans et al., 2011), zero-shot learning (Lampert et al., 2009), and object detection (Kumar Singh et al., 2018).
Revised: It usually plays the role of intermedia between pixels and higher-level concepts, e.g., prompting object recognition (Farhadi et al., 2009), affordance learning (Hermans et al., 2011), zero-shot learning (Lampert et al., 2009), and object detection (Kumar Singh et al., 2018).

Original: Recently, several largescale datasets (Farhadi et al., 2009; Xiao et al., 2010; Liu et al., 2017; Patterson & Hays, 2016; Isola et al., 2015; Krishna et al., 2016; Hudson & Manning, 2019) are released.
Revised: Recently, several largescale datasets (Farhadi et al., 2009; Xiao et al., 2010; Liu et al., 2017; Patterson & Hays, 2016; Isola et al., 2015; Krishna et al., 2016; Hudson & Manning, 2019) are released.

Original: For attribute recognition, recent works have made progress.
Revised: For attribute recognition, recent works have made progress.

Original: Beside direct multi-label classiﬁcation (Lampert et al., 2009;
Revised: Beside direct multi-label classiﬁcation (Lampert et al., 2009;

Original: Parikh & Grauman, 2011; Xiao et al., 2010; Patterson & Hays, 2016) or leveraging the correlation between attribute-attribute and attribute-object (Hwang et al., 2011; Chen & Grauman, 2014; Mahajan et al., 2011), intrinsic properties (compositionality, contextuality (Misra et al., 2017; Nagarajan & Grauman, 2018), symmetry (Li et al., 2020b)) of attribute-object are also proven useful.
Revised: Parikh & Grauman, 2011; Xiao et al., 2010; Patterson & Hays, 2016) or leveraging the correlation between attribute-attribute and attribute-object (Hwang et al., 2011; Chen & Grauman, 2014; Mahajan et al., 2011), intrinsic properties (compositionality, contextuality (Misra et al., 2017; Nagarajan & Grauman, 2018), symmetry (Li et al., 2020b)) of attribute-object are also proven useful.

Original: Object Affordance.
Revised: Object Affordance.

Original: The concept of affordance is introduced by Gibson (2014).
Revised: The concept of affordance is introduced by Gibson (2014).

Original: Affordance learning has two canonical paradigms: direct mapping (Fouhey et al., 2015) or indirect method (Zhu et al., 2014; Zhao & Zhu, 2013; Wang et al., 2017; Roy & Todorovic, 2016) with intermedia like object category, attribute, and 3D contents.
Revised: Affordance learning has two canonical paradigms: direct mapping (Fouhey et al., 2015) or indirect method (Zhu et al., 2014; Zhao & Zhu, 2013; Wang et al., 2017; Roy & Todorovic, 2016) with intermedia like object category, attribute, and 3D contents.

Original: Some studies also tried to learn affordance from human-object interactions, i.e., encoding the relationship between object and action (Gupta & Davis, 2007; Yao et al., 2013; Kato et al., 2018).
Revised: Some studies also tried to learn affordance from human-object interactions, i.e., encoding the relationship between object and action (Gupta & Davis, 2007; Yao et al., 2013; Kato et al., 2018).

Original: Visual Genome (Krishna et al., 2016) provides visual relationships between objects in the scene graph, including some human actions instead of affordances.
Revised: Visual Genome (Krishna et al., 2016) provides visual relationships between objects in the scene graph, including some human actions instead of affordances.

Original: However, these relationships cover limited and sparse affordances.
Revised: However, these relationships cover limited and sparse affordances.

Original: Differently, we use easily accessible object images as the knowledge source, and densely annotate all attributes/affordances for all object instances.
Revised: Differently, we use easily accessible object images as the knowledge source, and densely annotate all attributes/affordances for all object instances.

Original: Besides the vision community, the robot community also pays much attention to affordance (Do et al., 2018; Pinto & Gupta, 2016;
Revised: Besides the vision community, the robot community also pays much attention to affordance (Do et al., 2018; Pinto & Gupta, 2016;

Original: Thermos et al., 2017; Pinto et al., 2016), especially for object grasping and manipulation.
Revised: Thermos et al., 2017; Pinto et al., 2016), especially for [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: object grasping and manipulation.

Original: Recently, several datasets (Nguyen et al., 2017; Zhu et al., 2014;
Revised: Recently, several datasets (Nguyen et al., 2017; Zhu et al., 2014;

Original: Chao et al., 2015; Chuang et al., 2018) have been proposed.
Revised: Chao et al., 2015; Chuang et al., 2018) have been proposed.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2014) built a knowledge base containing attribute, affordance, human pose, and human-object spatial conﬁguration.
Revised: (2014) built a knowledge base containing attribute, affordance, human pose, and human-object spatial conﬁguration.

Original: But labeling pose and human-object relative location are costly.
Revised: But labeling pose and human-object relative location are costly.

Original: Chao et al.
Revised: Chao et al.

Original: (2015) proposed a semantic category-level affordance dataset including 91 COCO objects (Lin et al., 2014) and 957 affordances.
Revised: (2015) proposed a semantic category-level affordance dataset including 91 COCO objects (Lin et al., 2014) and 957 affordances.

Original: Causal Inference.
Revised: Causal Inference.

Original: There is increasing literature on exploiting causal inference (Pearl et al., 2016) in machine learning, especially with causal graphical models (Spirtes et al., 2000; Pearl et al., 2016), including feature selection (Guyon et al., 2007) and learning (Chalupka et al., 2014), video analysis (Pickup et al., 2014; Lebeda et al., 2015), reinforcement learning (Nair et al., 2019; Dasgupta et al., 2019), etc.
Revised: There is increasing literature on exploiting causal inference (Pearl et al., 2016) in machine learning, especially with causal graphical models (Spirtes et al., 2000; Pearl et al., 2016), including feature selection (Guyon et al., 2007) and learning (Chalupka et al., 2014), video analysis (Pickup et al., 2014; Lebeda et al., 2015), reinforcement learning (Nair et al., 2019; Dasgupta et al., 2019), etc.

Original: Recently, Wang et al.
Revised: Recently, Wang et al.

Original: (2020) studied the causal relation between objects in images and used intervention (Pearl et al., 2016) to alleviate the observation bias in detection.
Revised: (2020a) studied the causal relation between objects in images and used intervention (Pearl et al., 2016) to alleviate the observation bias in detection.

Original: Atzmon et al.
Revised: Atzmon et al.

Original: (2020) analyze the causal generative model of compositional zero-shot learning and disentangle the representations of attributes and objects.
Revised: (2020) analyze the causal generative model of compositional zero-shot learning and disentangle the representations of attributes and objects.

Original: In this work, we explore the causal relations between three object levels and apply backdoor adjustment (Pearl et al., 2016) to alleviate the existing bias.
Revised: In this work, we explore the causal relations between three object levels and apply backdoor adjustment (Pearl et al., 2016) to alleviate the existing bias.

Original:  K NOWLEDGE B ASE C ONSTRUCTION
Revised:  K NOWLEDGE B ASE C ONSTRUCTION

Original: Our goal is to construct a dataset that can characterize abundant knowledge in object category, attribute, affordance, and causal relations.
Revised: Our goal is to construct a dataset that can characterize abundant knowledge in object category, attribute, affordance, and causal relations.

Original: The construction process is given as follows (Fig. 2).
Revised: The construction process is given as follows (Fig. 2).

Original: Data Collection.
Revised: Data Collection.

Original: (1) Affordance : We collect 170 affordances out of 1,006 candidates from widelyused action/affordance datasets (Chao et al., 2015; Gu et al., 2018; Chao et al., 2018; Gupta &
Revised: (1) Affordance : We collect 170 affordances out of 1,006 candidates from widelyused action/affordance datasets (Chao et al., 2015; Gu et al., 2018; Chao et al., 2018; Gupta &

Original: Malik, 2015; Zhu et al., 2014; Nguyen et al., 2017) in view of generality and commonness.
Revised: Malik, 2015; Zhu et al., 2014; Nguyen et al., 2017) in view of generality and commonness.

Original: Category :
Revised: Category :

Original: Considering the taxonomy (WordNet (Fellbaum, 2012)) and diversity, we collectobjects out of 1,742 candidates from object datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016;
Revised: Considering the taxonomy (WordNet (Fellbaum, 2012)) and diversity, we collectobjects out of 1,742 candidates from object datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016;

Original: Liu et al., 2017; Lin et al., 2014).
Revised: Liu et al., 2017; Lin et al., 2014).

Original: (3) Attribute : When viewed through the lens of the causal relation between attribute-affordance, we manually ﬁlter 500 most frequent attributes from large-scale attribute datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016;
Revised: (3) Attribute : When viewed through the lens of the causal relation between attribute-affordance, we manually ﬁlter 500 most frequent attributes from large-scale attribute datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016;

Original: Liu et al., 2017; Krishna et al., 2016) and choose 114 attributes, covering colors, deformations, supercategories, various surface, geometrical, and physical properties.
Revised: Liu et al., 2017; Krishna et al., 2016) and choose 114 attributes, covering colors, deformations, supercategories, various surface, geometrical, and physical properties.

Original: (4) Image : We extract 75, images from object datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016; Liu et al., 2017; Krishna et al., 2016), together with their Ground Truth (GT) boxes.
Revised: (4) Image : We extract 75, images from object datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016; Liu et al., 2017; Krishna et al., 2016), together with their Ground Truth (GT) boxes.

Original: To ensure diversity, we further manually collected 4,885 Internet images of selected categories.
Revised: To ensure diversity, we further manually collected 4,885 Internet images of selected categories.

Original: Then, we annotate the missing box and category labels for all instances via crowdsourcing.
Revised: Then, we annotate the missing box and category labels for all instances via crowdsourcing.

Original: Finally, 185,941 instances of 381 categories from 80,463 images are collected, i.e., average 488 instances per category and 2. boxes per image.
Revised: Finally, 185,941 instances of 381 categories from 80,463 images are collected, i.e., average 488 instances per category and 2.31 boxes per image.

Original: More details and class lists are in Appendix Sec.
Revised: More details and class lists are in Appendix Sec.

Original: A.7 .
Revised: A.11 .

Original: OCL is long-tail distributed, where the head categories have over 5,000 instances each, but the rarest categories have onlyinstances, which challenges the robustness of vision systems.
Revised: OCL is long-tail distributed, where the head categories have over 5,000 instances each, but the rarest categories have only 9 instances, which challenges the robustness of vision systems.

Original: Attribute Annotation.
Revised: Attribute Annotation.

Original: We offer annotations in two levels of granularity: (1) Category-level attribute ( A ) contains common sense.
Revised: We offer annotations in two levels: (1) Category-level attribute ( A ) contains common sense.

Original: For each category , we annotate its most common attributes.
Revised: For each category , we annotate its most common attributes.

Original: In concept learning, the usage of category-level annotation as commonobject knowledge can date back to Osherson et al.
Revised: In concept learning, the usage of category-level annotation as common knowledge can date back to Osherson et al.

Original: Following Osherson et al.
Revised: Following Osherson et al.

Original: (1991), to avoid bias, annotators are given category-attribute pairs (category names , not images).
Revised: (1991), to avoid bias, annotators are given category-attribute [Figure]

Original: They propose a 0-3 score according to the category concept in their minds (0: No, 1: Normally No, 2: Normally Yes, 3: Yes). Each pair is annotated by three annotators and takes the plurality as A label.
Revised: Dataset # Image # Instance # Object # Attribute # Affordance APY (Farhadi et al., 2009) 15, 15,   / SUN (Xiao et al., 2010) 14, 14,   / COCO-a (Patterson & Hays, 2016) 84, 188,   / ImageNet150k (Liu et al., 2017) 150, 150, 1,  / Chao et al.

Original: If the range of 3 proposals exceeds 1, another three annotators will re-annotate this pair until achieving consensus.
Revised: (2015) / / / 957 ( B ) Hermans et al. (2011) -  7 Zhu et al.

Original: We binarize the annotations (0: No, 1: Yes) with a threshold of 2 and get a category-level attribute matrix M A ( [381 , 114] ).
Revised: [Figure] pairs (category names , not images) and multi-annotator vote to build the binary category-level attribute matrix M A .

Original: (2) Instance-level attribute ( α ) is the individual attributes of each instance .
Revised: (2) Instance-level attribute ( α ) is the individual attributes of each instance .

Original: The annotation unit is an attribute-instance pair .
Revised: The annotation unit is an attribute-instance pair .

Original: Two annotators label each pair with 0 (No) and 1 (Yes). If they give different labels, this pair will be handed over to another two annotators until meeting consensus.
Revised: Each pair is labeled by multi-annotator.

Original: Affordance Annotation.
Revised: Affordance Annotation.

Original: There are also two levels of granularity: (1) Category-level affordance
Revised: There are also two levels of granularity: (1) Category-level affordance

Original: B , similar to A , is annotated in category-affordance pairs , indicating the common affordances of 
Revised: B , similar to A , is annotated in category-affordance pairs , indicating the common affordances of each category.

Original: [Figure]

Original: a. Data collection b. Category-level annotation c. Instance-level annotation

Original: [Figure]

Original: d. Causality annotation

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: each category.

Original: Following Chao et al.
Revised: Following Chao et al.

Original: , the annotators are given category-affordance pairs.
Revised: , the annotators label category-level affordance matrix

Original: The pairs are annotated in four bins (0-3) and normalized (same as A ) to describe the possibility of an affordance in a category. Each pair is annotated by three annotators and makes consensus the same as A . The 0-3 scores are binarized (1: Yes, 0: No) with a threshold of 2. The ﬁnal categorylevel affordance matrix M B is [381 , 170] .
Revised: M B ( [381 , 170] ) similar to A annotation.

Original: (2) Instance-level affordance β is annotated for every instance with the help of object states (Isola et al., 2015).
Revised: (2) Instance-level affordance β is annotated for every instance with the help of object states (Isola et al., 2015).

Original: As B is determined by common states, objects in speciﬁc states may have different affordances from B , e.g., we cannot board a flying plane.
Revised: As B is determined by common states, objects in speciﬁc states may have different affordances from B , e.g., we cannot board a flying plane.

Original: Moreover, if a service robot ﬁnds a broken cup , it may infer that the cup can still hold water as it is trained with B labels.
Revised: Moreover, if a service robot ﬁnds a broken cup , it may infer that the cup can still hold water as it is trained with B labels.

Original: Thus, we need detailed β beyond B .
Revised: Thus, we need detailed β beyond B .

Original: As the instances in the same state should have similar β (all rotten apple s cannot be eaten ),
Revised: As the instances in the same state should have similar β (all rotten apple s cannot be eaten ),

Original: six experts
Revised: six experts

Original: ﬁrst conclude the states.
Revised: ﬁrst conclude the states.

Original: The experts scan all instances of each category and use their knowledge of affordance to deﬁne all the existing states.

Original: Then all 186 K instances are dispatched to the concluded states via crowdsourcing.

Original: If some instances do not belong to any predeﬁned states, they will be returned to the experts to add more states.

Original: In total, 1,376 states are deﬁned, and each category has 3.6 states on average.
Revised: In total, 1,376 states are deﬁned, and each category has 3.6 states on average.

Original: Next, β is annotated for each state. Given a state-affordance pair and example images, two annotators mark it with 0 (No) and 1 (Yes).
Revised: Next, β is annotated for each state and the instances are assigned with beta according to their states.

Original: The results are combined in the same way as α . Thus, each instance would have a state and the corresponding β . An annotator would recheck each instance together with its state and β labels to ensure the quality. If its state is inaccurate or the state β labels are unsuitable, this annotator would correct them.
Revised: More details please refer to Appendix Sec.

Original:  It is worth noting that, as states seem to be a beneﬁcial latent variable for OCL, in practice, we do not use it in models because the states we use are a subset of whole state space, and there can be unseen states in real-world data.
Revised: A.2 It is worth noting that, as states seem to be a beneﬁcial latent variable for OCL, in practice, we do not use it in models because the states we use are a subset of whole state space, and there can be unseen states in real-world data.

Original: Causal Relation Annotation.
Revised: Causal Relation Annotation.

Original: We annotate instance -level (considering the context of each instance) causality to answer which attribute(s) are the critical and direct causes of a certain affordance?
Revised: We annotate instance -level (considering the context of each instance) causality to answer which attribute(s) are the critical and direct causes of a certain affordance?

Original: Filtering to avoid the combinatorial explosion.
Revised: Filtering to avoid the combinatorial explosion.

Original: Initially, we need to make binary decisions on all instance- α - β triplets, which is far beyond handleable.
Revised: Initially, we need to make binary decisions on all instance- α - β triplets, which is far beyond handleable.

Original: Fortunately, we ﬁnd that most α - β classes (e.g., shiny and kick ) are meaningless and always of no causality on any instances.
Revised: Fortunately, we ﬁnd that most α - β classes (e.g., shiny and kick ) are meaningless and always of no causality on any instances.

Original: Thus, we can ﬁrstly exclude the most impossible pairs, meanwhile guarantee the completeness of causality.
Revised: Thus, we can ﬁrstly exclude the most impossible pairs, meanwhile, guarantee the completeness of causality.

Original: Starting from the [114,170] matrix of α - β classes, we ask three experts to vote the causal relation of each class.

Original: They scan all instances to answer whether the relation exists in any cases.

Original: Finally, we obtain about 10% α - β classes as candidates.
Revised: Finally, we obtain about 10% α - β classes as candidates.

Original: (2) Instance-level causality : we also adopt object states as a reference.
Revised: (2) Instance-level causality : we also adopt object states as a reference.

Original: For each state- α - β triplet, two annotators are asked whether the speciﬁc attribute is the direct cause of this affordance in this state and gives their binary answer.
Revised: Multiple annotators have been involved for each state- α - β triplet and are asked whether the speciﬁc attribute is the direct cause of this affordance in this state.

Original: We use the same method in annotating β to combine results and assign state-level labels to instances. Next, for all instances of a [Figure] state, an expert decides whether the state-level relations are reasonable for each instance in speciﬁc contexts and correct the inaccurate ones.
Revised: The answers are combined and checked for all instances of a state.

Original: Finally, we obtain about 2 M instance- α - β triplets of causal relations.
Revised: Finally, we obtain about 2 M instance- α - β triplets of causal relations.

Original: As we have labelled all α and β for all instances, the causal relations would be in four situations: [0,0], [1,1]; [0,1], [1,0].
Revised: As we have labelled all α and β for all instances, the causal relations would be in four situations: [0,0], [1,1]; [0,1], [1,0].

Original: The former two are “positive”, e.g., fresh (1/0) → eat (1/0) for an apple .
Revised: The former two are “positive”, e.g., fresh (1/0) → eat (1/0) for an apple .

Original: While the last two are “negative”, e.g.,  for a car .[Figure] [Table]
Revised: While the last two are “negative”, e.g., broken (1/0) → drive (0/1) for a car .

Original: OCL provides the category- and instance- level attributes ( A , α ), affordances ( B , β ).

Original: Discussion.
Revised: Discussion.

Original: Fig.
Revised: Fig.

Original: 4 shows some examples of OCL.
Revised: 3 shows some examples of OCL.

Original: These clear causalities are not thoroughly studied in previous datasets (Zhu et al., 2014;
Revised: These clear causalities are not thoroughly studied in previous datasets (Zhu et al., 2014;

Original: Nguyen et al., 2017; Fouhey et al., 2015; Hermans et al., 2011).
Revised: Nguyen et al., 2017; Fouhey et al., 2015; Hermans et al., 2011).

Original: We compare OCL with previous dense datasets in Tab. 1.
Revised: We compare OCL with previous dense datasets in Tab. 1.

Original: More analysis ﬁgures and tables are in Appendix Sec.
Revised: More analysis ﬁgures and tables are in Appendix Sec.

Original: A.2 .
Revised: A.3 .

Original:  O BJECT C ONCEPT L EARNING 4. T ASK O VERVIEW
Revised:  O BJECT C ONCEPT L EARNING 4. T ASK O VERVIEW

Original: Given an instance I (image region in box b o ), OCL aims to infer attribute α and affordance β while following the causalities.
Revised: Given an instance I (image region in box b o ), OCL aims to infer attribute α and affordance β while following the causalities.

Original: Formally, OCL can be described as:
Revised: Formally, OCL can be described as:

Original: [Equation]
Revised: [Equation]

Original: where S α , S β are the scores of α, β , given an object instance I and its predicted category probability P ( O | I ) .
Revised: where P α , P β are the probabilities of α, β , given an object instance I and its predicted category probability P ( O | I ) .

Original: The α, β probabilities P α = Sigmoid ( S α ) , P β = Sigmoid ( S β ) .
Revised: ∆ P β means the Total Direct Effect ( TDE [ · ] ) (Pearl et al., 2016) of affordance prediction change after we operate upon a model F ( · ) .

Original: And the → indicates the estimated causal → effect relations.
Revised: ∆ P β is expected to follow the GT causal relation between attribute-affordance.

Original: In Sec. 5, we detail theevaluation of reasoning via causal inference tools (Pearl et al., 2016).
Revised: We will detail the reasoning evaluation in Sec. 5.

Original: We spilt images into train, validation (val) and test sets with 56,916:14,446:9,101 images.
Revised: We spilt images into train, validation (val) and test sets with 56,916:14,446:9,101 images.

Original: The val and test sets cover 221 of the 381 categories, and the train set covers all categories.
Revised: The val and test sets cover 221 of the 381 categories, and the train set covers all categories.

Original: OCL is a long-tailed recognition task and requires generalization to cover the whole object category-attribute-affordance space with imbalanced information.
Revised: OCL is a long-tailed recognition task and requires generalization to cover the whole object category-attributeaffordance space with imbalanced information.

Original: Thus, it is challenging for current vision systems without the reasoning ability to understand the causalities.
Revised: Thus, it is challenging for current vision systems without the reasoning ability to understand the causalities.

Original: 4. O BJECT C ONCEPT R EASONING N ETWORK
Revised: 4. O BJECT C ONCEPT R EASONING N ETWORK

Original: Here, we ﬁrst analyze the causal structure of OCL and then propose a basic system, Object Concept
Revised: Here, we ﬁrst analyze the causal structure of OCL and then propose a basic system, Object Concept

Original: Reasoning Network (OCRN), as the future baseline for the OCL task.
Revised: Reasoning Network (OCRN), as the future baseline for the OCL task.

Original: [Figure]
Revised: [Figure]

Original: OCL Causal Graph.
Revised: √√ [Figure] ance. I, α, β are the instantiations of O

Original:  We use the causal graph to shed light on the subtle causalities of OCL in
Revised: A Causal Graph. We use the causal graph to shed light on the subtle causalities of OCL in

Original: Fig. 
Revised: Fig. 4.

Original: Causal graph (Pearl et al., 2016) indicates the underlying causalities between variables O, A, B ; I, α, β .
Revised: Causal graph (Pearl et al., 2016) indicates the underlying causalities between variables O, A, B ; I, α, β .

Original: Directed edges represent the causal directions.
Revised: Directed edges represent the causal directions.

Original: According to the prior knowledge about the causalities between three levels, a hierarchical structure is depicted:
Revised: According to the prior knowledge about the causalities between three levels, a hierarchical structure is depicted:

Original: (a) the inner triangle with dotted lines is the category -level: object category O decides the category-level attributes A and affordances B ; (b) the outer triangle is the instance -level: instance visual appearance I , instance-level attributes α and affordances β .
Revised: (a) the inner triangle with dotted lines is the category -level: object category O decides the category-level attributes A and affordances B ; (b) the outer triangle is the instance -level: instance visual appearance I , instance-level attributes α and affordances β .

Original: As I indicates the physical noumenon , it is the source of semantic and functional properties and decides α, β .
Revised: As I indicates the physical noumenon , it is the source of semantic and functional properties and decides α, β .

Original: As attributes can affect affordances, in both two levels, there are edges from attributes to affordances.
Revised: As attributes can affect affordances, in both two levels, there are edges from attributes to affordances.

Original: From another perspective, I, α, β are the instantiations of O, A, B respectively.
Revised: From another perspective, I, α, β are the instantiations of O, A, B respectively.

Original: We can use a whole node O (cid:48) to represent O, A, B (Fig. 6).
Revised: We can use a whole node O (cid:48) to represent O, A, B (Fig. 5)

Original: Object Category Bias.
Revised: Object Category Bias.

Original: Formally, OCL is depicted as P ( α | I ) and P ( β | I, α ) .
Revised: Formally, OCL is depicted as P ( α | I ) and P ( β | I, α ) .

Original: Following the causal graph, we represent nodes { I, A, B, α, β } with { f I , f A , f B , f α , f β } respectively in latent space.
Revised: Following the causal graph, we represent nodes { I, A, B, α, β } as { f I , f A , f B , f α , f β } respectively in latent space.

Original: f I is the RoI pooling feature of b o extracted by
Revised: f I is the RoI pooling feature of b o extracted by

Original: a COCO pre-trained ResNet-50 (He et al., 2016).
Revised: a COCO pre-trained ResNet-50 (He et al., 2016).

Original: As the samples of different object categories are usually imbalanced, conventional methods may suffer from severe category bias (Wang et al., 2020).
Revised: As the samples of different object categories are usually imbalanced, conventional methods may suffer from severe category bias (Wang et al., 2020a).

Original: For example, in OCL, animal accounts for 22% instances and home appliance only accounts for 3%.
Revised: For example, in OCL, animal accounts for 22% instances and home appliance only accounts for 3%.

Original: In P ( α | I ) , category bias is imported following
Revised: In P ( α | I ) , category bias is imported following

Original: [Equation]
Revised: [Equation]

Original: where P ( O i | I ) is the predicted category probability.
Revised: where P ( O i | I ) is the predicted category probability.

Original: That is, O is a confounder (Pearl et al., 2016) and pollutes attribute inference, especially for the rare categories (analyzed in Fig.
Revised: That is, O is a confounder (Pearl et al., 2016) and pollutes attribute inference, especially for the rare categories (analyzed in Fig.

Original: Causal Intervention.
Revised: Causal Intervention.

Original: To tackle this problem, we propose OCRN utilizing intervention (Pearl et al., 2016) to deconfound the confounder O for α (Fig. 6).
Revised: To tackle this problem, we propose OCRN utilizing intervention (Pearl et al., 2016) to deconfound the confounder O for α (Fig. 5).

Original: That said, in α estimation, we use do ( · ) operation (Pearl et al., 2016) to eliminate the edge from O to I , i.e., P ( α | do ( I )) is m 
Revised: That said, in α estimation, we use do ( · ) operation (Pearl et al., 2016) to eliminate the edge from O to I , i.e., P ( α | do ( I )) is m (cid:88) i

Original: [Equation]
Revised: [Equation]

Revised: Expectation w.r.t

Revised: P 𝑂𝑂 𝑖𝑖 𝑓𝑓 𝐼𝐼 𝑓𝑓 𝐴𝐴 𝑖𝑖 𝑓𝑓 𝛼𝛼 ̅𝑓𝑓 𝑂𝑂 𝑖𝑖 𝑓𝑓 𝐼𝐼 𝑓𝑓 𝐵𝐵 𝑖𝑖 𝑓𝑓 𝛽𝛽

Revised: Expectation w.r.t

Revised: P 𝑂𝑂 𝑖𝑖 ̅𝑓𝑓 𝑂𝑂 𝑖𝑖 ℱ 𝐵𝐵 𝑓𝑓 𝛼𝛼′ 𝑓𝑓 𝛼𝛼 = � 𝑖𝑖

Revised: 𝑚𝑚 ℱ 𝛼𝛼 𝑓𝑓 𝐼𝐼 , 𝑓𝑓 𝐴𝐴 𝑖𝑖 𝑃𝑃 𝑂𝑂 𝑖𝑖 𝑓𝑓 𝛽𝛽 =

Revised: � 𝑖𝑖

Revised: 𝑚𝑚 ℱ 𝛽𝛽 𝑓𝑓 𝐼𝐼 , 𝑓𝑓 𝛼𝛼 , 𝑓𝑓 𝐵𝐵 𝑖𝑖 𝑃𝑃 𝑂𝑂 𝑖𝑖 𝐼𝐼 𝛼𝛼 𝛽𝛽 𝑂𝑂𝑂 𝐼𝐼 𝛼𝛼 𝛽𝛽 𝑂𝑂𝑂 𝑔𝑔 𝐵𝐵 𝐴𝐴 𝑖𝑖 𝐵𝐵 𝑖𝑖 𝛽𝛽 𝛼𝛼 𝑔𝑔 𝛽𝛽 𝑔𝑔 𝐴𝐴 𝑔𝑔 𝛼𝛼 ℱ 𝐴𝐴 𝑓𝑓 𝐴𝐴 𝑖𝑖 ) 𝑃𝑃 ( 𝛽𝛽 | 𝑑𝑑𝑑𝑑 ( 𝐼𝐼 , 𝛼𝛼 ) )

Revised: 𝑃𝑃 ( 𝛼𝛼 | 𝑑𝑑𝑑𝑑 ( 𝐼𝐼 ) 𝐼𝐼 𝛼𝛼 𝛽𝛽 𝑂𝑂𝑂 𝑂𝑂 𝐴𝐴 𝐵𝐵 𝑂𝑂𝑂 = 𝑃𝑃 ( 𝛽𝛽 | 𝐼𝐼 , 𝛼𝛼 ) )

Revised: 𝑃𝑃 ( 𝛼𝛼 | 𝐼𝐼

Revised: Intervention 𝑓𝑓 𝛼𝛼⋯ 𝑓𝑓 𝛼𝛼𝑓𝑓 𝛼𝛼𝑓𝑓 𝛼𝛼 𝑝𝑝

Revised: Aggregation ℱ 𝛽𝛽 ℱ 𝛼𝛼 ℱ 𝛼𝛼 and ℱ 𝛽𝛽 :

Revised: s o ft m a x

Revised: s o ft m a x 𝑉𝑉𝑂𝐾𝐾𝑂 𝑄𝑄𝑂 𝑄𝑄𝐾𝐾 𝑉𝑉 ⨂ self-attention self-attention

Revised: ⨂ FC ⨂

Revised: ⨂

Original:  where m = 381 .
Revised: FC concat ⋯ where m = 381 .

Revised: A j is the category-attribute vector of j th category.

Original: As A is decided by O , P ( A j | O i ) =
Revised: As A is decided by O , P ( A j | O i ) =

Original: 1 iif i =
Revised: 1 iif i =

Original: j and P ( A j | O i )
Revised: j and P ( A j | O i )

Original: = 

Revised: =

Original: if i (cid:54) =
Revised: if i (cid:54) =

Revised: j , where O i is the i th category and A j is the category-attribute of j th category.

Original: j . P ( O i ) is the prior probability of the i -th category (frequency in the OCL train set).
Revised: P ( O i ) is the prior probability of the i -th category (frequency in the OCL train set).

Original: Attribute Estimation Debiasing.
Revised: 4.2. A TTRIBUTE E STIMATION

Original: To implement Eq. 3, we use mean object category feature to represent category-level attribute A .
Revised: To implement Eq. 3, we use mean object category feature to represent category-level attribute A .

Original: We take the mean feature ( ¯ f O i ) of all instances in one category as the representation of the category ( O i ), which is then mapped to attribute feature space:
Revised: We take the mean feature ( ¯ f O i ) of all instances in one category as the representation of the category ( O i ). It is then mapped to attribute feature space:

Original: f A i =
Revised: f A i =

Original: F A ( ¯ f O i ) , where F A ( · ) is a fully-connected layer (FC) (Fig. 6).
Revised: F A ( ¯ f O i ) , where F A ( · ) is a fully-connected layer (FC) (Fig. 5).

Original: The representation set of A is f A
Revised: The representation set of A is f A

Original: = { f A i } mi =1 .
Revised: = { f A i } mi =1 .

Original: For each category, we use f I of its all instances (train set) to calculate ¯ f o i in training and inference.
Revised: For each category, we use f I of its all instances (train set) to calculate ¯ f O i in training and inference.

Original: Attribute Instantiation.
Revised: Attribute Instantiation.

Original: Next, we obtain α representation according to Eq. 3:
Revised: Next, we obtain α representation according to Eq. 3:

Original: [Equation]
Revised: [Equation]

Original: where P O i is the prior category probability ( P ( O i ) in Eq. 3).
Revised: where P O i is the prior category probability ( P ( O i ) in Eq. 3).

Original: Eq
Revised: Eq.

Original: [Equation]

Original: 4 indicates the attribute instantiation from A to α with I as the condition .
Revised: 4 indicates the attribute instantiation from A to α with I as the condition .

Original: Hence, we can equally translate the α estimation problem into a conditioned instantiation problem .
Revised: Hence, we can equally translate the α estimation problem into a conditioned instantiation problem .

Original: F α ( · ) is implemented with multi-head attention mechanism (Vaswani et al., 2017), as shown in Fig.
Revised: F α ( · ) is implemented with multi-head attention mechanism (Vaswani et al., 2017), as shown in Fig.

Original: Both f A
Revised: Both f A

Original: i and f I are fed to three independent linear transformations to obtain key, query, and value vectors K, Q, V and K (cid:48) , Q (cid:48) , V (cid:48) .
Revised: i and f I are fed to three independent linear transformations to obtain key, query, and value vectors K, Q, V and K (cid:48) , Q (cid:48) , V (cid:48) .

Original: The attention output is softmax ([ Q, Q (cid:48) ]
Revised: The attention output is softmax ([ Q, Q (cid:48) ]

Original: T [ K, K (cid:48) ] / √ d )[ V, V (cid:48) ] T where d is the feature dimension.
Revised: T [ K, K (cid:48) ] / √ d )[ V, V (cid:48) ] T where d is the feature dimension.

Original: The output is compressed by a linear layer to the instantiated representation F α ( f I , f A i ) .
Revised: The output is compressed by a linear layer to the instantiated representation F α ( f I , f A i ) .

Original: The debiased representation f α is the expectation of F α ( f I , f A i ) w.r.t P O i according to back-door adjustment (Eq. 3).
Revised: The debiased representation f α is the expectation of F α ( f I , f A i ) w.r.t P O i according to back-door adjustment (Eq. 3).

Original: Affordance Estimation Debiasing.
Revised: 4.2. A FFORDANCE E STIMATION

Original: Similar to α , in β estimation, category bias also exists:
Revised: Similar to α , in β estimation, category bias also exists:

Original: [Equation]
Revised: [Equation]

Original: With Eq.
Revised: With Eq.

Original: 3, α can be seen as “enforced” and deconfounded, as it is beforehand estimated.
Revised: 3, α can be seen as “enforced” and deconfounded, as it is beforehand estimated.

Original: For I , we again use intervention (Pearl et al., 2016):
Revised: For I , we again use intervention (Pearl et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: Affordance Instantiation.
Revised: Affordance Instantiation.

Original: Similar to Eq.
Revised: Similar to Eq.

Original: 3, P ( B j | O i )
Revised: 3, P ( B j | O i )

Original: = 1 iif i =
Revised: = 1 iif i =

Original: j , P ( B j |
Revised: j , P ( B j |

Original: O i )
Revised: O i )

Original: = 

Revised: =

Original: if i (cid:54) =
Revised: if i (cid:54) =

Original: j , we omit the process for simplicity.
Revised: j , we omit the process for simplicity.

Original: An FC is used to obtain f B and Eq.
Revised: An FC is used to obtain f B and Eq.

Original: 6 is implemented as:
Revised: 6 is implemented as:

Original: [Equation]
Revised: [Equation]

Original: Expectation w.r.t P 𝑂𝑂 𝑖𝑖 𝑓𝑓 𝐼𝐼 𝑓𝑓 𝐴𝐴 𝑖𝑖 𝑓𝑓 𝛼𝛼 ̅𝑓𝑓 𝑂𝑂 𝑖𝑖 𝑓𝑓 𝐼𝐼 𝑓𝑓 𝐵𝐵 𝑖𝑖 𝑓𝑓 𝛽𝛽 Expectation w.r.t P 𝑂𝑂 𝑖𝑖 ̅𝑓𝑓 𝑂𝑂 𝑖𝑖 ℱ 𝐵𝐵 𝑓𝑓 𝛼𝛼′ 𝑓𝑓 𝛼𝛼 = � 𝑖𝑖
Revised: As shown in Fig.

Original: 𝑚𝑚 ℱ 𝛼𝛼 𝑓𝑓 𝐼𝐼 , 𝑓𝑓 𝐴𝐴 𝑖𝑖 𝑃𝑃 𝑂𝑂 𝑖𝑖 𝑓𝑓 𝛽𝛽 = � 𝑖𝑖 𝑚𝑚 ℱ 𝛽𝛽 𝑓𝑓 𝐼𝐼 , 𝑓𝑓 𝛼𝛼 , 𝑓𝑓 𝐵𝐵 𝑖𝑖 𝑃𝑃 𝑂𝑂 𝑖𝑖 𝐼𝐼 𝛼𝛼 𝛽𝛽 𝑂𝑂𝑂 𝐼𝐼 𝛼𝛼 𝛽𝛽 𝑂𝑂𝑂 𝑔𝑔 𝐵𝐵 𝐴𝐴 𝑖𝑖 𝐵𝐵 𝑖𝑖 𝛽𝛽 𝛼𝛼 𝑔𝑔 𝛽𝛽 𝑔𝑔 𝐴𝐴 𝑔𝑔 𝛼𝛼 ℱ 𝐴𝐴 𝑓𝑓 𝐴𝐴 𝑖𝑖 ) 𝑃𝑃 ( 𝛽𝛽 | 𝑑𝑑𝑑𝑑 ( 𝐼𝐼 , 𝛼𝛼 ) ) 𝑃𝑃 ( 𝛼𝛼 | 𝑑𝑑𝑑𝑑 ( 𝐼𝐼 ) 𝐼𝐼 𝛼𝛼 𝛽𝛽 𝑂𝑂𝑂 𝑂𝑂 𝐴𝐴 𝐵𝐵 𝑂𝑂𝑂 = 𝑃𝑃 ( 𝛽𝛽 | 𝐼𝐼 , 𝛼𝛼 ) )
Revised: 5, for causal inference operation (Sec. 5), f α is ﬁrst separated to attribute categorywise features f α p by multiple independent FCs, where f α p is the feature of p th attribute.

Original: 𝑃𝑃 ( 𝛼𝛼 | 𝐼𝐼 Intervention 𝑓𝑓 𝛼𝛼⋯ 𝑓𝑓 𝛼𝛼𝑓𝑓 𝛼𝛼𝑓𝑓 𝛼𝛼 𝑝𝑝 Aggregation ℱ 𝛽𝛽 ℱ 𝛼𝛼 ℱ 𝛼𝛼 and ℱ 𝛽𝛽 : s o ft m a x
Revised: The features are aggregated via concatenating-compressing by an FC to f (cid:48)

Original: s o ft m a x 𝑉𝑉𝑂𝐾𝐾𝑂 𝑄𝑄𝑂 𝑄𝑄𝐾𝐾 𝑉𝑉 ⨂ self-attention self-attention ⨂ FC ⨂ ⨂ FC concat ⋯
Revised: α .

Original: As shown in Fig. 6, for causal inference operation (Sec. 5), f  αis ﬁrst separated to attribute categorywise features by an FC and then aggregated via concatenating-compressing by an FC to f(cid:48) α .
Revised: In this way, we can manipulate speciﬁc attributes of f (cid:48) α by masking some certain f α p , in order to evaluate the causal learning of our model.

Original: We evaluate different aggregation methods in the ablations.
Revised: We compare different aggregation methods in the ablations.

Original: And f B = { f B i } mi =1 .
Revised: And f B = { f B i } mi =1 .

Original: Given the conditions { f I , f (cid:48) α , f B i } , F β ( · ) operate the instantiation.
Revised: Given the conditions { f I , f (cid:48) α , f B i } , F β ( · ) operate the instantiation.

Original: F B ( · ) , F β ( · ) are implemented the same as F A ( · ) , F α ( · ) , except the ﬁrst input condition of F β is the concatenation of f (cid:48)
Revised: F B ( · ) , F β ( · ) are implemented the same as F A ( · ) , F α ( · ) , except the ﬁrst input condition of F β is the concatenation of f (cid:48)

Original: α , f I .
Revised: α , f I .

Original: Learning Objectives.
Revised: Learning Objectives.

Original: To drive the learning, we devise several objectives.
Revised: To drive the learning, we devise several objectives.

Original: (1) Category-level loss L C .
Revised: (1) Category-level loss L C .

Original: After obtaining the category-level representations f A , f B , we input them to two linear classiﬁers g A , g B to classify A, B , given GT O in training:S A i =
Revised: After obtaining the category-level representations f A , f B , we input them to two linear-Sigmoid classiﬁers g A , g B to classify A, B , given GT O in training:

Original: g A ( f A i ) , S B i = g B ( f B i ) . And S A = {S A i } mi =1 , S B = {S B i } mi =1 . Next, We use multi-Sigmoid to get A i, B i probabilities: P
Revised: P A i

Original: A i =
Revised: =

Original: Sigmoid ( S A i ) , P B i = 
Revised: g A ( f A i ) , P B i = g B

Original: Sigmoid ( S B i ) .
Revised: ( f B i ) .

Original: And P A = { P A i } mi =1 , P B = { P B i } mi =1 .

Original: The Sig- moid cross-entropy losses of the i -th object category are represented as L A i and L B i .
Revised: The binary cross-entropy losses of the i -th object category are represented as L A i and L B i .

Original: So the total category-level loss is L C
Revised: So the total category-level loss is L C

Original: = (cid:80) mi =

Revised: = (cid:80) mi =

Original: ( L A i + L B i ) .
Revised: ( L A i + L B i ) .

Original: (2) Instance-level loss L I .
Revised: (2) Instance-level loss L I .

Original: As for the instancelevel, we send f α , f β , together with the features before expectation F α ( f I , f A i ) , F β ( f I , f (cid:48) α , f B i ) to linear classiﬁers g α and g β .
Revised: As for the instance-level, we send f α , f β , together with the features before expectation F α ( f I , f A i ) , F β ( f I , f (cid:48) α , f B i ) to linear classiﬁers g α and g β .

Original: The  separated attribute features  are also sent to independent binary classiﬁers g α p . The Sigmoid binary cross-entropy losses  of these output logits are represented as L α, L β .
Revised: The binary crossentropy losses of these output logits are represented as L α , L β . The separated attribute features f α p are also sent to independent binary classiﬁers g α p and computed losses with ground truth of p th attribute, which is included in L α

Original: The instance-level loss is then L I =
Revised: The instance-level loss is then L I =

Original: L α + L β .
Revised: L α + L β .

Original: Overall, the total loss is L = λ C L C + L I .
Revised: Finally, the total loss is L = λ C L C + L I .

Original: OCRN adopts a two-stage policy, i.e., ﬁrst inferring attributes and then reasoning affordances.
Revised: We adopt a two-stage policy: ﬁrst inferring attributes, then reasoning affordances.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: Metric .
Revised: Metric .

Original: (a) α, β Learning . An object can have multiple attributes and affordances (multi-label classiﬁcation) thus we use mean Average Precision (mAP) .(b) Causal Reasoning .
Revised: For α, β learning, as an object can have multiple attributes and affordances (multi-label classiﬁcation), we use mean Average Precision (mAP) .

Original: We adopt Total Direct Effect (TDE) (Pearl et al., 2016) for evaluation.
Revised: For causal reasoning, we adopt TDE (Pearl et al., 2016) for evaluation.

Original: The general idea of TDE is to assess the effect of a variable: constructing the counterfactual scenario via eliminating the effect of the controlled variable, then subtracting the counterfactual from the original.
Revised: The general idea of TDE is to assess the effect of a variable: constructing the counterfactual scenario via eliminating the effect of the controlled variable, then subtracting the counterfactual from the original.

Original: Give an intuitive example, the effect of cargo can be measured by the speed difference between a loaded car and an empty car.
Revised: Give an intuitive example, the effect of cargo can be measured by the speed difference between a loaded car and an empty car.

Original: In OCL, we formulate TDE as the affordance probability change  between P ( β q ) and P α p TDE ( β q )  ( p ∈ [1 , 114] , q ∈ [1 , 170] ): comparing the original with its counterfactual without the effect of attribute α p .
Revised: In OCL, we formulate TDE as the affordance probability change ∆ P β . Here, we redeﬁne ∆ P β as the difference between P ( β q ) and P α p TDE ( β q ) : ( p ∈ [1 , 114] , q ∈ [1 , 170] ): comparing the original with its counterfactual without the effect of attribute α p .

Original: For an object, let P ( β q ) denotes its original probability of β q estimated by a trained model.
Revised: For an object, let P ( β q ) denote its original probability of β q estimated by a trained model.

Original: As we use learned attribute feature to infer β q , we assign zero-mask (Tang et al., 2020) to the feature of α p and keep the other attribute features, to get P α p TDE ( β q ) with the same model parameters.
Revised: As we use learned attribute feature to infer β q , we assign zero-mask (Tang et al., 2020b) to the feature of α p and keep the other attribute features, to get P α p TDE ( β q ) with the same model parameters.

Original: After concurrently operating TDE for all [ α p , β q ] , we obtain a huge cube of instance-attribute-affordance , where each grid indicates the assessed effect of certain [ α p , β q ] for each instance .
Revised: After concurrently operating TDE for all [ α p , β q ] , we obtain a huge cube of instance-attribute-affordance , where each grid indicates the assessed effect of certain [ α p , β q ] for each instance .

Original: For each [ α p , β q ] , We score instances following ( Appendix Sec.
Revised: For each [ α p , β q ] , We score instances following ( Appendix Sec.

Original: A.3 ): (1) TDE :
Revised: A.4 ): ( 1) TDE :

Original: If [ α p , β q ] has annotated causality for an instance, when eliminating the effect of α p, | P ( β q ) − P α p TDE ( β q ) | should be large.
Revised: If [ α p , β q ] has annotated causality, when eliminating the effect of α p | P ( β q ) − P α p TDE ( β q ) | should be large.

Original: If [ α p , β q ] does nothave causality, the difference should be small.
Revised: If [ α p , β q ] does not

Original: Beside the change scale, the right change direction is also expected.
Revised: have causality, the difference should

Original: We set TDE score TDE α p β q as max ( P ( β q ) − P α p TDE ( β q ) , 0) when GT β q = 1 or max ( P α p TDE ( β q ) − P ( β q ) , 0)
Revised: be

Original: when
Revised: small

Original: GT β q = 0 , where GT β q is the label of β q .
Revised: Beside the change scale, the right change direction is also expected.

Original: (2) α - β -TDE : [Table] [Figure] [Table]
Revised: We set TDE score TDE α p β q as:

Original: [Figure]
Revised: [Equation]

Original: [Table] [Table] [Table]
Revised: where GT β q is the label of β q .

Original: [Figure]
Revised: (2) α β

Original: Machine should also infer α p , β q  accurately.
Revised: TDE : α p , β q should alse be infered accurately.

Original: We multiply TDE α p β q with P ( α p =
Revised: We multiply TDE α p β q with P ( α p =

Original: GT α p ) ∗ P ( β q = GT β q ) as a uniﬁed score.
Revised: GT α p ) ∗ P ( β q = GT β q ) as a uniﬁed score

Revised: We compute AP for

Revised: each

Original: We compute AP for each [ α p , β q ] and average them to mAP.
Revised: α p , β q ] and average them to mAP.

Original: Baselines .
Revised: Baselines .

Original: We list all baselines here brieﬂy (detailed in Appendix Sec.
Revised: We list all baselines here brieﬂy (detailed in Appendix Sec.

Original: A.4 ): (1) Direct Mapping from f I to P α , P β (DM-V); (2) DM from Attribute (DM-At); (3) DM from Attribute and Object (DM-AtO); (4) DM from Bert (Devlin et al., 2018)
Revised: A.5 ): (1) Direct Mapping from f I to P α , P β (DM-V); (2) DM from Attribute (DM-At); (3) DM from Attribute and Object [Table] [Figure] [Table]

Original: Linguistic Representation (DM-L); (5) VisualLinguistic Representation, i.e., Multi-Modality (MM); (6) Markov Logic Network (Richardson & Domingos, 2006), using GT α to infer β (MLN-GT); (7) Linguistic Correlation of O - α , O - β (LingCorr); (8) Kernelized Probabilistic Matrix Factorization (KPMF)
Revised: [Figure]

Original: (Zhou et al., 2012); (9) A & B
Revised: [Table]

Original: Lookup, directly getting P A , P B from M (cid:48) A , M (cid:48) B as P α , P β ; (10) Cause as Attention (Attention); (11) Hierarchical Mapping from I to A / B then to α / β (HMa).
Revised: [Table]

Original: (12) Ngram (Lin et al., 2012).
Revised: [Table] [Figure]

Original: TDE loss .
Revised: TDE loss .

Original: Though the machine is expected to learn the causalities given α, β labels only, as an ablation, we wonder how it would perform giventhe causal supervision.
Revised: Though the machine is expected to learn the causalities given α, β labels only. We also wonder how it would perform given causal supervision.

Original: Thus, we adopt an extra Hinge loss to maximize the TDE score of all [ α p , β q ].
Revised: We adopt an extra Hinge loss to maximize the TDE score of all [ α p , β q ].

Original: In detail, without the effect of one certain α p , we intend the probability of causal-related β q to keep farther from GT β q by a margin τ (= 0.1 in our experiments), formulated as max { 0 , τ − ( P ( β q ) − P α p TDE ( β q )) } when GT β q = 1 and max { 0 , τ − ( P α p TDE ( β q ) − P ( β q )) } when GT β q = 0 .
Revised: In detail, without the effect of one certain α p , we intend the probability of causal-related β q to keep farther from GT β q by a margin τ (= 0.1 in experiments) formulated as: [Equation]

Original: We enumerate all annotated [ α p , β q ] of an instance to obtain L TDE .
Revised: We enumerate all annotated [ α p , β q ] of an instance to obtain L TDE .

Original: Different from the default, the total loss here is L = λ C L C + L I + λ TDE L TDE .
Revised: Different from the default, the total loss here is L = λ C [Equation] + [Equation]

Revised: +

Revised: [Equation]

Original: Implementation Details .
Revised: Implementation Details .

Original: For a fair comparison, all methods adopt a shared COCO (Lin et al., 2014)
Revised: For a fair comparison, all methods adopt a shared COCO (Lin et al., 2014)

Original: pre-trained ResNet-50 (He et al., 2016) (frozen) to extract f I and use the same object boxes in training and inference.
Revised: pre-trained ResNet-50 (He et al., 2016) (frozen) to extract f I and use the same object boxes in training and inference.

Original: In OCRN, the dimension of f I and all representations f A i , f B i , f α , f β is 1024.
Revised: In OCRN, the dimension of f I and all representations f A i , f B i , f α , f β is 1024.

Original: The individual features of each attribute category is 512d and aggregated to 1024d by an FC.
Revised: The individual features of each attribute category is 512d and aggregated to 1024d by an FC.

Original: We train the attribute module with a learning rate of 0.3 and batch size of 1024 for 470 epochs.
Revised: We train the attribute module with a learning rate of 0.3 and batch size of 1024 for 470 epochs.

Original: Then the attribute module is frozen, and the affordance module is trained with a learning rate of 3.0eand batch size of 768 for 20 epochs.
Revised: Then the attribute module is frozen, and the affordance module is trained with a learning rate of 3.0eand batch size of 768 for 20 epochs.

Original: In training, λ C = 0 .
Revised: In training, λ C = 0 .

Original: 03 , λ TDE = 3 .
Revised: 03 , λ TDE = 3 .

Original: Results .
Revised: Results .

Original: Tab.
Revised: Tab.

Original: 2 presents the results.
Revised: 2 presents the results.

Original: OCRN outperforms the baselines and achieves decent improvements on all tracks.
Revised: OCRN outperforms the baselines and achieves decent improvements on all tracks.

Original: In terms of α , OCRN largely outperforms DM-V with 1.7 mAP (default) and
Revised: In terms of α , OCRN largely outperforms DM-V with 1.7 mAP (default) and

Original: DM-At with 3.1 mAP (w/ L TDE ).
Revised: DM-At with 3.1 mAP (w/ L TDE ).

Original: As for β , OCRN also achieves 1.0 mAP (default) and 1.4 mAP (w/ L TDE ) improvements compared to DM-At and DM-AtO w/ L TDE .
Revised: As for β , OCRN also achieves 1.0 mAP (default) and 1.4 mAP (w/ L TDE ) improvements compared to DM-At and DM-AtO w/ L TDE .

Original: Comparatively, HMa utilizes the supervision of A, B , but it performs much worse.
Revised: Comparatively, HMa utilizes the supervision of A, B , but it performs much worse.

Original: A & B Lookup directly uses GT A, B to infer α, β , but its poor performance veriﬁes the signiﬁcant difference between A, B and α, β .
Revised: A & B Lookup directly uses GT A, B to infer α, β , but its poor performance veriﬁes the signiﬁcant difference between A, B and α, β .

Original: Moreover, we ﬁnd that all methods perform better on β than α .
Revised: Moreover, we ﬁnd that all methods perform better on β than α .

Original: This may because α are more diverse than β , e.g., we can eat lots of food s, but food s usually have various attributes ( fruit v.s. pizza ).
Revised: This may because α are more diverse than β , e.g., we can eat lots of food s, but food s usually have various attributes ( fruit v.s. pizza ).

Original: Another reason is that the positive samples in β labels (23.2%) are much more than the positives in α labels (9.4%).
Revised: Another reason is that the positive samples in β labels (23.2%) are much more than the positives in α labels (9.4%).

Original: The different pos-neg ratio affect the learning a lot and result in the above gap.
Revised: The different pos-neg ratio affect the learning a lot and result in the above gap.

Original: [Figure]

Original: α β TDE α - β [Figure] TDE 
Revised: In TDE evaluation, without the guidance of L TDE , all methods achieve unsatisﬁed performances.

Original: [Figure] 17.0 w [Figure] 16.1 w/o L A
Revised: However, OCRN still has an advantage.

Original: i ,L [Figure] [Figure] [Figure]
Revised: Only MLN adopting the ﬁrst-order logic and GT α labels is

Original: [Figure]

Original: 16.7 w/o L α

Original: [Figure]
Revised: [Figure]

Original: L β

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: 2048Dims

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] In TDE evaluation, without the guidance of L TDE , all methods achieve unsatisﬁed performances. However, OCRN still has an advantage. Only MLN adopting the ﬁrst-order logic and GT α labels is comparable with OCRN.
Revised: comparable with OCRN.

Original: If trained with L TDE and direct causality labels, all methods perform much better to learn complex causalities.
Revised: If trained with L TDE and direct causality labels, all methods perform much better to learn complex causalities.

Original: Particularly, the typical deep learning model Attention performs best in baselines, but MLN no longer holds the previous advantage.
Revised: Particularly, the typical deep learning model Attention performs best in baselines, but MLN no longer holds the previous advantage.

Original: Relatively, OCRN shows greater improvements and outperforms Attention with 2.5 and 1.4 mAP on two TDE tracks.
Revised: Relatively, OCRN shows greater improvements and outperforms Attention with 2.5 and 1.4 mAP on two TDE tracks.

Original: Due to the page limit, we provide more analyses in Appendix Sec.
Revised: Due to the page limit, we provide more analyses in Appendix Sec.

Original: A.5 .
Revised: A.6 .

Original: In Appendix Sec.
Revised: In Appendix Sec.

Original: A.6 , we further apply OCRN to Human-Object Interaction Detection (Chao et al., 2018), where OCRN boosts the performance and veriﬁes the efﬁcacy and generalization of OCL.
Revised: A.7 , we further apply OCRN to Human-Object Interaction Detection (Chao et al., 2018), where OCRN boosts the performance and veriﬁes the efﬁcacy and generalization of OCL.

Original: Visualization.
Revised: Visualization.

Original: (1) In Fig.
Revised: (1) In Fig.

Original: 7, we show the correct instance proportions ( % ) of OCRN and Attention after TDE.
Revised: 6, we show the correct instance proportions ( % ) of OCRN and Attention after TDE.

Original: (a) depicts the randomly chosen causal [ α p , β q ] pairs with GT β q = 1 expecting P ( β q ) >
Revised: (a) depicts the randomly chosen causal [ α p , β q ] pairs with GT β q = 1 expecting P ( β q ) >

Original: P α p TDE ( β q ) .
Revised: P α p TDE ( β q ) .

Original: (b) depicts the randomly chosen causal [ α p , β q ] pairs with GT β q = 0 expecting P ( β q ) <
Revised: (b) depicts the randomly chosen causal [ α p , β q ] pairs with GT β q = 0 expecting P ( β q ) <

Original: P α p TDE ( β q ) .
Revised: P α p TDE ( β q ) .

Original: The higher proportions indicate that OCRN performs better than Attention on TDE.
Revised: The higher proportions indicate that OCRN performs better than Attention on TDE.

Original: To verify the deconfounding, we compare the model bias of OCRN w/ or w/o deconfounding .
Revised: To verify the deconfounding, we compare the model bias of OCRN w/ or w/o deconfounding .

Original: The bias of category O upon an attribute α is measured following Zhao et al.
Revised: The bias of category O upon an attribute α is measured following Zhao et al.

Original: (2017), by b ( O, α )
Revised: (2017), by b ( O, α )

Original: =
Revised: =

Original: c ( O, α ) / (cid:80) α (cid:48) c
Revised: c ( O, α ) / (cid:80) α (cid:48) c

Original: ( O, α (cid:48) ) .
Revised: ( O, α (cid:48) ) .

Original: When measuring data bias, c
Revised: When measuring data bias, c

Original: ( O, α ) is the number of co-occurrence of O and α in OCL, and when it comes to model bias,
Revised: ( O, α ) is the number of co-occurrence of O and α in OCL, and when it comes to model bias,

Original: c
Revised: c

Original: ( O, α ) is the sum of probabilities that O are predicted positive with α .
Revised: ( O, α ) is the sum of probabilities that O are predicted positive with α .

Original: The bias of β is measured in the same manner.
Revised: The bias of β is measured in the same manner.

Original: Fig. 8 and 9 show some examples of the biases of training data and models, indicating that OCRN deconfounding effectively prevents the model from bias toward the train set.
Revised: Fig. 7 and 8 show some examples of the biases of training data and models, indicating that OCRN deconfounding effectively prevents the model from bias toward the train set.

Original: Ablation Study .
Revised: Ablation Study .

Original: We conduct ablation studies to verify the components of OCRN w/ L TDE (Tab. 3) on the val set.
Revised: We conduct ablation studies to verify the components of OCRN w/ L TDE (Tab. 3) on the val set.

Original: (1) Deconfounding.
Revised: (1) Deconfounding.

Original: OCRN w/o deconfounding is implemented following Eq. 2 and 5, where P ( O | I ) and P ( O | I, α ) are the category predictions of pre-trained detectors (Liu et al., 2021).
Revised: OCRN w/o deconfounding is implemented following Eq. 2 and 5, where P ( O | I ) and P ( O | I, α ) are the category predictions of pre-trained detectors (Liu et al., 2021).

Original: Both α and β performances signiﬁcantly decline, as explained by Fig.
Revised: Both α and β performances signiﬁcantly decline, as explained by Fig.

Original: 8 and 9.
Revised: 7 and 8.

Original: (2) Losses.
Revised: (2) Losses.

Original: The performances slightly drop after removing category-level L A i , L B i , but signiﬁcantly drop without instance-level L α , L β .
Revised: The performances slightly drop after removing category-level L A i , L B i , but signiﬁcantly drop without instance-level L α , L β .

Original: (3) Feature dimension.
Revised: (3) Feature dimension.

Original: For f A i , f B i , f α , f β , smaller and larger feature sizes all have degrading effects.
Revised: For f A i , f B i , f α , f β , smaller and larger feature sizes all have degrading effects.

Original: (4) TDE-related implementations.
Revised: (4) TDE-related implementations.

Original: We probe some different methods: (a) Mean aggregation: f (cid:48)
Revised: We probe some different methods: (a) Mean aggregation: f (cid:48)

Original: α = (cid:80) i f
Revised: α = (cid:80) i f

Original: α p ; (b) Max-pooling aggregation: f (cid:48)
Revised: α p ; (b) Max-pooling aggregation: f (cid:48)

Original: α is the max value of f α p as each component; (c) Random counterfactual feature: assigned random vector as the counterfactual attribute feature (instead of zero vector) during TDE.
Revised: α is the max value of f α p as each component; (c) Random counterfactual feature: assigned random vector as the counterfactual attribute feature (instead of zero vector) during TDE.

Original: These methods perform worse than the chosen setting on TDE performance but are comparable on α and β performance.
Revised: These methods perform worse than the chosen setting on TDE performance but are comparable on α and β performance.

Original: Discussion .
Revised: Discussion .

Original: Overall, OCL poses extreme challenges to current AI systems.
Revised: Overall, OCL poses extreme challenges to current AI systems.

Original: It expects representative learning to accurately recognize attribute and affordance from raw data meanwhile causal inference to capture the causalities within diverse instances and contexts, i.e., both the intuitive System 1 and logical System 2 (Bengio, 2019).
Revised: It expects representative learning to accurately recognize attribute and affordance from raw data meanwhile causal inference to capture the causalities within diverse instances and contexts, i.e., both the intuitive System 1 and logical System 2 (Bengio, 2019).

Original: From the experiments, we ﬁnd that models struggle to achieve satisfying results on all tracks simultaneously.
Revised: From the experiments, we ﬁnd that models struggle to achieve satisfying results on all tracks simultaneously.

Original: Notably, it is difﬁcult to achieve a satisfying TDE score via data ﬁtting, and there is much room for improvement.
Revised: Notably, it is difﬁcult to achieve a satisfying TDE score via data ﬁtting, and there is much room for improvement.

Original: For future studies, a harmonious performance on α, β , and causality learning are encouraged to capture object knowledge better.
Revised: For future studies, a harmonious performance on α, β , and causality learning are encouraged to capture object knowledge better.

Original: Potential directions may include causal representation learning (Sch¨olkopf et al., 2021), casual reinforcement learning (Bareinboim, 2020), neural-symbolic reasoning (Besold et al., 2017), etc.
Revised: Potential directions may include causal representation learning (Sch¨olkopf et al., 2021), casual reinforcement learning (Bareinboim, 2020), neural-symbolic reasoning (Besold et al., 2017), etc.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduce object concept learning (OCL), which expects machines to infer affordances and explain what attributes enable an object to possess these affordances.
Revised: In this work, we introduce object concept learning (OCL), which expects machines to infer affordances and explain what attributes enable an object to possess these affordances.

Original: Accordingly, we build an extensive real-world dataset and present a baseline OCRN based on casual intervention and instantiation.
Revised: Accordingly, we build an extensive real-world dataset and present a baseline OCRN based on casual intervention and instantiation.

Original: OCRN achieves decent performance and follows the causalities well.
Revised: OCRN achieves decent performance and follows the causalities well.

Original: However, OCL remains challenging and would inspire a line of studies upon reasoning-based object understanding.
Revised: However, OCL remains challenging and would inspire a line of studies upon reasoning-based object understanding.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: This work primarily proposes a daily object dataset to advance machine understanding of objects for the future study of embodied AI, intelligent robots, etc.
Revised: This work primarily proposes a daily object dataset to advance machine understanding of objects for the future study of embodied AI, intelligent robots, etc.

Original: Our work may help the home service robots better understand the human living scenes and then help to improve our lives, especially for the health care of patients, elders, and children.
Revised: Our work may help the home service robots better understand the human living scenes and then help to improve our lives, especially for the health care of patients, elders, and children.

Original: All the data used in OCL are publicly available.
Revised: All the data used in OCL are publicly available.

Original: We have cited the creators and followed the ofﬁcial licenses.
Revised: We have cited the creators and followed the ofﬁcial licenses.

Original: For all images, we only provide the URL links in our future web for research uses only.
Revised: For all images, we only provide the URL links in our future web for research uses only.

Original: Moreover, modern deep learning models are usually computationally expensive.
Revised: Moreover, modern deep learning models are usually computationally expensive.

Original: Our proposed model OCRN also needs multiple GPUs to train.
Revised: Our proposed model OCRN also needs multiple GPUs to train.

Original: In the future, OCL encourages a new line of studies to understand objects’ common sense and combine causal reasoning and deep learning.
Revised: In the future, OCL encourages a new line of studies to understand objects’ common sense and combine causal reasoning and deep learning.

Original: We believe the future works focusing on learning the causality instead of heavily ﬁtting data would be more computationally lite, less data-hungry, and low-carbon.
Revised: We believe the future works focusing on learning the causality instead of heavily ﬁtting data would be more computationally lite, less data-hungry, and low-carbon.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide our source code, OCL dataset samples, and a video demo in the supplementary material (the zip ﬁle).
Revised: We provide our source code, OCL dataset samples, and a video demo in the supplementary material (the zip ﬁle).

Original: It includes all the manuscripts to preprocess and load data, construct the OCRN model with the hyper-parameters, train and test the model, evaluate the performance on the OCL benchmark, etc.
Revised: It includes all the manuscripts to preprocess and load data, construct the OCRN model with the hyper-parameters, train and test the model, evaluate the performance on the OCL benchmark, etc.

Original: Please refer to the code ﬁle for more details.
Revised: Please refer to the code ﬁle for more details.

Original: The details of related datasets, OCL data processing and annotation are in the main text Sec. 3, Appendix Sec.
Revised: The details of related datasets, OCL data processing and annotation are in the main text Sec. 3, Appendix Sec.

Original: A.2, A.7.
Revised: A.3, A.11.

Original: Our code and data will be publicly available.
Revised: Our code and data will be publicly available.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: We report more details and analyses about OCL in the appendix:
Revised: We report more details and analyses about OCL in the appendix:

Original: Sec.
Revised: Sec.

Original: A.1: Category/Attribute/Affordance Selection
Revised: A.1: Category/Attribute/Affordance Selection

Original: Sec.
Revised: Sec.

Revised: A.2: Annotation Details

Revised: Sec.

Original: A.2: OCL Characteristics
Revised: A.3: OCL Characteristics

Original: Sec.
Revised: Sec.

Original: A.3: TDE Metric Details
Revised: A.4: TDE Metric Details

Original: Sec.
Revised: Sec.

Original: A.4:
Revised: A.5:

Original: Baseline Details
Revised: Baseline Details

Original: Sec.
Revised: Sec.

Original: A.5:
Revised: A.6:

Original: Detailed Result Analysis
Revised: Detailed Result Analysis

Original: Sec.
Revised: Sec.

Original: A.6: Application on Human-Object Interaction (HOI) Detection 
Revised: A.7: Application on Human-Object Interaction (HOI) Detection Sec. A.8:

Revised: Comparison on Debiasing

Original: Sec.
Revised: Sec.

Original: A.7: Detailed Lists
Revised: A.9: About States

Original: A.
Revised: Sec. A.10: Discussion about Causality and Causal Graph Sec. A.11:

Revised: Detailed Lists

Original: [Figure]
Revised: [Figure]

Original: We ﬁrst choose affordances, categories, and attributes, considering their causal relations.
Revised: We ﬁrst choose affordances, categories, and attributes, considering their causal relations.

Revised: We present their word cloud in Fig.

Original: The complete lists can be found in Appendix Sec.
Revised: The complete lists can be found in Appendix Sec.

Original: A.7.
Revised: A.11.

Original: (1) Affordance : To build a general and applicable knowledge base, we collect 1,006 affordance candidates from several widely-used action or affordance datasets (Chao et al., 2015;
Revised: (1) Affordance : To build a general and applicable knowledge base, we collect 1,006 affordance candidates from several widely-used action or affordance datasets (Chao et al., 2015;

Original: Gu et al., 2018; Chao et al., 2018; Gupta & Malik, 2015; Zhu et al., 2014; Nguyen et al., 2017) (957 from
Revised: Gu et al., 2018; Chao et al., 2018; Gupta & Malik, 2015; Zhu et al., 2014; Nguyen et al., 2017) (957 from

Original: Chao et al.
Revised: Chao et al.

Original: (2015), 160 from Gu et al.
Revised: (2015), 160 from Gu et al.

Original: (2018), 146 from Chao et al.
Revised: (2018), 146 from Chao et al.

Original: (2018), 97 from Gupta & Malik (2015), 41 from Zhu et al.
Revised: (2018), 97 from Gupta & Malik (2015), 41 from Zhu et al.

Original: (2014), 21 from Nguyen et al.
Revised: (2014), 21 from Nguyen et al.

Original: (2017), with overlaps ).
Revised: (2017), with overlaps ).

Original: We ﬁnd that not all affordances are in common use and some of them are difﬁcult for visual recognition, e.g., accept (consider right and proper).
Revised: We ﬁnd that not all affordances are in common use and some of them are difﬁcult for visual recognition, e.g., accept (consider right and proper).

Original: So each candidate is scored by 5 human experts from 0.0 to 5.0 according to generality and commonness.
Revised: So each candidate is scored by 5 human experts from 0.0 to 5.0 according to generality and commonness.

Original: We keep 170 top-scored affordances in our base ( from Chao et al.
Revised: We keep [Table] [Table]

Original: (2015), 78 from Gu et al. (2018), 127 from Chao et al. (2018), 53 from Gupta &
Revised: desk ipod vase ibex rabbit swing sock axe lion panda helmet koala ambulance hen gown swan scarf carrot owl bathtub bee microwave oven shield wallet mask barbell yurt pillow broom coffee meat loaf taxi golf cart drill telephone mobile phone hay pig harmonica lemon couch closet cabbage ski jeans measuring cup unicycle pen boat knee pad cardigan cello parachute trumpet oar curtain piano vestment guitar clog zebra bookcase fox spider box fish lizard truck pomegranate sculpture towel vehicle bread worm wheel minibus butterfly steel drum toaster yawl clock lamp glove goat

Original: Malik (2015), 13 from Zhu
Revised: [Figure] control [Figure]

Original: et al.
Revised: chime

Original: (2014), 11 from Nguyen et al. (2017), with
Revised: [Figure]

Original: (2) Category :
Revised: (2) Category :

Original: Considering the taxonomy (WordNet (Fellbaum, 2012)), we collect a pool 
Revised: Considering the taxonomy (WordNet (Fellbaum, 2012)), we collect a pool with over 1,742 object categories from previous datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016;

Original: with over 1, object categories from previous datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016; Liu et al., 2017) (32 from Farhadi et al.
Revised: Liu et al., 2017) (32 from Farhadi et al.

Original: (2009), 28 from Patterson & Hays (2016),  from Xiao et al.
Revised: (2009), 28 from Patterson & Hays (2016), 717 from Xiao et al.

Original: (2010), 1000 from Liu et al.
Revised: (2010), 1000 from Liu et al.

Original: (2017), with overlaps )
Revised: (2017), with overlaps ).

Original: Then we merge the similar categories according to WordNet (Fellbaum, 2012), and ﬁlter out the categories which are not common daily objects (e.g. “man”, “planet”), unrelated to the above 170 affordances (e.g.
Revised: Then we merge the similar categories according to WordNet (Fellbaum, 2012), and ﬁlter out the categories which are not common daily objects (e.g. “man”, “planet”), unrelated to the above 170 affordances (e.g.

Original: “skyscraper”) or too uncommon (e.g. “malleefowl”).
Revised: “skyscraper”) or too uncommon (e.g. “malleefowl”).

Original: Finally our database has 381 common object categories.
Revised: Finally our database has 381 common object categories.

Original: These object categories are divided into 12 super categories, showed in Fig.
Revised: These object categories are divided into 12 super categories, showed in Fig.

Original: (3) Attribute : We extract the attribute classes from several large-scale attribute datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016; Liu et al., 2017; Krishna et al., 2016)
Revised: (3) Attribute : We extract the attribute classes from several large-scale attribute datasets (Farhadi et al., 2009; Xiao et al., 2010; Patterson & Hays, 2016; Liu et al., 2017; Krishna et al., 2016)

Original: ( from Farhadi et al.
Revised: ( from Farhadi et al.

Original: (2009), 203 from Patterson & Hays (2016), 66 from Xiao et al.
Revised: (2009), 203 from Patterson & Hays (2016), 66 from Xiao et al.

Original: (2010), 25 from
Revised: (2010), 25 from

Original: Liu et al.
Revised: Liu et al.

Original: (2017), top 500 from Krishna et al.
Revised: (2017), top 500 from Krishna et al.

Original: and manually ﬁlter the 500 most frequent attributes.
Revised: and manually ﬁlter the 500 most frequent attributes.

Original: Five experts give 0.0 to 5.0 scores based on their relevance to human actions and the selected 170 affordances to better explore the causal relations between attributes and affordances.
Revised: Five experts give 0.0 to 5.0 scores based on their relevance to human actions and the selected 170 affordances to better explore the causal relations between attributes and affordances.

Original: Some attributes (e.g. cloudy , competitive ) that are not helpful for affordance reasoning are discarded.
Revised: Some attributes (e.g. cloudy , competitive ) that are not helpful for affordance reasoning are discarded.

Original: Finally, 114 attributes are kept, covering colors, deformations, supercategories, various surface, geometrical, and physical properties.
Revised: Finally, 114 attributes are kept, covering colors, deformations, supercategories, various surface, geometrical, and physical properties.

Original: A. OCL C HARACTERISTICS A.2. O BJECT B OX S IZE
Revised: A. A NNOTATION D ETAILS

Revised: Attribute Annotation. (1) Category-level attribute ( A ). Following Osherson et al.

Revised: (1991), to avoid bias, annotators are given category-attribute pairs (category names , not images).

Revised: They propose a 0-3 score according to the category concept in their minds (0: No, 1: Normally No, 2:

Revised: Normally Yes, 3: Yes).

Revised: Each pair is annotated by three annotators and takes the plurality as A label.

Revised: If the range of 3 proposals exceeds 1, another three annotators will re-annotate this pair until achieving consensus.

Revised: We binarize the annotations (0: No, 1: Yes) with a threshold of 2 and get a category-level attribute matrix M A ( [381 , 114] ).

Revised: (2) Instance-level attribute ( α ).

Revised: Two annotators label each pair with 0 (No) and 1 (Yes).

Revised: If they give different labels, this pair will be handed over to another two annotators until meeting consensus.

Revised: Affordance Annotation.

Revised: (1) Category-level affordance B .

Revised: Following Chao et al.

Revised: (2) Instance-level affordance β is annotated for every instance with the help of object states (Isola et al., 2015).

Revised: As B is determined by common states, objects in speciﬁc states may have different affordances from B , e.g., we cannot board a flying plane.

Revised: Moreover, if a service robot ﬁnds a broken cup , it may infer that the cup can still hold water as it is trained with B labels.

Revised: Thus, we need detailed β beyond B .

Revised: As the instances in the same state should have similar β (all rotten apple s cannot be eaten ),

Revised: six experts

Revised: ﬁrst conclude the states.

Revised: The experts scan all instances of each category and use their knowledge of affordance to deﬁne all the existing states.

Revised: Then all 186 K instances are dispatched to the concluded states via crowdsourcing.

Revised: If some instances do not belong to any predeﬁned states, they will be returned to the experts to add more states.

Revised: In total, 1, states are deﬁned, and each category has 3.6 states on average.

Revised: Next, β is annotated for each state.

Revised: Given a state-affordance pair and example images, two annotators mark it with 0 (No) and 1 (Yes).

Revised: The results are combined in the same way as α .

Revised: Thus, each instance would have a state and the corresponding β .

Revised: An annotator would recheck each instance together with its state and β labels to ensure the quality.

Revised: If its state is inaccurate or the state β labels are unsuitable, this annotator would correct them.

Revised: Causal Relation Annotation.

Revised: (1) Filtering .

Revised: Starting from the [114,170] matrix of α - β classes, we ask three experts to vote the causal relation of each class.

Revised: They scan all instances to answer whether the relation exists in any cases.

Revised: Finally, we obtain about 10% α - β classes as candidates.

Revised: (2) Instance-level causality : we also adopt object states as a reference.

Revised: For each state- α - β triplet, two annotators are asked whether the speciﬁc attribute is the direct cause of this affordance in this state and gives their binary answer.

Revised: We use the same method in annotating β to combine results and assign state-level labels to instances.

Revised: Next, for all instances of a state, an expert decides whether the state-level relations are reasonable for each instance in speciﬁc contexts and correct the inaccurate ones.

Revised: Finally, we obtain about 2 M instance- α - β triplets of causal relations.

Revised: A Running Example of Dataset Construction.

Revised: A running example is shown in Fig.

Revised: 11 to show the process of annotations clearly.

Revised: banana model/toy … … banana model/toy banana model/toy fake √ furry √ tasty × … touch √ eat × hold √ … …

Revised: Collect images (banana)

Revised: Detect objects

Revised: Propose states

Revised: Annotate attributes and

Revised: [Figure]

Revised: Annotate direct causal relations fake √ eat × fake √ furry √ tasty × touch √ eat × hold √ running example of dataset construction.

Revised: A. OCL C HARACTERISTICS A.3. O BJECT B OX S IZE

Original: We visualize the distribution of normalized object box size in Fig.
Revised: We visualize the distribution of normalized object box size in Fig.

Original: 11, where the box width and height are normalized by the width and height of the whole image.
Revised: 12, where the box width and height are normalized by the width and height of the whole image.

Original: It shows that most objects in our knowledge base are small objects , providing abundant regional information.
Revised: It shows that most objects in our knowledge base are small objects , providing abundant regional information.

Original: [Figure]

Original: A.2. A NNOTATOR I NFORMATION
Revised: A.3. A NNOTATOR I NFORMATION

Original: Annotators’ age, major, and education degree are presented in Fig. 
Revised: Annotators’ age, major, and education degree are presented in Fig. 13, 14, and 15.

Original: [Figure]
Revised: [Figure]

Original: and 14.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A.2. M ATRIX S AMPLES
Revised: A.3. M ATRIX S AMPLES

Original: The category-level attribute and affordance ( A, B ) matrices are detailed in Fig.
Revised: The category-level attribute and affordance ( A, B ) matrices are detailed in Fig.

Original: 15, 16 as heatmaps, the cells with dark color indicate positive samples.
Revised: 16, 17 as heatmaps, the cells with dark color indicate positive samples.

Original: For example, ice cream is cold while

Original: [Figure]

Original: [Figure] lemon guitar rabbit coffee aircraft eraser ski mask broccoli [Figure] clock is not natural , cake can be eaten while eraser can not be cooked .
Revised: For example, ice cream is cold while clock is not natural , cake can be eaten while eraser can not be cooked .

Original: These are in line with our common sense.
Revised: These are in line with our common sense.

Original: A.2. S TATE D ISTRIBUTION
Revised: A.3. S TATE D ISTRIBUTION

Original: Before annotating the affordances, we ﬁrst deﬁne the object states for all object categories and annotate the state affordances.
Revised: Before annotating the affordances [Figure] we ﬁrst deﬁne the object states for all object categories and annotate the state affordances.

Original: In total, we deﬁne 1,376 states for 381 object categories.
Revised: In total [Figure] we deﬁne 1,376 states for 381 object categories.

Original: The state list can be found in Appendix Sec.
Revised: The state list can be found in Appendix Sec.

Original: A.7.3.
Revised: A.11.

Revised: [Figure]

Original: And Fig.
Revised: And Fig.

Original: 17 shows the state distribution per object category.
Revised: 18 shows the state distribution per object category.

Original: [Figure]

Original: A.2. A TTRIBUTE -A FFORDANCE R ELATION
Revised: A.3. A TTRIBUTE -A FFORDANCE R ELATION

Original: We analyze the instance-level attribute-affordance relations in our knowledge base under three criteria.
Revised: We analyze the instance-level attribute-affordance relations in our knowledge base under three criteria.

Original: (1) Attribute Conditioned Affordance Probability.
Revised: (1) Attribute Conditioned Affordance Probability.

Original: It’s computed as P ( β | α ) to estimate affordance probability given an attribute.
Revised: It’s computed as P ( β | α ) to estimate affordance probability given an attribute.

Original: The range is [0,1].
Revised: The range is [0,1].

Original: (2) Attribute-Affordance Correlation.
Revised: (2) Attribute-Affordance Correlation.

Original: For all instances in our dataset, we evaluate the label correlation of each attribute-affordance pair, whose scale is in [-1,1].
Revised: For all instances in our dataset we evaluate the label correlation of each attribute pair appear our

Original: (3) Attribute-Affordance Causality .

Original: Start with the annotated causeeffect ( α − β ) labels, we count for how many times each attribute-affordance pair appear in our dataset and normalize the value by the maximum occurrences, leading to value in the range [0,1].

Original: It should be mentioned that we only annotate whether an attribute-affordance pair has explicit and key causality [Figure] but the detailed effect (positive or negative) should be referred to instance labels.
Revised: It should be mentioned that we only annotate whether an attribute-affordance pair has explicit and key causality, but the detailed effect (positive or negative) should be referred to instance labels.

Original: We visualize the samples of attribute-affordance relation matrices in Fig.
Revised: We visualize the samples of attribute-affordance relation matrices in Fig.

Original: 18, 19, 20 and observe some interesting properties of them.
Revised: and observe some interesting properties of them.

Original: They reveal some common relations, such as what between tasty and eat .
Revised: They reveal some common relations, such as what between tasty and eat .

Original: However, some of criteria suffer from data bias.
Revised: However, some of criteria suffer from data bias.

Original: For the condition matrix in Fig.
Revised: For the condition matrix in Fig.

Original: 18, it only cares about cases with positive attribute labels, which is not good in highlighting the negative relations, e
Revised: 19, it only cares about cases with positive attribute labels, which is not good in highlighting the negative [Figure]

Revised: cake lemon guitar rabbit coffee aircraft eraser ski mask broccoli cat

Revised: N u m b e r o f s t a t e s

Original: [Figure]
Revised: [Figure]

Original: g. the relation between natural and produce .
Revised: relations, e.g. the relation between natural and produce .

Original: For the former two matrices in Fig.
Revised: For the former two matrices in Fig.

Original: 18, 19,  they all point out the relation between tasty and pick , since most tasty objects are pickable food .
Revised: 19, 20, they all point out the relation between tasty and pick , since most tasty objects are pickable food .

Original: This ﬁnding is simply misled by the data bias but violate the causal graph (inferece from attribute to object category, then affordance).
Revised: This ﬁnding is simply misled by the data bias but violate the causal graph (inferece from attribute to object category, then affordance).

Original: Last, the matrix obtained from our causal annotation in Fig.
Revised: Last, the matrix obtained from our causal annotation in Fig.

Original: 20 is more sparse, clear of causality.
Revised: 21 is more sparse, clear of causality.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A.2. U NIFIED O BJECT R EPRESENTATION
Revised: A.3. U NIFIED O BJECT R EPRESENTATION

Original: To compare attribute-only and attribute-affordance representation abilities, we cluster the object instances of two similar animals (zebra and horse) with their attribute labels and attribute-affordance labels, respectively.
Revised: To compare attribute-only and attribute-affordance representation abilities, we cluster the object instances of two similar animals (zebra and horse) with their attribute labels and attribute-affordance labels, respectively.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: 21 via t-SNE (Maaten & Hinton, 2008).
Revised: 22 via t-SNE (Maaten & Hinton, 2008).

Original: With both attribute and affordance labels, zebra and horse can be better separated than attribute only.
Revised: With both attribute and affordance labels, zebra and horse can be better separated than attribute only.

Original: And attribute and affordance together can differentiate speciﬁc states well, such as riding , pulling car , etc.
Revised: And attribute and affordance together can differentiate speciﬁc states well, such as riding , pulling car , etc.

Original: [Figure]
Revised: [Figure]

Revised: p i c k

Revised: i g n i t e p r o d u c e

Revised: [Figure]

Original: A.2. D IFFERENCE BETWEEN C ATEGORY - AND I NSTANCE -L EVEL L ABELS
Revised: [Figure] [Figure] [Figure] A.3. D IFFERENCE BETWEEN C ATEGORY - AND I NSTANCE -L EVEL L ABELS

Original: We analyze the differences between category-level A, B labels and instance-level α, β labels.
Revised: We analyze the differences between category-level A, B labels and instance-level α, β labels.

Original: For each object category, we compute the average ratio of changed attribute/affordance classes during each instantiation from A to α or from B to β .
Revised: For each object category, we compute the average ratio of changed attribute/affordance classes during each instantiation from A to α or from B to β .

Original: The top-50 categories with the most signiﬁcant differences between A and α as well as B and β are reported respectively in Fig.
Revised: The top-50 categories with the most signiﬁcant differences between A and α as well as B and β are reported respectively in Fig.

Original: We ﬁnd that affordance labels change more dramatically than attribute labels during instantiations.
Revised: We ﬁnd that affordance labels change more dramatically than attribute labels during instantiations.

Original: This is because each attribute change may affect several affordances, e.g., when a common book becomes wet , we can neither open nor read it.
Revised: This is because each attribute change may affect several affordances, e [Figure] g., when a common book becomes wet , we can neither open nor read it

Original: (a) Difference between A and α labels.

Original: [Figure]
Revised: [Figure]

Original: labels.

Original: A.2. A TTRIBUTE -A FFORDANCE C AUSAL R ELATIONS
Revised: A.3. A TTRIBUTE -A FFORDANCE C AUSAL R ELATIONS

Original: We annotate all object instances’ causal relations of ﬁltered [ α p , β q ] pairs.
Revised: We annotate all object instances’ causal relations of ﬁltered [ α p , β q ] pairs.

Original: In total, 1,085 [ α p , β q ] pairs are chosen for the causality annotation, and over 2 M instance- α - β triplets are annotated.
Revised: In total, 1,085 [ α p , β q ] pairs are chosen for the causality annotation, and over 2 M instance- α - β triplets are annotated.

Original: In
Revised: In

Original: TDE evaluation (main text Sec. 5), we report the mean AP of top-300 [ α p , β q ] pairs to avoid the biased inﬂuence of very rare [ α p , β q ] pairs that include less than 35 object instances.
Revised: TDE evaluation (main text Sec. 5), we report the mean AP of top-300 [ α p , β q ] pairs to avoid the biased inﬂuence of very rare [ α p , β q ] pairs that include less than 35 object instances.

Original: A.2. D ATA P ARTITIONING
Revised: A.3. D ATA P ARTITIONING

Original: For the OCL task, our knowledge base is split into train, val, and test sets.
Revised: For the OCL task, our knowledge base is split into train, val, and test sets.

Original: The statistical details of the split are listed in Tab. 4.
Revised: The statistical details of the split are listed in Tab. 4.

Original: The image number ratio of the three sets is nearly 4:1:0.6, and the instance ratio is around 5:1:1.
Revised: The image number ratio of the three sets is nearly 4:1:0.6, and the instance ratio is around 5:1:1.

Original: [Table]
Revised: [Table]

Original: A.2. I MAGES AND I NSTANCES
Revised: A.3. I MAGES AND I NSTANCES

Original: Some additional  samples of our knowledge base are shown in Fig.
Revised: Some additional data samples of our knowledge base are shown in Fig.

Revised: 24, 25a, 25b, 26, and 27, including samples of diverse object categories with various bounding box distributions, different attributes and affordances, and human-labeled object states and obvious causal relations.

Original: We also detail the image and object instance counts in Fig.
Revised: We also detail the image and object instance counts in Fig.

Original: 24, 25, and 26.
Revised: 28, 29, and 30.

Original: A. TDE M ETRIC D ETAILS
Revised: A.3. M ORE S TATISTICS OF A NNOTATION

Revised: We divide A, B, α, β, causality annotation into multiple ﬁner-grained small sets in our pipeline.

Revised: Generally, we have 13, 19, 124, 140, 85 annotators-sets (381 totally) for A, B, α, β , causality annotation respectively.

Revised: We assign each small set to 2 annotators.

Revised: However, considering the controversial situations introduced, part of the annotation are confused cases based on their results.

Revised: In the whole process, 9.6% of A , 7.7% of B , 5.2% of α , 7.9% of β , and 13.7% of causality are confusing and re-assigned to additional annotators.

Revised: These indeterminable ones will be sent to two extra two annotators until making an agreement.

Revised: The quality of the dataset is guaranteed by a low confusion ratio and multiple reﬁning stages.

Revised: A.3. P OTENTIAL B IAS

Revised: We have considered the bias issue in the construction of our dataset. 1.

Revised: In our dataset, the existing datasets (ImageNet Deng et al.

Revised: COCO Lin et al.

Revised: (2014), aPY Farhadi et al.

Revised: (2009), SUN Xiao et al. (2010)) are open-sourced datasets and the images collected from Internet are publicly accessible too.

Revised: The dataset is constructed for only non-commercial purposes.

Revised: We will only provide URLs of these images to avoid copyright infringement.

Revised: During image collection, we choose images with general objects and are particularly careful with the images selection to avoid unsuitable content, private images, or implicit biases.

Revised: During annotation, the annotators cover different genders, ages, and ﬁelds of expertise to avoid potential annotation biases.

Revised: And they are all informed how we will use the annotations in our research.

Revised: apple bicycle bird boat cake car cat horse sandwich

Revised: A. TDE M ETRIC D ETAILS

Revised: TDE is to measure whether a model infers affordance with proper attention to the causality-related attribute.

Revised: That said, when removing the attribute, the model is expected to have large prediction difference further away from the ground truth .

Original: We detail some settings in our TDE metric here.
Revised: We detail some settings in our TDE metric here.

Original: For the TDE score:
Revised: For the TDE score:

Original: [Equation]
Revised: [Equation]

Original: we want the affordance probablity change direction is right according to the GT affordance labels.
Revised: we want the affordance probablity change direction is right according to the GT affordance labels.

Original: Concretely, for an instance with the labeled causal relation between [ α p , β q ], if the label GT β q = 1 , we expect the probability change P ( β q ) − P α p TDE ( β q ) to be larger after eliminating the effect of α p .
Revised: Concretely, for an instance with the labeled causal relation between [ α p , β q ], if the label GT β q = 1 , we expect the probability change P ( β q ) − P α p TDE ( β q ) to be larger after eliminating the effect of α p .

Original: Because without the effect of α p , the probability of β q should be contrary to the fact ( GT β q = 1 ), i.e.,
Revised: Because without the effect of α p , the probability of β q should be contrary to the fact ( GT β q = 1 ), i.e.,

Original: P α p TDE ( β q ) should be much smaller than P ( β q ) .
Revised: P α p TDE ( β q ) should be much smaller than P ( β q ) .

Original: Similarly, if its GT β q = 0 , we expect P α p TDE ( β q ) 
Revised: Similarly, if its GT β q = 0 , we expect P α p TDE ( β q ) [Figure] framed

Original: [Figure]
Revised: [Figure]

Revised: round tasty

Revised: [Figure]

Revised: [Figure]

Revised: √ touch √ walking × pull

Original: [Figure]
Revised: [Figure]

Revised: chase × lit ×

Revised: [Figure]

Revised: drive

Revised: [Figure]

Revised: rectangular × lift √ load × moving ×

Revised: [Figure]

Revised: × eat ×

Revised: [Figure]

Revised: × cook √ wash × natural ×

Revised: [Figure]

Revised: wash ×

Revised: [Figure]

Revised: ×

Revised: [Figure]

Revised: √

Revised: [Figure]

Revised: ×

Revised: [Figure]

Revised: √

Revised: [Figure]

Revised: √

Original:  should be much larger than P ( β q ) .
Revised: [Figure] √ [Figure] should be much larger than P ( β q ) .

Original: That said, P α p TDE ( β q ) − P ( β q ) should be large enough to follow the causal relation.
Revised: That said, P α p TDE ( β q ) − P ( β q ) should be large enough to follow the causal relation.

Original: Furthermore, in TDE loss, the setting follows this thought too.
Revised: Furthermore, in TDE loss, the setting follows this thought too.

Revised: apple fresh √ eat √

Revised: [Figure]

Revised: parked √ stop × bird dead × feed √

Revised: [Figure]

Revised: flip √

Original:  Moreover, in α - β -TDE, P ( α p =
Revised: [Figure] shear √ [Figure] Moreover, in α - β -TDE, P ( α p =

Original: GT α p ) means that the probabilities of predicted P ( α p ) accords with the label GT α p .
Revised: GT α p ) means that the probabilities of predicted P ( α p ) accords with the label GT α p .

Original: In detail, if GT α p = 1 , then P ( α p =
Revised: In detail, if GT α p = 1 , then P ( α p =

Original: GT α p ) =
Revised: GT α p ) =

Original: P ( α p ) ; if GT α p = 0 , then
Revised: P ( α p ) ; if GT α p = 0 , then

Original: P ( α p =
Revised: P ( α p =

Original: GT α p )
Revised: GT α p )

Original: = 1 − P ( α p ) , where P ( α p ) is the model prediction.
Revised: = 1 − P ( α p ) , where P ( α p ) is the model prediction.

Original: P ( β q = GT β q ) is similar.
Revised: P ( β q = GT β q ) is similar.

Original: That said, if GT β q = 1 , then P ( α p =
Revised: That said, if GT β q = 1 , then P ( α p =

Original: GT β q ) = P ( β q ) ; if GT β q = 0 , then P ( β q = GT β q ) =
Revised: GT β q ) = P ( β q ) ; if GT β q = 0 , then P ( β q = GT β q ) =

Original: 1 − P ( β q ) .
Revised: 1 − P ( β q ) .

Original: Finally [Figure] TDE α p β q ∗ P ( α p =
Revised: Finally, TDE α p β q ∗ P ( α p =

Original: GT α p ) ∗ P ( β q = GT β q ) indicates the uniﬁed performance of attribute, affordance recognition and causal inference.A. B ASELINE D ETAILS
Revised: GT α p ) ∗ P ( β q = GT β q ) indicates the uniﬁed performance of attribute, affordance recognition and causal inference.

Original: We introduce the details of all baselines here:

Original: [Figure]
Revised: [Figure]

Original: rectangularwideshortsharpnaturalnarrowyellowsmallfreshtastyfluffylargepuffystandingsmoothbrownmetalcurvedfurrysoftwhitewholehorizontalsolidgraydryquadrupedblackparked
Revised: rectangularwideshortsharpnaturalnarrowyellowsmallfreshtastyfluffylargepuffystandingsmoothbrownmetalcurvedfurrysoftwhitewholehorizontalsolidgraydryquadrupedblackparked

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: supporttiebuyreachcarrypushforcecheckpullcleandragshootholdstrikemoveinspect
Revised: For attribute/affordance recognition only, all methods adopt the labels to learn knowledge from the data. Moreover, in the evaluation of causal relation, only the “w/ L TDE ” adopts the causal relation labels. We hope the models can self-learn to mine and follow the intrinsic causalities. Thus, we design the TDE operation to evaluate this ability.

Revised: Some works Wang et al.

Revised: (2020a)

Revised: ; Tang et al.

Original: [Figure]
Revised: (2020a;b) also try to marry supervised deep learning and causal inference which are similar to our OCRN method. Besides, current techniques especially the deep learning methods perform unsatisfying in our OCL task, and all show poor ability to capture the causality. We think our main aim and contribution here is to reveal this important problem via our OCL dataset instead of directly addressing the task we propose. A lot of works should be done to prompt this ﬁeld which we hope our OCL can support.

Revised: A. B ASELINE D ETAILS

Original:  (1) Direct Mapping from Visual Feature (DM-V) : feeding f I into MLP-Sigmoids to predict
Revised: We introduce the details of all baselines here: (1) Direct Mapping from Visual Feature (DM-V) : feeding f I into MLP-Sigmoids to predict

Original: P α , P β .
Revised: P α , P β .

Original: Each α and β class owns customized MLP followed by LayerNorm Ba et al.
Revised: Each α and β class owns customized MLP followed by LayerNorm Ba et al.

Original: (2016) to generate class-speciﬁc feature and shares a same MLP-Sigmoid in classiﬁcation.
Revised: (2016) to generate class-speciﬁc feature and shares a same MLP-Sigmoid in classiﬁcation.

Original: (2) DM from Attribute (DM-At) : training an α classiﬁer with f I same with DM-V, but use the concatenated representation of attributes as f α to train the β classiﬁer.
Revised: (2) DM from Attribute (DM-At) : training an α classiﬁer with f I same with DM-V, but use the concatenated representation of attributes as f α to train the β classiﬁer.

Original: (3) DM from Attribute and Object (DM-AtO) : training an α classiﬁer with f I same with DM-V, but use the concatenated representation of attributes f α and objects f o to train the β classiﬁer.
Revised: (3) DM from Attribute and Object (DM-AtO) : training an α classiﬁer with f I same with DM-V, but use the concatenated representation of attributes f α and objects f o to train the β classiﬁer.

Original: (4) DM from Linguistic Representation (DM-L) : replace the input representation
Revised: (4) DM from Linguistic Representation (DM-L) : replace the input representation

Original: f I of DM-V with linguistic feature f L , which is the expectation of Bert (Devlin et al., 2018) of category names w.r.t P ( O i | I ) .
Revised: f I of DM-V with linguistic feature f L , which is the expectation of Bert (Devlin et al., 2018) of category names w.r.t P ( O i | I ) .

Original:  Multi-Modality (MM) : mapping f I to the semantic space via minimizing the distance to its f L .
Revised: (5) Multi-Modality (MM) : mapping f I to the semantic space via minimizing the distance to its f L .

Original: The multi-modal aligned f I is fed to MLP-Sigmoids to predict P α , P β .
Revised: The multi-modal aligned f I is fed to MLP-Sigmoids to predict P α , P β .

Original: (6) Markov Logic Network (MLN-GT) (Richardson & Domingos, 2006): adopt MLN to model the α − β relations following Zhu et al.(2014).
Revised: (6) Markov Logic Network (MLN-GT) (Richardson & Domingos, 2006): adopt MLN to model the α − β relations following Zhu et al.

Original: After training on OCL, we infer β with GT α to estimate its performance upper bound .
Revised: After training on OCL, we infer β with GT α to estimate its performance upper bound .

Original: (7) Linguistic Correlation (LingCorr) : measure the correlation between object and α / β classes via their Bert (Devlin et al., 2018) cosine similarity.
Revised: (7) Linguistic Correlation (LingCorr) : measure the correlation between object and α / β classes via their Bert (Devlin et al., 2018) cosine similarity.

Original: P α , P β are given by multiplying P ( O | I ) to correlation matrices.
Revised: P α , P β are given by multiplying P ( O | I ) to correlation matrices.

Original: (8) Kernelized Probabilistic Matrix Factorization (KPMF)
Revised: (8) Kernelized Probabilistic Matrix Factorization (KPMF)

Original: (Zhou et al., 2012) : calculate the Softmax normalized cosine similarity between each testing instance and all training samples as weights.
Revised: (Zhou et al., 2012) : calculate the Softmax normalized cosine similarity between each testing instance and all training samples as weights.

Original: Then P α or P β is generated as the weighted sum of GT α or β of training samples.
Revised: Then P α or P β is generated as the weighted sum of GT α or β of training samples.

Original: (9) A & B Lookup : return the expectation of category-level attribute or affordance vectors
Revised: (9) A & B Lookup : return the expectation of category-level attribute or affordance vectors

Original: A i , B
Revised: A i , B

Original: i w.r.t P
Revised: i w.r.t P

Original: ( O i | I ) .
Revised: ( O i | I ) .

Original: (10) Attention : map the A and f I into a shared latent space then use Sigmoid to convert f I into attention to generate instantialized f α followed by MLP-Sigmoids to predict α .
Revised: (10) Attention : map the A and f I into a shared latent space then use Sigmoid to convert f I into attention to generate instantialized f α followed by MLP-Sigmoids to predict α .

Original: As for β , we use the concatenated representation of f I and f α to generate attenion on B .
Revised: As for β , we use the concatenated representation of f I and f α to generate attenion on B .

Original: (11) Hierarchical Mapping (HMa) :
Revised: (11) Hierarchical Mapping (HMa) :

Original: ﬁrst mapping f
Revised: ﬁrst mapping f

Original: I to category-level attribute or affordance space by an MLP supervised by GT A or B .
Revised: I to category-level attribute or affordance space by an MLP supervised by GT A or B .

Original: Then the mapped features are fed to an MLP-Sigmoid to predict P α or P β .
Revised: Then the mapped features are fed to an MLP-Sigmoid to predict P α or P β .

Original: (12) Ngram (Lin et al., 2012):
Revised: (12) Ngram (Lin et al., 2012):

Original: Ngram is adopted to retrieve the relevance between α and β and generate an association matrix M α − β .
Revised: Ngram is adopted to retrieve the relevance between α and β and generate an association matrix M α − β .

Original: Then we multiply DM predicted P α with M α − β to estimate P β .
Revised: Then we multiply DM predicted P α with M α − β to estimate P β .

Original: Besides, the TDE calculation needs feature zero-masking to eliminate the effect of speciﬁc attributes (Tang et al., 2020).
Revised: Besides, the TDE calculation needs feature zero-masking to eliminate the effect of speciﬁc attributes (Tang et al., 2020b).

Original: Thus, some baselines (DM-V, DM-L, MM, LingCorr, KPMF, A & BLookup, HMa) cannot be evaluated with TDE.
Revised: Thus, some baselines (DM-V, DM-L, MM, LingCorr, KPMF, A & BLookup, HMa) cannot be evaluated with TDE.

Original: The other methods (DM-At, DM-AtO, Attention,
Revised: The other methods (DM-At, DM-AtO, Attention,

Original: OCRN) follow the same TDE calculation (feature masking).
Revised: OCRN) follow the same TDE calculation (feature masking).

Original: Two unique cases are Ngram and MLNGT.
Revised: Two unique cases are Ngram and MLNGT.

Original: Ngram uses attribute probabilities to infer affordance.
Revised: Ngram uses attribute probabilities to infer affordance.

Original: Thus, we randomize the speciﬁc attribute probabilities for Ngram to operate the TDE calculation.
Revised: Thus, we randomize the speciﬁc attribute probabilities for Ngram to operate the TDE calculation.

Original: And MLN-GT must use GT attribute labels to distinguish the “positive” and “negative” causes and then reason out the effect affordance.
Revised: And MLN-GT must use GT attribute labels to distinguish the “positive” and “negative” causes and then reason out the effect affordance.

Original: Thus, in TDE, we directly eliminate its corresponding attribute input.
Revised: Thus, in TDE, we directly eliminate its corresponding attribute input.

Original: A. D ETAILED R ESULT A NALYSIS A.5. D ETAILED A TTRIBUTE AND A FFORDANCE P ERFORMANCES
Revised: A. D ETAILED R ESULT A NALYSIS A.6. D ETAILED A TTRIBUTE AND A FFORDANCE P ERFORMANCES

Original: We compute and analysis the performance (AP) of OCRN on each attribute or affordance class in Fig.
Revised: We compute and analysis the performance (AP) of OCRN on each attribute or affordance class in Fig.

Original: 27 and Fig. 28, which suggest that visually abstract concepts like fake are more difﬁcult to model than concrete ones like metal , breakable .
Revised: 31 and Fig. 32, which suggest that visually abstract concepts like fake are more difﬁcult to model than concrete ones like metal , breakable .

Original: The performance of attribute classes is lower than affordance classes.
Revised: The performance of attribute classes is lower than affordance classes.

Original: This is mainly because the attributes have more diversities.
Revised: This is mainly because the attributes have more diversities.

Original: Thus the positive instances of each attribute class are less than the affordance class.[Figure] [Figure] [Figure]
Revised: Thus the positive instances of each attribute class are less than the affordance class.

Original: [Figure]

Original: [Figure]

Original: A.5. A TTRIBUTE AND A FFORDANCE R ECOGNITION G IVEN D ETECTED B OXES
Revised: A.6. A TTRIBUTE AND A FFORDANCE R ECOGNITION G IVEN D ETECTED B OXES

Original: Though OCL is a high-level concept learning task with object boxes as inputs, we can also consider object detection in evaluation for practical applications.
Revised: Though OCL is a high-level concept learning task with object boxes as inputs, we can also consider object detection in evaluation for practical applications.

Original: We adopt Swin transformer (Swin) (Liu et al., 2021) as the detector.
Revised: We adopt Swin transformer (Swin) (Liu et al., 2021) as the detector.

Original: It is pretrained on COCO (Lin et al., 2014) and ﬁnetuned on OCL train  set with GT boxes of 381 categories.
Revised: It is pretrained on COCO (Lin et al., 2014) and ﬁnetuned on OCL train A tt r i b u t e A P metal sandy fake hot curled thin rusty A ff o r d a n c e A P touch dilute stir hold push transport load swing set with GT boxes of 381 categories.

Original: On OCL test set, it achieves 22.9 AP 50 on object detection.
Revised: On OCL test set, it achieves 22.9 AP 50 on object detection.

Original: Subsequently, it will provide detected box b o for all models in inference.
Revised: Subsequently, it will provide detected box b o for all models in inference.

Original: We can consider the detection effect in the attribute and affordance recognition metric to build a more strict criterion.
Revised: We can consider the detection effect in the attribute and affordance recognition metric to build a more strict criterion.

Original: Namely, all false positive detections (IoU < 0.3 with refer to GT boxes) as the false positives of α and β recognition too.
Revised: Namely, all false positive detections (IoU < 0.3 with refer to GT boxes) as the false positives of α and β recognition too.

Original: Moreover, TDE calculation needs to construct the counterfactual of an object instance.
Revised: Moreover, TDE calculation needs to construct the counterfactual of an object instance.

Original: If the inaccurately detected object box shifts according to the GT box, it is difﬁcult to know whether the counterfactual comes from the attribute masking or visual content change, using the corresponding attribute-affordance causal relation labels of this GT box.
Revised: If the inaccurately detected object box shifts according to the GT box, it is difﬁcult to know whether the counterfactual comes from the attribute masking or visual content change, using the corresponding attribute-affordance causal relation labels of this GT box.

Original: Thus, considering the unique property of causal inference different from common recognition, here we do not report the TDE score.
Revised: Thus, considering the unique property of causal inference different from common recognition, here we do not report the TDE score.

Original: Tab. 5 shows the results given detected boxes.
Revised: Tab. 5 shows the results given detected boxes.

Original: Due to the more strict criterion and detection quality, the performances of all methods degrade greatly.
Revised: Due to the more strict criterion and detection quality, the performances of all methods degrade greatly.

Original: But OCRN still holds the superiority on two tracks.
Revised: But OCRN still holds the superiority on two tracks.

Original: [Table]
Revised: [Table]

Original: A.5. OCL-B ASED I MAGE R ETRIEVAL
Revised: A.6. OCL-B ASED I MAGE R ETRIEVAL

Original: We visualize the OCL reasoning performance by retrieval the top-score instances with OCRN.
Revised: We visualize the OCL reasoning performance by retrieval the top-score instances with OCRN.

Original: Some results are shown in Fig.
Revised: Some results are shown in Fig.

Original: The model can correctly retrieve the related images, especially on some common concepts e.g. columnar , sit .[Figure] [Figure]
Revised: The model can correctly retrieve the related images, especially on some common concepts e.g. columnar , sit .

Original: A. A PPLICATION ON H UMAN -O BJECT I NTERACTION (HOI) D ETECTION
Revised: A. A PPLICATION ON H UMAN -O BJECT I NTERACTION (HOI) D ETECTION

Original: To further verify the generalization ability, we apply OCL to Human-Object Interaction (HOI) detection and help HOI methods boost their performances.
Revised: To further verify the generalization ability, we apply OCL to Human-Object Interaction (HOI) detection and help HOI methods boost their performances.

Original: HOI depicts the actions performed upon objects by humans.
Revised: HOI depicts the actions performed upon objects by humans.

Original: Usually, an object has multi-affordance, i.e., a person can perform different actions upon it.
Revised: Usually, an object has multi-affordance, i.e., a person can perform different actions upon it.

Original: But in an image, just one or several actions/affordances are usually happening/ activated .
Revised: But in an image, just one or several actions/affordances are usually happening/ activated .

Original: Without object knowledge, previous methods (Li et al., 2019; Gao et al., 2018;
Revised: Without object knowledge, previous methods (Li et al., 2019; Gao et al., 2018;

Original: Li et al., 2020a) can ﬁnd the activated affordances from hundreds of actions (Chao et al., 2018).
Revised: Li et al., 2020a) can ﬁnd the activated affordances from hundreds of actions (Chao et al., 2018).

Original: For example, for each human-object pair in HICO-DET (Chao et al., 2018), a model has to select one or several actions from the deﬁned 116 actions.
Revised: For example, for each human-object pair in HICO-DET (Chao et al., 2018), a model has to select one or several actions from the deﬁned 116 actions.

Original: With OCL, things  are different.
Revised: With OCL, things [Figure] [Figure] are different.

Original: OCL covers many actions, so we can use OCRN to infer P β of an object to narrow the solution space.
Revised: OCL covers many actions, so we can use OCRN to infer P β of an object to narrow the solution space.

Original: Thus, we propose two ways:(
Revised: Thus, we propose two ways:

Original: ) OCL Filtering : 
Revised: (1) OCL Filtering : We use P β to narrow the action space with a threshold

Original: We use P β to narrow the action space with a threshold γ and generate P γβ . 
Revised: γ and generate P γβ . Affordances with

Original: Affordances with probabilities higher than γ are kept and others are set to zero ( γ = 0 .
Revised: probabilities higher than γ are kept and others are set to zero ( γ = 0 .

Original: Then,
Revised: Then

Original:  HOI model only needs to predict in a narrowed action space.
Revised: , HOI model only needs to predict in a narrowed action space

Original: In practice, we multiplythe prediction
Revised: In practice, we multiply

Original:  P HOI from HOI model with P γβ element-wisely to obtain the ﬁnal prediction P (cid:48)
Revised: the prediction P HOI from HOI model with P γβ element- wisely to obtain the ﬁnal prediction P (cid:48)

Original: HOI =
Revised: HOI =

Original: P HOI ∗ P γβ .
Revised: P HOI ∗ P γβ .

Original: (2) Human-as-Probe : Another more straightforward way is to predict HOI via OCL directly.
Revised: Human-as- Probe : Another more straightforward way is to predict HOI via OCL directly.

Original: We treat the human paired with the object as a probe .
Revised: We treat the human paired with the object as a probe .

Original: Assuming the human feature is f h and humanobject spatial conﬁguration feature is f sp (from Li et al.
Revised: Assuming the human feature is f h and humanobject spatial conﬁguration feature is f sp (from Li et al.

Original: ; Gao et al.
Revised: ; Gao et al.

Original: As P β indicates all possible affordances, the ongoing actions can be seen as the instantiation of P β , i.e., they are activated by the “probe” f h and f sp .
Revised: As P β indicates all possible affordances, the ongoing actions can be seen as the instantiation of P β , i.e., they are activated by the “probe” f h and f sp .

Original: So we use f h and f sp to generate an attention A h + sp via
Revised: So we use f h and f sp to generate an attention A h + sp via

Original: MLP-Sigmoid.
Revised: MLP-Sigmoid.

Original: Then we operate P β ∗ A h + sp and late fusion to get the ﬁnal prediction P (cid:48) HOI
Revised: Then we operate P β ∗ A h + sp and late fusion to get the ﬁnal prediction P (cid:48) HOI

Original: =
Revised: =

Original: ( P β ∗ A h + sp + P HOI ) / 2 .
Revised: ( P β ∗ A h + sp + P HOI ) / 2 .

Original: Concretely, we use OCRN to enhance HOI detection models (iCAN (Gao et al., 2018), TIN (Li et al., 2019), IDN (Li et al., 2020a)) on HICO-DET (Chao et al., 2018).
Revised: Concretely, we use OCRN to enhance HOI detection models (iCAN (Gao et al., 2018), TIN (Li et al., 2019), IDN (Li et al., 2020a)) on HICO-DET (Chao et al., 2018).

Original: As OCL merely contains 15 object categories in HICO-DET (Chao et al., 2018), the rest 65 object categories are unseen .
Revised: As OCL merely contains 15 object categories in HICO-DET (Chao et al., 2018), the rest 65 object categories are unseen .

Original: We embed OCRN into three HOI models according to OCL ﬁltering and Human-as-Probe (Appendix
Revised: We embed OCRN into three HOI models according to OCL ﬁltering and Human-as-Probe (Appendix

Original: Sec. A.6), and the public model checkpoints of Gao et al.
Revised: Sec. A.7), and the public model checkpoints of Gao et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2019; 2020a) are used.
Revised: (2019; 2020a) are used.

Original: The results are shown in Tab. 6.
Revised: The results are shown in Tab. 6.

Original: With OCL ﬁltering, iCAN (Gao et al., 2018), TIN (Li et al., 2019), and IDN (Li et al., 2020a) achieve a gain of mAP by 0.65%, 0.90%, and 0.77% respectively.
Revised: With OCL ﬁltering, iCAN (Gao et al., 2018), TIN (Li et al., 2019), and IDN (Li et al., 2020a) achieve a gain of mAP by 0.65%, 0.90%, and 0.77% respectively.

Original: The
Revised: The

Original: Human-as-Probe is more suitable for HOI detection and contributes a performance boost of 1.50%, 1.46%, and 0.98% to three models.
Revised: Human-as-Probe is more suitable for HOI detection and contributes a performance boost of 1.50%, 1.46%, and 0.98% to three models.

Original: These strongly verify the efﬁcacy and generalization ability of
Revised: These strongly verify the efﬁcacy and generalization ability of

Original: OCL.
Revised: OCL.

Original: [Table]
Revised: [Table]

Revised: A. C OMPARISON ON D EBIASING

Revised: The motivation of the OCRN is to follow the prior knowledge of the three levels of objects with a deep learning-based causal graph model, to pursue the object understanding beyond the common direct mapping from pixels to labels.

Revised: To avoid the bias estimation such as in the Simpson’s paradox Pearl et al.

Revised: Thus, we use intervention to deconfound the confounder category and exclusive the possible spurious bias and correlation imported bias from imbalanced object categories.

Revised: Overall, we propose our OCRN in a causal inference perspective instead of the pure classiﬁcation viewpoint, which also suits our causal graphical model well.

Revised: Similar cases are also proposed in recent works like Wang et al.

Revised: (2020a)

Revised: ; Tang et al.

Revised: (2020a;b).

Revised: Moreover, to better compare our method with the common debiasing methods, we further conduct the experiments as follows.

Revised: We regard α, β recognition as multiple independent binary classiﬁcation tasks and implement some methods introduced in Wang et al.

Revised: (2020b)

Revised: on our strong baseline DM-V to reduce bias from object categories.

Revised: We use mean bias ampliﬁcation ( Amp ) in Zhao et al.

Revised: (2017) as bias evaluation metric: small Amp means model suffers less from data category bias.

Revised: The test results are shown in Tab. 7.

Revised: The proposed OCRN has comparable or smaller bias ampliﬁcation than the variants of DM-V since our model follows the causal graph and exploits the tools of causal inference, while most methods for category bias are from the view of classiﬁcation.

Revised: [Table]

Revised: A. A BOUT S TATES

Revised: We did not use object states in our model because there is a compositional zero-shot problem for object-state pairs, i.e., there can be unseen states in real-world data.

Revised: Differently, affordances are more general.

Revised: The models explicitly incorporating object states will fail to generalize to these zero-shot states and it adds to the object category bias.

Revised: In experiments, the state supervision during training would indeed slightly improve the affordance recognition performance, since instances in the same state lies in a tight cluster in affordance label space.

Revised: But this will hurt the TDE performance greatly.

Revised: A. D ISCUSSION ABOUT C AUSALITY AND C AUSAL G RAPH

Revised: Annotating causality in the real-world is extremely difﬁcult.

Revised: In data annotation, we have met numerous ambiguities and difﬁculties to conﬁrm the “right” causal relations.

Revised: To address these challenges, we follow the following principles: (1) Firstly, we only emphasize on clear and strong causal relations via crowdsourcing, but omit the vague ones.

Revised: (2) Second, we take an object affordance-centric viewpoint to look at the possible causal relations.

Revised: (3) We would rather discard than condone the controversial situation in annotation.

Revised: (4) We only focus on the simple relations between one attribute and one affordance, instead of the very complex compositions of multiple attributes and affordances which are almost impossible to annotate.

Revised: Therefore, we ﬁnally ﬁnd that we can label a very small percentage of all edges with the whole causal graph consisting of so many nodes (category, attributes, affordances, contexts, etc.)

Revised: while keeping the quality.

Revised: Our causal graph follows the human priors from our experts and crowdsourcing annotators.

Revised: Some previous works also follow this before designing the method, such as Zhu et al.

Revised: From the viewpoint of causal discovery Pearl et al.

Revised: ; Tang et al.

Revised: (2020a;b); Wang et al.

Revised: (2020a), the above arcs (e.g., the inverted arc from attribute to category in the causal graph directed acyclic graph,

Revised: DAG) are indeed possible.

Revised: However, here, we mainly study the object concept learning problem, especially attribute and affordance learning.

Revised: Thus, from the perspective of affordance learning, we think the arcs from category to attribute and affordance is more vital and meaningful to us.

Revised: More speciﬁcally: (1) The causal graph follows the nature of human-like recognition .

Revised: Given a new object, we refer to the category level commonsense ﬁrst, then a speciﬁc instantiation.

Revised: Considering a simple case, when a human wants to ﬁnd an apple to eat, he will pick up a fresh one instead of a bad one based on the recognition of object category and attribute.

Revised: (2) Deﬁnitely, the causal graph reﬂects the data collection pipeline .

Revised: We ﬁrst select object categories before collecting images and complementing instances.

Revised: After that, category-level A, B and instance-level α, β are annotated.

Revised: (3) Different arcs are mainly attributed to task context , e.g., we have “image to label” in image recognition task, but “label to image” in image generation task.

Revised: In OCL, the taxonomy viewpoint (how we deﬁne an object due to its attribute or affordance, e.g., a container is a container because it can be used to hold or transport something) lies out of the area of our concern.

Revised: Instead, we consider more on the physical side of learning the property and functionality of an object for intelligent agent/robot .

Revised: It will have a larger impact on boosting downstream tasks like robot manipulation.

Original: A. D ETAILED L ISTS
Revised: A. D ETAILED L ISTS

Original: The detailed object categories, states, attributes, and affordances are listed as follows.
Revised: The detailed object categories, states, attributes, and affordances are listed as follows.

Original: [Equation] abaya, academic gown, accordion, acorn, aircraft, alpaca, ambulance, apple, armadillo, artichoke, axe, backpack, bagel, baked goods, balance beam, ball, balloon, banana, banjo, barbell, barrel, bathtub, beach wagon, beaker, bear, bed, bee, bell pepper, bench, bib, bicycle, billiard table, binoculars, bird, bison, boat, bookcase, boot, bottle, bottle opener, box, brace, brassiere, bread, broccoli, broom, bubble, buckeye, bull, burrito, bus, butterﬂy, cabbage, cake, camel, camera, can opener, candle, cannon, car, cardigan, carnivore, carrot, cart, cat, cattle, cavy, cello, chain, chainsaw, chair, cheetah, chest of drawers, chicken, chime, clock, clog, closet, coat, cocktail shaker, coffee, computer keyboard, computer mouse, conch, container, corn, couch, cowboy hat, crab, crocodile, crossword puzzle, crutch, curtain, custard apple, desk, dessert, dhole, diaper, digital clock, dinosaur, dishwasher,  chair, rugby ball, safety pin, sandal, sandwich, sarong, saxophone, scarf, scoreboard, scorpion, screw, screwdriver, sculpture, sea lion, seat belt, sewing machine, shark, sheep, shield, shirt, shoji, shower, shower cap, ski, ski mask, sleeping bag, slide rule, snail, snake, snorkel, snowmobile, snowplow, soap dispenser, sock, solar dish, sombrero, spatula, spider, squash, squirrel, starﬁsh, steel drum, stethoscope, strawberry, stretcher, submarine, suit, sunglasses, sunscreen, swan, swimwear, swing, syringe, tank, taxi, teapot, teddy bear, telephone, tennis ball, tent, tick, tie, timber wolf, toaster, toilet paper, tool, torch, towel, tower, trafﬁc light, train, tripod, trombone, truck, trumpet, turnstile, turtle, umbrella, unicycle, vase, vegetable, vehicle, vending machine, vestment, violin, volleyball, vulture, wafﬂe iron, wall clock, wallaby, wallet, wardrobe, warthog, washing machine, watch, weapon, whale, wheel, whistle, wild boar, windmill, window shade, wine, wok, wooden spoon, wool, worm, yawl, yurt, zebra.
Revised: abaya, academic gown, accordion, acorn, aircraft, alpaca, ambulance, apple, armadillo, artichoke, axe, backpack, bagel, baked goods, balance beam, ball, balloon, banana, banjo, barbell, barrel, bathtub, beach wagon, beaker, bear, bed, bee, bell pepper, bench, bib, bicycle, billiard table, binoculars, bird, bison, boat, bookcase, boot, bottle, bottle opener, box, brace, brassiere, bread, broccoli, broom, bubble, buckeye, bull, burrito, bus, butterﬂy, cabbage, cake, camel, camera, can opener, candle, cannon, car, cardigan, carnivore, carrot, cart, cat, cattle, cavy, cello, chain, chainsaw, chair, cheetah, chest of drawers, chicken, chime, clock, clog, closet, coat, cocktail shaker, coffee, computer keyboard, computer mouse, conch, container, corn, couch, cowboy hat, crab, crocodile, crossword puzzle, crutch, curtain, custard apple, desk, dessert, dhole, diaper, digital clock, dinosaur, dishwasher, dog, door, dough, dragonﬂy, drill, drum, drumstick, duck, dugong, dumbbell, eagle, eggnog, elephant, envelope, eraser, face powder, ﬁling cabinet, ﬁsh, ﬂower, ﬂowerpot, ﬂute, football helmet, footwear, fountain, fox, french horn, frog, frying pan, furniture, gazelle, giraffe, glove, goat, goblet, golf ball, golf cart, goose, gown, guacamole, guitar, hair dryer, hair slide, hair spray, hamburger, hammer, hamster, handgun, handkerchief, harmonica, harp, hartebeest, hat, hay, heater, hedgehog, helmet, hen, hippopotamus, home appliance, honeycomb, horizontal bar, horse, hot dog, hyena, ibex, ice cream, impala, insect, invertebrate, ipod, isopod, jaguar, jeans, jellyﬁsh, jinrikisha, joystick, jug, kimono, kitchen utensil, knee pad, knife, knot, koala, lab coat, ladle, lamp, lampshade, lantern, laptop, lemon, letter opener, lighthouse, limousine, lion, lipstick, lizard, loudspeaker, loupe, madagascar cat, mailbox, maracas, marimba, mask, matchstick, measuring cup, meat loaf, megalith, microphone, microwave oven, military uniform, minibus, miniskirt, mirror, missile, mixing bowl, mobile home, mobile phone, modem, mongoose, monkey, mortarboard, motorcycle, mug, mushroom, nail, necklace, nipple, oar, oboe, ocarina, oil ﬁlter, orange, otter, oven, owl, oxygen mask, packet, paintbrush, panda, panpipe, paper towel, parachute, parking meter, pen, pencil case, pencil sharpener, penguin, petri dish, piano, picket fence, pig, pillow, pinwheel, pizza, plastic bag, plate, polar bear, pomegranate, porch, potato, pretzel, printer, rabbit, radio telescope, rain barrel, red panda, reel, refrigerator, remote control, riﬂe, rocking chair, rugby ball, safety pin, sandal, sandwich, sarong, saxophone, scarf, scoreboard, scorpion, screw, screwdriver, sculpture, sea lion, seat belt, sewing machine, shark, sheep, shield, shirt, shoji, shower, shower cap, ski, ski mask, sleeping bag, slide rule, snail, snake, snorkel, snowmobile, snowplow, soap dispenser, sock, solar dish, sombrero, spatula, spider, squash, squirrel, starﬁsh, steel drum, stethoscope, strawberry, stretcher, submarine, suit, sunglasses, sunscreen, swan, swimwear, swing, syringe, tank, taxi, teapot, teddy bear, telephone, tennis ball, tent, tick, tie, timber wolf, toaster, toilet paper, tool, torch, towel, tower, trafﬁc light, train, tripod, trombone, truck, trumpet, turnstile, turtle, umbrella, unicycle, vase, vegetable, vehicle, vending machine, vestment, violin, volleyball, vulture, wafﬂe iron, wall clock, wallaby, wallet, wardrobe, warthog, washing machine, watch, weapon, whale, wheel, whistle, wild boar, windmill, window shade, wine, wok, wooden spoon, wool, worm, yawl, yurt, zebra.

Original: A.7. O BJECT A TTRIBUTE L IST
Revised: A.11. O BJECT A TTRIBUTE L IST

Original: crumpled, ceramic, cold, curled, furry, black, wet, orange, brown, yellow, striped, cool, gray, leather, large, wooden, small, soft, round, old, portable, ﬂuffy, hard, horn, messy, heavy, blue, purple, closed, new, red, thin, full, vertical, strong, dry, spotted, quadruped, whole, sharp, long, fake, open, toy, plastic, white, columnar, empty, ﬂat, cloth, warm, leashed, solid, smooth, worn, rectangular, bipedal, tasty, curved, pink, hot, digital, electric, fresh, horizontal, short, natural, metal, cooked, green, folded, broken, bent, sliced, thick, wide, narrow, arched, puffy, cream, stone, cement, marble, ﬂoral, glass, water, rubber, brick, sandy, plaid, paper, checkered, parked, moving, melted, lit, wearing, framed, stacked, tiled, standing, hanging, sitting, walking, sleeping, ﬂying, dead, ripe, in the picture, reﬂective, grassy, leafy, painted, rusty.
Revised: crumpled, ceramic, cold, curled, furry, black, wet, orange, brown, yellow, striped, cool, gray, leather, large, wooden, small, soft, round, old, portable, ﬂuffy, hard, horn, messy, heavy, blue, purple, closed, new, red, thin, full, vertical, strong, dry, spotted, quadruped, whole, sharp, long, fake, open, toy, plastic, white, columnar, empty, ﬂat, cloth, warm, leashed, solid, smooth, worn, rectangular, bipedal, tasty, curved, pink, hot, digital, electric, fresh, horizontal, short, natural, metal, cooked, green, folded, broken, bent, sliced, thick, wide, narrow, arched, puffy, cream, stone, cement, marble, ﬂoral, glass, water, rubber, brick, sandy, plaid, paper, checkered, parked, moving, melted, lit, wearing, framed, stacked, tiled, standing, hanging, sitting, walking, sleeping, ﬂying, dead, ripe, in the picture, reﬂective, grassy, leafy, painted, rusty.

Original: A.7. O BJECT S TATE L IST
Revised: A.11. O BJECT S TATE L IST

Original: (1) abaya : common, damaged, special material.
Revised: (1) abaya : common, damaged, special material.

Original: (2) academic gown : common, damaged, special material.
Revised: (2) academic gown : common, damaged, special material.

Original: (3) accordion : common, in picture.
Revised: (3) accordion : common, in picture.

Original: (4) acorn : common, rotten.
Revised: (4) acorn : common, rotten.

Original: (5) aircraft : broken, ﬂying, in factory, model, on ground, on show, part, toy.
Revised: (5) aircraft : broken, ﬂying, in factory, model, on ground, on show, part, toy.

Original: (6) alpaca : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking.
Revised: (6) alpaca : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking.

Original: (7) ambulance : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.
Revised: (7) ambulance : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.

Original: (8) apple : eating, fake, holding, model, picture, piece, rotten, sliced, toy, whole.
Revised: (8) apple : eating, fake, holding, model, picture, piece, rotten, sliced, toy, whole.

Original: (9) armadillo : dead, model, moving, resting, with shell.
Revised: (9) armadillo : dead, model, moving, resting, with shell.

Original: (10) artichoke : common. 
Revised: (10) artichoke : common. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: (11) axe : common, model, toy.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: (12) backpack : normal canvas.
Revised: [Table] [Table] [Table] [Table]

Revised: , fake, picture, raw, sliced, whole.

Original: (13) bagel : eating, fake,holding, picture, raw, sliced, whole.
Revised: (45) broccoli : cooked, fake, raw, rotten. (46) broom : common, fractured. (47) bubble : plastic, soap. (48) buckeye : with shell.

Original: (14) baked goods :  eating, fake, holding, picture, raw, sliced, whole.
Revised: (49) bull : caged, dead, eating, lying, model, moving, picture, toy.

Original: (15) balance beam : common. (16) ball : ﬂying large ball, ﬂying small ball, holding large ball, holding small ball, kicking, playing large ball, playing small ball. (17) balloon : ﬂying, normal landed.
Revised: (50) burrito : eating, fake, holding, picture, raw, sliced, whole.

Original: (18) banana : cooked, immature, model, on tree, peeled, picture, piece, rotten, sliced, toy, unripe.
Revised: (51) bus : broken, building, food, full, load on vehicle, model, moving, parked, picture, repairing, stained, toy, wrapped.

Original: (19) banjo : common, playing.
Revised: (52) butterﬂy : ﬂying, on hand, on some places.

Original: (20) barbell : common.
Revised: (53) cabbage : common.

Original: (21) barrel : common, full. (22) bathtub : common. (23) beach wagon : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.
Revised: (54) cake : eating, fake, picture, raw, sliced, whole.

Original: [Table] [Table] [Table] [Table]
Revised: (55) camel : sitting, standing, walking.

Original: [Table] [Table] [Table] [Table]
Revised: (56) camera : common.

Original: [Table] [Table] [Table] [Table]
Revised: (57) can opener : hidden blade, with blade, with drill.

Original: [Table] [Table] [Table] [Table]
Revised: (58) candle : common, lit.

Original: [Table] [Table] [Table] [Table]
Revised: (59) cannon : common, model.

Original: [Table] [Table] [Table] [Table]
Revised: (60) car : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.

Original: [Table] [Table] [Table] [Table]
Revised: (61) cardigan : common, damaged, special material.

Original: [Table] [Table] [Table] [Table]
Revised: (62) carnivore : caged, moving, sitting, toy.

Original: [Table] [Table] [Table] [Table]
Revised: (63) carrot : cooked, dirty, fake, rotten, sliced, whole.

Original: [Table] [Table] [Table] [Table]
Revised: (64) cart : metal, plastic, wood.

Original: [Table] [Table] [Table] [Table]
Revised: (65) cat : caged, fake, holding, lying, moving, picture, standing, toy.

Original: [Table] [Table] [Table] [Table]
Revised: (66) cattle : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking.

Original: [Table] [Table] [Table] [Table]
Revised: (67) cavy : eating, holding, playing, resting.

Original: [Table] [Table] [Table] [Table]
Revised: (68) cello : common.

Original: [Table] [Table] [Table] [Table]
Revised: (69) chain : thick, thin.

Original: [Table] [Table] [Table] [Table]
Revised: (70) chainsaw : common.

Original: [Table] [Table] [Table] [Table]
Revised: (71) chair : broken, burnable, fragile, fragile and burnable, solid.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: ) goat : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (135) goblet : empty, full.
Revised: (72) cheetah : caged, moving, sitting, toy. (73) chest of drawers : wood. (74) chicken : caged, dead, ﬂying, high place, hold, meat, model, perched, picture, swimming.

Revised: (75) chime : large, small, wood plastic.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: damaged

Revised: [Table]

Revised: - mon

Revised: [Table]

Revised: keyboard :

Revised: [Table]

Revised: [Table]

Revised: ) conch : alive,

Revised: [Table]

Revised: [Table]

Revised: : kernels, rotten, whole, with

Revised: [Table]

Revised: crab

Revised: [Table]

Revised: [Table]

Revised: fox : caged, moving, sitting, toy.

Original: (136) golf ball :building, common. 
Revised: (127) french horn : common. (128) frog : in hand, on ground, swimming. (129) frying pan : common, hold by human, using. (130) furniture : common, using.

Revised: (131) gazelle : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking.

Revised: (132) giraffe : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking.

Revised: (133) glove : not wearing, wearing, weaving.

Original:  (137) golf cart : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.
Revised: (134) goat : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (135) goblet : empty, full. (136) golf ball : building, common. (137) golf cart : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.

Original: (138) goose : caged, dead, ﬂying, high place, hold, meat, model, perched, picture, swimming.
Revised: (138) goose : caged, dead, ﬂying, high place, hold, meat, model, perched, picture, swimming.

Original: (139) gown : common, damaged, special material.
Revised: (139) gown : common, damaged, special material.

Original: (140) guacamole : pure, with food.
Revised: (140) guacamole : pure, with food.

Original: (141) guitar : common, playing.
Revised: (141) guitar : common, playing.

Original: (142) hair dryer : model, place, use.
Revised: (142) hair dryer : model, place, use.

Original: (143) hair slide : hold by human, place.
Revised: (143) hair slide : hold by human, place.

Original: (144) hair spray : building, hold, place.
Revised: (144) hair spray : building, hold, place.

Original: (145) hamburger : eating, fake, picture, raw, sliced, whole.
Revised: (145) hamburger : eating, fake, picture, raw, sliced, whole.

Original: (146) hammer : common.
Revised: (146) hammer : common.

Original: (147) hamster : holding, playing, resting eating.
Revised: (147) hamster : holding, playing, resting eating.

Original: (148) handgun : common, hold by human.
Revised: (148) handgun : common, hold by human.

Original: (149) handkerchief : hold by human, placed.
Revised: (149) handkerchief : hold by human, placed.

Original: (150) harmonica : place, playing.
Revised: (150) harmonica : place, playing.

Original: (151) harp : contain something, heating.
Revised: (151) harp : place, playing.

Original: (181) knee pad : common, wearing. (182) knife : common, hold by human.
Revised: (152) hartebeest : caged, dead, ﬁghting, hugging, lying, milking, picture, playing, running, shearing, standing, statue, toy, walking. (153) hat : on hand, place, toy, wearing. (154) hay : common.

Original: (183) knot : common, noddles knot.
Revised: (155) heater : closed, running 156) hedgehog : common eating 157) helmet : place, wear

Original: (184) koala : common.
Revised: 158) hen : ground

Original: (185) lab coat : common, damaged, special material.
Revised: (161) honeycomb : common, processed food common, or sports lying, model,

Original: (186) ladle :contain something, hold by human, placed plastic or metal, placed wood.
Revised: , pulling, running standing, dog : eating, walking food jaguar wearing common jinrikisha metal, wood plastic joystick : hold by place jug : common

Original: (187) lamp : common, uninstalled, using.
Revised: (179) kimono common damaged, special material 180) kitchen utensil : broken, - mon, contain something, heating wearing hold by human common noddles : common coat common damaged, special material contain something hold human, placed plastic or metal, placed wood lamp : common, uninstalled, using.

Original: (188) lampshade : common.
Revised: (188) lampshade : common.

Original: (189) lantern : lighting, making, no light.
Revised: (189) lantern : lighting, making, no light.

Original: (190) laptop : placed, using.
Revised: (190) laptop : placed, using.

Original: (191) lemon : sliced, whole.
Revised: (191) lemon : sliced, whole.

Original: (192) letter opener : hold by human, place.
Revised: (192) letter opener : hold by human, place.

Original: (193) lighthouse : common.
Revised: (193) lighthouse : common.

Original: (194) limousine : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.
Revised: (194) limousine : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.

Original: (195) lion : caged, moving, sitting, toy.
Revised: (195) lion : caged, moving, sitting, toy.

Original: (196) lipstick : close, open, using.
Revised: (196) lipstick : close, open, using.

Original: (197) lizard : dead, model, moving, resting, with shell.
Revised: (197) lizard : dead, model, moving, resting, with shell.

Original: (198) loudspeaker : common, hold by human.
Revised: (198) loudspeaker : common, hold by human.

Original: (199) loupe : hold by human, place.
Revised: (199) loupe : hold by human, place.

Original: (200) madagascar cat : caged, fake, holding, lying, moving, picture, standing, toy.
Revised: (200) madagascar cat : caged, fake, holding, lying, moving, picture, standing, toy.

Original: (201) mailbox : steal, wooden.
Revised: (201) mailbox : steal, wooden.

Original: (202) maracas : common, using.(203) marimba : common, fake, using. (204) mask : common, wearing. (205) matchstick : common, in ﬁre, in hand, used.
Revised: (202) maracas : common, using.

Original: (206) measuring cup : ceramic glass, wood plastic.

Original: (207) meat loaf : eating, on table. (208) megalith : common. (209) microphone : common, using. (210) microwave oven : common,  using.
Revised: (203) marimba : common, fake, using.

Original: (211) military uniform : common,damaged, special material. (212) minibus : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped. (213) miniskirt : wearing.(214) mirror : rear view in car, rear view out car, street mirror.
Revised: (204) mask : common, wearing.

Original: (215) missile : launched, unlaunched. (216) mixing bowl : common. (217) mobile home : close, moving, open, stop. [Table]
Revised: (205) matchstick : common, in ﬁre, in hand, used.

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: light : common, ﬁxing.

Original: (342) train : food, model, moving, parked, picture, stained, toy.

Original: (343) tripod : common.

Original: (344) trombone : common, separated.

Original: (345) truck : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.

Original: (346) trumpet : common.

Original: (347) turnstile : common.

Original: (348) turtle : in hand, in water, on ground.

Original: (349) umbrella : common, hold by human.

Original: (350) unicycle : common, riding.

Original: (351) vase : common.

Original: (352) vegetable : common, in hand, rotten, sliced cooked, unpicked.

Original: (353) vehicle : broken, building, food, full, load on vehicle, model, moving, parked, part, picture, repairing, stained, wrapped.

Original: (354) vending machine : off, on.

Original: (355) vestment : common, damaged, special material.

Original: (356) violin : common, playing.

Original: (357) volleyball : ﬂying, holding, static.

Original: (358) vulture : ﬂying, in high place, on ground.

Original: (359) wafﬂe iron : broken, fake, part, picture, placed, using.

Original: (360) wall clock : common.

Original: (361) wallaby : eating, lying, running walking, standing.

Original: (362) wallet : common.

Original: (363) wardrobe : close, open.

Original: (364) warthog : eating, standing lying, walking running.

Original: (365) washing machine : close, in wild, open, using.

Original: (366) watch : common, wearing.

Original: (367) weapon : common, fake, ﬁring, hold by human, using.

Original: (368) whale : common, dead.

Original: (369) wheel : damaged, holding, in water, sculpture, stop, uninstalled, using, using in water, working.

Original: (370) whistle : common.

Original: (371) wild boar : eating, lying, standing, walking running.

Original: (372) windmill : common.

Original: (373) window shade : put down, taken up.

Original: (374) wine : common, fake, pouring.

Original: (375) wok : common, hold by human, using.

Original: wooden spoon : contain something, hold by human, placed wood

Original: wool

Original: : cloth, skein

Original: , toy

Original: worm : common, in water, microorganism

Original: model, moving,

Original: parked,

Original: stacked, stained

Original: close, fake, open,

Original: , lying

Original: model

Original: running

Original: standing

Original: statue

Original: wild

Original: A.7. O BJECT A FFORDANCE L IST
Revised: A.11. O BJECT A FFORDANCE L IST

Original: aim (point or cause to go (blows, weapons, or objects such as photographic equipment) towards), assemble (create by putting components or members together), bend (form a curve), bend (change direction), blast (ﬁre a shot), blow (exhale hard), board (get on board of (trains, buses, ships, aircraft, etc.)), break (become separated into pieces or fragments), brush (rub with a brush, or as if with a brush), buy (obtain by purchase; acquire by means of a ﬁnancial transaction), capture (capture as if by hunting, snaring, or trapping), carry (have with oneself; have on one’s person), catch (attract and ﬁx), catch (take hold of so as to seize or restrain or stop the motion of), chase (go after with the intent to catch), check (examine so as to determine accuracy, quality, or condition), clean (make clean by removing dirt, ﬁlth, or unwanted substances from), climb (go upward with gradual or continuous progress), close (become closed), collect (gather or collect), cook (transform and make suitable for consumption by heating), cut (separate with or as if with an instrument), decant (pour out), dilute (lessen the strength or ﬂavor of a solution or mixture), drag (pull, as against a resistance), draw (bring, take, or pull out of a container or from under a cover), drink (take in liquids), drive (operate or control a vehicle), eat (eat a meal; take a meal), eat (take in solid food), edit (cut and assemble the components of), embrace (squeeze (someone) tightly in your arms, usually with fondness), exit (move out of or depart from), feed (give food to), ﬂy (operate an airplane), function (perform as expected when applied), get (succeed in catching or seizing, especially after a chase), hit (hit against; come into sudden contact with), hit (cause to move by striking), hitch (to hook or entangle), hold (be the physical support of; carry the weight of), hold (have or hold in one’s hands or grip), hop on (get up on the back of), ignite (cause to start burning; subject to ﬁre or great heat), inject (give an injection to), inspect (look over carefully), install (set up for use), jab (poke or thrust abruptly), jump (bypass), kick (drive or propel with the foot), kick (thrash about or strike out with the feet), let go of (release, as from one’s grip), lie (be located or situated somewhere; occupy a certain position), lie (be lying, be prostrate; be in a horizontal position), lie down (assume a reclining position), lift (take hold of something and move it to a different location), load (ﬁll or place a load on), melt (reduce or cause to be reduced from a solid to a liquid state, usually by heating), move (cause to move or shift into a new position or place, both in a concrete and in an abstract sense), obstruct (block passage through), occupy (occupy the whole of), open (cause to open or to become open), open (start to operate or function or cause to start operating or functioning), operate (handle and cause to function), pack (arrange in a container), paint (apply paint to; coat with paint), pare (remove the edges from and cut down to the desired size), photograph (record on photographic ﬁlm), pick (look for and gather), plow (to break and turn over earth especially with a plow), produce (create or manufacture a man-made product), pull (apply force so as to cause motion towards the source of the motion), push (move with force, ”He pushed the table into a corner”), push (press against forcefully without moving), put (put into a certain place or abstract location), race (compete in a race), raise (raise from a lower to a higher position), reach (reach a destination, either real or abstract), read (interpret something that is written or printed), rend (tear or be torn violently), repair (restore by replacing a part or putting together what is torn or broken), ride (sit and travel on the back of animal, usually while controlling its motions), ride (be carried or travel on or in a vehicle), score (gain points in a game), shoot (hit with a missile from a weapon), shoot (send forth suddenly, intensely, swiftly), sign (mark with one’s signature; write one’s name (on)), sit (be around, often idly or without speciﬁc purpose), sit (be seated), slide (move smoothly along a surface), smoke (inhale and exhale smoke from cigarettes, cigars, pipes), spin (revolve quickly and repeatedly around one’s own axis), squeeze (press ﬁrmly), stand (hold one’s ground; maintain a position; be steadfast or upright), stand (be standing; be upright), stand (remain inactive or immobile), steer (direct the course; determine the direction of travelling), sweep (move with sweeping, effortless, gliding motions), swerve (turn sharply; change direction abruptly), swing (make a big sweeping gesture or movement), switch off (cause to stop operating by disengaging a switch), throw (propel through the air), tie (fasten or secure with a rope, string, or cord), toast (propose a toast to), touch (perceive via the tactile sense), transport (move while supporting, either in a vehicle or in one’s hands or on one’s body), turn (let (something) fall or spill from a container), turn (cause to move around or rotate), turn (change orientation or direction, also in the abstract sense), unfold (spread out or open from a closed or folded state), walk (use one’s feet to advance; advance by steps), wash (cleanse (one’s body) with soap and water), wash (clean with some chemical process), work (shape, form, or improve a material), write (record data on a computer), write (mark or trace on a surface), adjust (make correspondent or conformable), ﬂip (toss with a sharp movement so as to cause to turn over in the air), ﬂush (rinse, clean, or empty with a liquid), grind (make a grating or grinding sound by rubbing together), groom (give a neat appearance to), herd (to gather and move a group of animals), hose (water with a hose), jump (cause to jump or leap), lasso (catch with a lasso), launch (to put (a boat or ship) on the water), lick (pass the tongue over), milk (take milk from female mammals), park (place temporarily), pay (give money, usually in exchange for goods or services), pet (stroke or caress gently), point (direct into a position for use), row (propel with oars), run (cause an animal to move fast), serve (put the ball into play), shear (shear the wool from), sip (drink in sips), smell (inhale the odor of; perceive by the olfactory sense), stick (pierce or penetrate or puncture with something pointed), stir (mix or add by stirring), straddle (sit or stand astride of), text on (send a message), type on (write by means of a keyboard with types), zip (close with a zipper), swim, talk on phone, work on computer, ski, surf, skateboard, dry (remove the moisture from and make dry), ﬂy (travel through the air; be airborne), greet (express greetings upon meeting someone), hunt (pursue for food or sport (as of wild animals)), launch (propel with force), lose (fail to keep or to maintain; cease to have, either physically or in an abstract sense), prepare (educate for a future role or function), rub (scrape or rub as if to relieve itching), set up (get ready for a particular purpose or event), snog (touch with the lips or press the lips (against someone’s mouth or other body part) as an expression of love, greeting, etc.), stop (stop from happening or developing), talk (exchange thoughts; talk with), teach (impart skills or knowledge to), walk (accompany or escort), watch (see or watch), watch (look attentively), wear (have on one’s person), wield (handle effectively), tag (touch a player while he is holding the ball), wave (signal  hands or nod).
Revised: aim (point or cause to go (blows, weapons, or objects such as photographic equipment) towards), assemble (create by putting components or members together), bend (form a curve), bend (change direction), blast (ﬁre a shot), blow (exhale hard), board (get on board of (trains, buses, ships, aircraft, etc.)), break (become separated into pieces or fragments), brush (rub with a brush, or as if with a brush), buy (obtain by purchase; acquire by means of a ﬁnancial transaction), capture (capture as if by hunting, snaring, or trapping), carry (have with oneself; have on one’s person), catch (attract and ﬁx), catch (take hold of so as to seize or restrain or stop the motion of), chase (go after with the intent to catch), check (examine so as to determine accuracy, quality, or condition), clean (make clean by removing dirt, ﬁlth, or unwanted substances from), climb (go upward with gradual or continuous progress), close (become closed), collect (gather or collect), cook (transform and make suitable for consumption by heating), cut (separate with or as if with an instrument), decant (pour out), dilute (lessen the strength or ﬂavor of a solution or mixture), drag (pull, as against a resistance), draw (bring, take, or pull out of a container or from under a cover), drink (take in liquids), drive (operate or control a vehicle), eat (eat a meal; take a meal), eat (take in solid food), edit (cut and assemble the components of), embrace (squeeze (someone) tightly in your arms, usually with fondness), exit (move out of or depart from), feed (give food to), ﬂy (operate an airplane), function (perform as expected when applied), get (succeed in catching or seizing, especially after a chase), hit (hit against; come into sudden contact with), hit (cause to move by striking), hitch (to hook or entangle), hold (be the physical support of; carry the weight of), hold (have or hold in one’s hands or grip), hop on (get up on the back of), ignite (cause to start burning; subject to ﬁre or great heat), inject (give an injection to), inspect (look over carefully), install (set up for use), jab (poke or thrust abruptly), jump (bypass), kick (drive or propel with the foot), kick (thrash about or strike out with the feet), let go of (release, as from one’s grip), lie (be located or situated somewhere; occupy a certain position), lie (be lying, be prostrate; be in a horizontal position), lie down (assume a reclining position), lift (take hold of something and move it to a different location), load (ﬁll or place a load on), melt (reduce or cause to be reduced from a solid to a liquid state, usually by heating), move (cause to move or shift into a new position or place, both in a concrete and in an abstract sense), obstruct (block passage through), occupy (occupy the whole of), open (cause to open or to become open), open (start to operate or function or cause to start operating or functioning), operate (handle and cause to function), pack (arrange in a container), paint (apply paint to; coat with paint), pare (remove the edges from and cut down to the desired size), photograph (record on photographic ﬁlm), pick (look for and gather), plow (to break and turn over earth especially with a plow), produce (create or manufacture a man-made product), pull (apply force so as to cause motion towards the source of the motion), push (move with force, ”He pushed the table into a corner”), push (press against forcefully without moving), put (put into a certain place or abstract location), race (compete in a race), raise (raise from a lower to a higher position), reach (reach a destination, either real or abstract), read (interpret something that is written or printed), rend (tear or be torn violently), repair (restore by replacing a part or putting together what is torn or broken), ride (sit and travel on the back of animal, usually while controlling its motions), ride (be carried or travel on or in a vehicle), score (gain points in a game), shoot (hit with a missile from a weapon), shoot (send forth suddenly, intensely, swiftly), sign (mark with one’s signature; write one’s name (on)), sit (be around, often idly or without speciﬁc purpose), sit (be seated), slide (move smoothly along a surface), smoke (inhale and exhale smoke from cigarettes, cigars, pipes), spin (revolve quickly and repeatedly around one’s own axis), squeeze (press ﬁrmly), stand (hold one’s ground; maintain a position; be steadfast or upright), stand (be standing; be upright), stand (remain inactive or immobile), steer (direct the course; determine the direction of travelling), sweep (move with sweeping, effortless, gliding motions), swerve (turn sharply; change direction abruptly), swing (make a big sweeping gesture or movement), switch off (cause to stop operating by disengaging a switch), throw (propel through the air), tie (fasten or secure with a rope, string, or cord), toast (propose a toast to), touch (perceive via the tactile sense), transport (move while supporting, either in a vehicle or in one’s hands or on one’s body), turn (let (something) fall or spill from a container), turn (cause to move around or rotate), turn (change orientation or direction, also in the abstract sense), unfold (spread out or open from a closed or folded state), walk (use one’s feet to advance; advance by steps), wash (cleanse (one’s body) with soap and water), wash (clean with some chemical process), work (shape, form, or improve a material), write (record data on a computer), write (mark or trace on a surface), adjust (make correspondent or conformable), ﬂip (toss with a sharp movement so as to cause to turn over in the air), ﬂush (rinse, clean, or empty with a liquid), grind (make a grating or grinding sound by rubbing together), groom (give a neat appearance to), herd (to gather and move a group of animals), hose (water with a hose), jump (cause to jump or leap), lasso (catch with a lasso), launch (to put (a boat or ship) on the water), lick (pass the tongue over), milk (take milk from female mammals), park (place temporarily), pay (give money, usually in exchange for goods or services), pet (stroke or caress gently), point (direct into a position for use), row (propel with oars), run (cause an animal to move fast), serve (put the ball into play), shear (shear the wool from), sip (drink in sips), smell (inhale the odor of; perceive by the olfactory sense), stick (pierce or penetrate or puncture with something pointed), stir (mix or add by stirring), straddle (sit or stand astride of), text on (send a message), type on (write by means of a keyboard with types), zip (close with a zipper), swim, talk on phone, work on computer, ski, surf, skateboard, dry (remove the moisture from and make dry), ﬂy (travel through the air; be airborne), greet (express greetings upon meeting someone), hunt (pursue for food or sport (as of wild animals)), launch (propel with force), lose (fail to keep or to maintain; cease to have, either physically or in an abstract sense), prepare (educate for a future role or function), rub (scrape or rub as if to relieve itching), set up (get ready for a particular purpose or event), snog (touch with the lips or press the lips (against someone’s mouth or other body part) as an expression of love, greeting, etc.), stop (stop from happening or developing), talk (exchange thoughts; talk with), teach (impart skills or knowledge to), walk (accompany or escort), watch (see or watch), watch (look attentively), wear (have on one’s person), wield (handle effectively), tag (touch a player while he is holding the ball), wave (signal with the hands or nod).