Original: T OWARDS G ENERIC I NTERFACE FOR H UMAN -N EURAL N ETWORK K NOWLEDGE E XCHANGE
Revised: T OWARDS G ENERIC I NTERFACE FOR H UMAN -N EURAL N ETWORK K NOWLEDGE E XCHANGE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural Networks (NN) outperform humans in multiple domains.
Revised: Neural Networks (NN) outperform humans in multiple domains.

Original: Yet they suffer from a lack of transparency and interpretability, which hinders intuitive and effective human interactions with them.
Revised: Yet they suffer from a lack of transparency and interpretability, which hinders intuitive and effective human interactions with them.

Original: Especially when NN makes mistakes, humans can hardly locate the reason for the error, and correcting it is even harder.
Revised: Especially when NN makes mistakes, humans can hardly locate the reason for the error, and correcting it is even harder.

Original: While recent advances in explainable AI have substantially improved the explainability of NNs, effective knowledge exchange between humans and NNs is still underexplored.
Revised: While recent advances in explainable AI have substantially improved the explainability of NNs, effective knowledge exchange between humans and NNs is still underexplored.

Original: To fill this gap, we propose Human-NN-Interface (HNI), a framework using a structural representation of visual concepts as a “language” for humans and
Revised: To fill this gap, we propose Human-NN-Interface (HNI), a framework using a structural representation of visual concepts as a “language” for humans and

Original: NN to communicate, interact, and exchange knowledge.
Revised: NN to communicate, interact, and exchange knowledge.

Original: Take image classification as an example, HNI visualizes the reasoning logic of a NN with class-specific
Revised: Take image classification as an example, HNI visualizes the reasoning logic of a NN with class-specific

Original: Structural Concept Graphs (c-SCG), which are human-interpretable.
Revised: Structural Concept Graphs (c-SCG), which are human-interpretable.

Original: On the other hand, humans can effectively provide feedback and guidance to the NN by modifying the c-SCG, and transferring the knowledge back to NN through HNI.
Revised: On the other hand, humans can effectively provide feedback and guidance to the NN by modifying the c-SCG, and transferring the knowledge back to NN through HNI.

Original: We demonstrate the efficacy of HNI with image classification tasks and 3 different types of interactions: (1) Explaining the reasoning logic of NNs so humans can intuitively identify and locate errors of NN; (2) human users can correct the errors and improve NN’s performance by modifying the c-SCG and distilling the knowledge back to the original NN; (3) human users can intuitively guide NN and provide a new solution for zero-shot learning.
Revised: We demonstrate the efficacy of HNI with image classification tasks and 3 different types of interactions: (1) Explaining the reasoning logic of NNs so humans can intuitively identify and locate errors of NN; (2) human users can correct the errors and improve NN’s performance by modifying the c-SCG and distilling the knowledge back to the original NN; (3) human users can intuitively guide NN and provide a new solution for zero-shot learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: From medicine (Shen et al., 2017) to self-driving cars (Buczak & Guven, 2015), machine learning (ML) systems are increasingly ubiquitous and outperform humans in multiple tasks.
Revised: From medicine (Shen et al., 2017) to self-driving cars (Buczak & Guven, 2015), machine learning (ML) systems are increasingly ubiquitous and outperform humans in multiple tasks.

Original: Given the breadth and importance of ML applications, it is critical for humans to use modern machine learning (ML) models, such as neural networks (NN), safely and trustfully, and to avoid unpredictable mistakes (Zerilli et al., 2019).
Revised: Given the breadth and importance of ML applications, it is critical for humans to use modern machine learning (ML) models, such as neural networks (NN), safely and trustfully, and to avoid unpredictable mistakes (Zerilli et al., 2019).

Original: This requires that humans should be involved in the ML loop.
Revised: This requires that humans should be involved in the ML loop.

Original: Thus, an interface between human and NN is needed, where NN and human can easily understand, interact, and influence each other.
Revised: Thus, an interface between human and NN is needed, where NN and human can easily understand, interact, and influence each other.

Original: Especially when NN makes mistakes, human prior and domain expertise knowledge with causal inference ability and common sense may help NN achieve better performance.
Revised: Especially when NN makes mistakes, human prior and domain expertise knowledge with causal inference ability and common sense may help NN achieve better performance.

Original: However, the lack of an effective interface makes it hard for a human to locate the reason for NN’s error, not to mention correct the error.
Revised: However, the lack of an effective interface makes it hard for a human to locate the reason for NN’s error, not to mention correct the error.

Original: There are two main challenges for the interaction between humans and NN: (1) Interpretability, for humans, i.e., how to understand the reasoning logic of NN and how to correctly locate the reason of errors.
Revised: There are two main challenges for the interaction between humans and NN: (1) Interpretability, for humans, i.e., how to understand the reasoning logic of NN and how to correctly locate the reason of errors.

Original: (2) Changing NN’s logic and decision, once humans locate the error of NN, i.e., how to correct it and improve NN’s performance (Fig. 1).
Revised: (2) Changing NN’s logic and decision, once humans locate the error of NN, i.e., how to correct it and improve NN’s performance (Fig. 1).

Original: To interpret a NN, current pixel-level interpretation methods (Zhou et al., 2016; Selvaraju et al., 2017) are limited to low-level relationships.
Revised: To interpret a NN, current pixel-level interpretation methods (Zhou et al., 2016; Selvaraju et al., 2017) are limited to low-level relationships.

Original: Even recent human-intuitive concept-level explanations (Ghorbani et al., 2019a; Kim et al., 2018) do not reveal the reasoning logic of NN.
Revised: Even recent human-intuitive concept-level explanations (Ghorbani et al., 2019a; Kim et al., 2018) do not reveal the reasoning logic of NN.

Original: Visual reasoning explanation (Ge et al., 2021) mimics the reasoning logic of original NN and provide logical, easy-tounderstand explanations for final decisions, but cannot directly influence NN’s performance to achieve closed-loop interaction.
Revised: Visual reasoning explanation (Ge et al., 2021) mimics the reasoning logic of original NN and provide logical, easy-tounderstand explanations for final decisions, but cannot directly influence NN’s performance to achieve closed-loop interaction.

Original: To interact with NN, current Human-In-the-Loop ML and Interacting ML are ML-centered methods that revolve around a pipeline of re-training a model using human-curated data instead of interacting with their reasoning logic (Dellermann et al., 2021).
Revised: To interact with NN, current Human-In-the-Loop ML and Interacting ML are ML-centered methods that revolve around a pipeline of re-training a model using human-curated data instead of interacting with their reasoning logic (Dellermann et al., 2021).

Original: To solve these problems, we propose Human-NN-Interface (HNI) for knowledge exchange, with the following key contributions: (1) HNI use high-level class-specific visual concepts and their relationships to build class-specific structural concepts graph (c-SCG) for each class of interest.
Revised: To solve these problems, we propose Human-NN-Interface (HNI) for knowledge exchange, with the following key contributions: (1) HNI use high-level class-specific visual concepts and their relationships to build class-specific structural concepts graph (c-SCG) for each class of interest.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We review research areas that share similarities with our work, to position our contributions
Revised: We review research areas that share similarities with our work, to position our contributions

Original: Human-AI Interaction for Machine Learning (ML) applications aims to best combine human domain expertise and computational power of ML.
Revised: Human-AI Interaction for Machine Learning (ML) applications aims to best combine human domain expertise and computational power of ML.

Original: To satisfy this need, Human-In-the-Loop ML (HILML) (Dellermann et al., 2021) and Interactive Machine Learning (IML)
Revised: To satisfy this need, Human-In-the-Loop ML (HILML) (Dellermann et al., 2021) and Interactive Machine Learning (IML)

Original: (Ware et al., 2002) have recently emerged.
Revised: (Ware et al., 2002) have recently emerged.

Original: However, both are ML-centered methods which let humans play a “server” role around the ML process, from data production, ML modeling, to model evaluation and refinement (Maadi et al., 2021).
Revised: However, both are ML-centered methods which let humans play a “server” role around the ML process, from data production, ML modeling, to model evaluation and refinement (Maadi et al., 2021).

Original: This limits human involvement and domain expert performance.
Revised: This limits human involvement and domain expert performance.

Original: “Tell me where to look” (Li et al., 2018) uses an explainable attention map to correct segmentation errors of NN, which is thus limited to low-level relationships.
Revised: “Tell me where to look” (Li et al., 2018) uses an explainable attention map to correct segmentation errors of NN, which is thus limited to low-level relationships.

Original: User interaction was also introduced in the image generation task, Interactive Image Generation (Mittal et al., 2019) can repeatedly modify images based on modifications to the scene graph while keeping the contents generated over previous steps.
Revised: User interaction was also introduced in the image generation task, Interactive Image Generation (Mittal et al., 2019) can repeatedly modify images based on modifications to the scene graph while keeping the contents generated over previous steps.

Original: In our work, we take a step toward an interface with which human users and NN can more efficiently communicate, interact and exchange knowledge between each other, with no dependency on a given list of attributes.
Revised: In our work, we take a step toward an interface with which human users and NN can more efficiently communicate, interact and exchange knowledge between each other, with no dependency on a given list of attributes.

Original: Interpreting neural networks.
Revised: Interpreting neural networks.

Original: Research on interpretability methods for neural networks (NNs) has received more and more attention.
Revised: Research on interpretability methods for neural networks (NNs) has received more and more attention.

Original: Some try to explain NN by visualizing the correlation between each pixel of the input image and the final outputs.
Revised: Some try to explain NN by visualizing the correlation between each pixel of the input image and the final outputs.

Original: For instance, CAM (Zhou et al., 2016) and
Revised: For instance, CAM (Zhou et al., 2016) and

Original: Grad-CAM (Selvaraju et al., 2017) can generate class-specific attention maps.
Revised: Grad-CAM (Selvaraju et al., 2017) can generate class-specific attention maps.

Original: Differently, several recent works focus on more human-intuitive concept-level explanations.
Revised: Differently, several recent works focus on more human-intuitive concept-level explanations.

Original: ACE and TCAV (Ghorbani et al., 2019a; Kim et al., 2018) proposed algorithms to extract meaningful concepts from images and then produce an understandable explanation.
Revised: ACE and TCAV (Ghorbani et al., 2019a; Kim et al., 2018) proposed algorithms to extract meaningful concepts from images and then produce an understandable explanation.

Original: VRX (Ge et al., 2021) uses visual concepts to explain an
Revised: VRX (Ge et al., 2021) uses visual concepts to explain an

Original: NN’s reasoning logic.
Revised: NN’s reasoning logic.

Original: Based on this work, our HNI goes beyond this and we treat logic explanation as a foundation, allowing humans to directly modify NN’s logic.
Revised: Based on this work, our HNI goes beyond this and we treat logic explanation as a foundation, allowing humans to directly modify NN’s logic.

Original: Then HNI can transfer human’s knowledge back to the original NN to achieve a closed-loop interaction between humans and NN.
Revised: Then HNI can transfer human’s knowledge back to the original NN to achieve a closed-loop interaction between humans and NN.

Original: Graph Neural Networks (GNN) are deep learning methods that operate on graph domains, which learn to represent graph nodes, edges, or subgraphs by low-dimensional vectors (Zhou et al., 2021).
Revised: Graph Neural Networks (GNN) are deep learning methods that operate on graph domains, which learn to represent graph nodes, edges, or subgraphs by low-dimensional vectors (Zhou et al., 2021).

Original: Considering the trackable information-communication properties of GNNs, many reasoning tasks have adopted GNNs as a tool, such as VQA (Teney et al., 2017; Norcliffe-Brown et al., 2018), scene understanding (Li et al., 2017a), and semantic explanations (Ge et al., 2021).
Revised: Considering the trackable information-communication properties of GNNs, many reasoning tasks have adopted GNNs as a tool, such as VQA (Teney et al., 2017; Norcliffe-Brown et al., 2018), scene understanding (Li et al., 2017a), and semantic explanations (Ge et al., 2021).

Original: In this work, we use a
Revised: In this work, we use a

Original: GNN-based Graph Reasoning Network (GRN) as the bridge between NNs and humans.
Revised: GNN-based Graph Reasoning Network (GRN) as the bridge between NNs and humans.

Original: It takes SCG as input and transfers knowledge with NN through knowledge distillation.
Revised: It takes SCG as input and transfers knowledge with NN through knowledge distillation.

Original: Knowledge distillation can effectively learn a small student model from a large ensembled teacher model using soft targets (Hinton et al., 2015).
Revised: Knowledge distillation can effectively learn a small student model from a large ensembled teacher model using soft targets (Hinton et al., 2015).

Original: Knowledge distillation has been widely used in privileged learning (Lopez-Paz et al., 2015), adversarial defense (Papernot et al., 2016), learning with noisy data (Li et al., 2017b).
Revised: Knowledge distillation has been widely used in privileged learning (Lopez-Paz et al., 2015), adversarial defense (Papernot et al., 2016), learning with noisy data (Li et al., 2017b).

Original: In this work, we use knowledge distillation to transfer knowledge
Revised: In this work, we use knowledge distillation to transfer knowledge

Original: [Figure]
Revised: [Figure]

Original: between a NN and a GNN-based graph reasoning network (GRN) which encodes human’s prior knowledge to achieve human-NN knowledge exchange.
Revised: between a NN and a GNN-based graph reasoning network (GRN) which encodes human’s prior knowledge to achieve human-NN knowledge exchange.

Original: Zero-shot learning aims to train a classifier that can classify testing instances that belong to classes that are never seen before.
Revised: Zero-shot learning aims to train a classifier that can classify testing instances that belong to classes that are never seen before.

Original: Existing works include using the one-vs-rest solution (Verma & Rai, 2017), synthesizing pseudo instances of the unseen class (Guo et al., 2017), projecting feature space instances and semantic space prototypes into a common space (Palatucci et al., 2009), and using similar seen class as the positive instance of the unseen class (Gan et al., 2016).
Revised: Existing works include using the one-vs-rest solution (Verma & Rai, 2017), synthesizing pseudo instances of the unseen class (Guo et al., 2017), projecting feature space instances and semantic space prototypes into a common space (Palatucci et al., 2009), and using similar seen class as the positive instance of the unseen class (Gan et al., 2016).

Original: Huynh & Elhamifar (2020) conduct compositional zero-shot learning by using a feature composition framework to extract and combine features of attributes to construct fine-grained attributes for unseen classes.
Revised: Huynh & Elhamifar (2020) conduct compositional zero-shot learning by using a feature composition framework to extract and combine features of attributes to construct fine-grained attributes for unseen classes.

Original: Jia et al.
Revised: Jia et al.

Original: (2021) use active zero-shot learning which promotes human-AI teaming by actively modifying the class-attribute matrix.
Revised: (2021) use active zero-shot learning which promotes human-AI teaming by actively modifying the class-attribute matrix.

Original: However such attribute labels may not be always available in real-world scenarios.
Revised: However such attribute labels may not be always available in real-world scenarios.

Original: Here, we propose a new pipeline with HNI, which makes no assumption on the availability of attribute label, humans help build the understanding of the new class using learned visual concepts and structure.
Revised: Here, we propose a new pipeline with HNI, which makes no assumption on the availability of attribute label, humans help build the understanding of the new class using learned visual concepts and structure.

Original: Then HNI transfers the knowledge of unseen class back to NN.
Revised: Then HNI transfers the knowledge of unseen class back to NN.

Original:  H UMAN -NN I NTERFACE
Revised:  H UMAN -NN I NTERFACE

Original: Our proposed Human-NN Interface (HNI) to bridge the interaction between human and neural networks is visually summarized in Fig.
Revised: Our proposed Human-NN Interface (HNI) to bridge the interaction between human and neural networks is visually summarized in Fig.

Original: There are two main path.
Revised: There are two main path.

Original: 3. NN- TO -H UMAN
Revised: 3. NN- TO -H UMAN

Original: NN-to-human path explains the NN’s reasoning logic for each decision (instance-level explanation, see Appendix Sec.
Revised: NN-to-human path explains the NN’s reasoning logic for each decision (instance-level explanation, see Appendix Sec.

Original: A.1) and more importantly, the understanding of NN for each class, represented as class-specific Structural Concept Graph (c-SCG).
Revised: A.1) and more importantly, the understanding of NN for each class, represented as class-specific Structural Concept Graph (c-SCG).

Original: Each c-SCG is bound to one class (Fig.
Revised: Each c-SCG is bound to one class (Fig.

Original: 2 shows the c-SCG of school bus), where the nodes represent the important visual concepts that original
Revised: 2 shows the c-SCG of school bus), where the nodes represent the important visual concepts that original

Original: NN considered most important in identifying the class of interest, and edges represent the pairwise structural relationships (dependencies) between concepts.
Revised: NN considered most important in identifying the class of interest, and edges represent the pairwise structural relationships (dependencies) between concepts.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2 (top), given a trained
Revised: 2 (top), given a trained

Original: NN, there are two main steps in order to explain the reasoning logic of NN to human users: (1) Using Visual Concept Extractor (VCE) to discover representative visual concepts for each class of interest.
Revised: NN, there are two main steps in order to explain the reasoning logic of NN to human users: (1) Using Visual Concept Extractor (VCE) to discover representative visual concepts for each class of interest.

Original: The detailed procedure follows Appendix Fig. 8(a) of (Ge et al., 2021):
Revised: The detailed procedure follows Appendix Fig. 8(a) of (Ge et al., 2021):

Original: To discover concepts for each class, we collect 50 to 100 images of the class.
Revised: To discover concepts for each class, we collect 50 to 100 images of the class.

Original: We first use top-down gradient attention (Grad-Cam (Selvaraju et al., 2017)) to constrain the relevant regions for concept proposals to the foreground segments, thereby ruling out irrelevant background patterns for this class.
Revised: We first use top-down gradient attention (Grad-Cam (Selvaraju et al., 2017)) to constrain the relevant regions for concept proposals to the foreground segments, thereby ruling out irrelevant background patterns for this class.

Original: Then we follow the same workflow as the ACE paper (Ghorbani et al., 2019b): multi-resolution segmentation, feature extraction, clustering patches in latent space to obtain the concept candidates, and sorting these concept candidates based on importance score, similar to (Kim et al., 2018).
Revised: Then we follow the same workflow as the ACE paper (Ghorbani et al., 2019b): multi-resolution segmentation, feature extraction, clustering patches in latent space to obtain the concept candidates, and sorting these concept candidates based on importance score, similar to (Kim et al., 2018).

Original: After that we obtain the concept pool (each concept is represented by one mean feature vector, sorted by importance score) for each class, which will serve as a source of concept candidates when human users modify concepts (nodes) for c-SCG.
Revised: After that we obtain the concept pool (each concept is represented by one mean feature vector, sorted by importance score) for each class, which will serve as a source of concept candidates when human users modify concepts (nodes) for c-SCG.

Original: To build c-SCG that reveals the reasoning logic for each class of interest of the original NN, we select the top k ( k =4 in our experiments) important concepts and their mean feature vectors as nodes.
Revised: To build c-SCG that reveals the reasoning logic for each class of interest of the original NN, we select the top k ( k =4 in our experiments) important concepts and their mean feature vectors as nodes.

Original: Each directed edge in an SCG edge ji = ( v j , v i ) has two attributes: 1) representation of the spatial structural relationship between nodes; 2) dependency e ji (a
Revised: Each directed edge in an SCG edge ji = ( v j , v i ) has two attributes: 1) representation of the spatial structural relationship between nodes; 2) dependency e ji (a

Original: [Figure]
Revised: [Figure]

Original: trainable scalar) between concepts v i , v j .
Revised: trainable scalar) between concepts v i , v j .

Original: The edge features can reveal the underlying causal logic of interactions between visual concepts which is crucial for the final decision.
Revised: The edge features can reveal the underlying causal logic of interactions between visual concepts which is crucial for the final decision.

Original: c-SCG extract such relationship between visual concepts during training and incorporate them as edge features in c-SCG.
Revised: c-SCG extract such relationship between visual concepts during training and incorporate them as edge features in c-SCG.

Original: By showing c-SCG for each class of interest, NN-to-human path provides easy-to-understand insights for human users on the reasoning process of NN, which is also a foundation for the Human-to-NN path.
Revised: By showing c-SCG for each class of interest, NN-to-human path provides easy-to-understand insights for human users on the reasoning process of NN, which is also a foundation for the Human-to-NN path.

Original: The edges (structural relationship and dependency) are fully connected at the beginning and then, using the learning in the following step 2, we select and only keep the important edges.
Revised: The edges (structural relationship and dependency) are fully connected at the beginning and then, using the learning in the following step 2, we select and only keep the important edges.

Original: (2) Using Graph Reasoning Network (GRN) to mimic the decision-making process of original NN with knowledge distillation.
Revised: (2) Using Graph Reasoning Network (GRN) to mimic the decision-making process of original NN with knowledge distillation.

Original: As a GNN-based network, GRN takes a graph as input.
Revised: As a GNN-based network, GRN takes a graph as input.

Original: To train GRN based on the built c-SCGs (one for each of n classes of interest), we need to establish connections between training images and the c-SCGs.
Revised: To train GRN based on the built c-SCGs (one for each of n classes of interest), we need to establish connections between training images and the c-SCGs.

Original: To this end, we create, for each training image
Revised: To this end, we create, for each training image

Original: I , a set of up to n image-level structural concept graphs (I-SCGs).
Revised: I , a set of up to n image-level structural concept graphs (I-SCGs).

Original: Each I-SCG is computed from both the training image I and one of the n c-SCGs:
Revised: Each I-SCG is computed from both the training image I and one of the n c-SCGs:

Original: Given the input image I , we use multi-resolution segmentation SLIC (Achanta et al., 2012) to break the image into patches, which become the concept candidates (similar as Fig.
Revised: Given the input image I , we use multi-resolution segmentation SLIC (Achanta et al., 2012) to break the image into patches, which become the concept candidates (similar as Fig.

Original: 4 step 1).
Revised: 4 step 1).

Original: In the concept matching step (similar as Fig.
Revised: In the concept matching step (similar as Fig.

Original: 4 step 2), for each class of interest c , we match features of the segmented patches to the stored anchor representation (i.e., mean feature vectors) of top k concepts deemed important for c using a similarity metric (e.g.
Revised: 4 step 2), for each class of interest c , we match features of the segmented patches to the stored anchor representation (i.e., mean feature vectors) of top k concepts deemed important for c using a similarity metric (e.g.

Original: Euclidean distance).
Revised: Euclidean distance).

Original: When at least one of the top k concepts of class c is detected in image I , an
Revised: When at least one of the top k concepts of class c is detected in image I , an

Original: I-SCG for class c will be constructed based on the template from the c-SCG of class c .
Revised: I-SCG for class c will be constructed based on the template from the c-SCG of class c .

Original: Here I-SCG uses patch features instead of the concept anchor features as node features and calculate the edge features based on spatial relationship between detected concepts in image I .
Revised: Here I-SCG uses patch features instead of the concept anchor features as node features and calculate the edge features based on spatial relationship between detected concepts in image I .

Original: This way we can have up to n I-SCGs generated for the input image I considering all n classes of interest (More details in
Revised: This way we can have up to n I-SCGs generated for the input image I considering all n classes of interest (More details in

Original: Appendix Sec.
Revised: Appendix Sec.

Original: B.2).
Revised: B.2).

Original: GRN takes the I-SCGs as input and we use knowledge distillation to transfer the decision-making logics of the original NN to GRN (similar to Appendix Fig.
Revised: GRN takes the I-SCGs as input and we use knowledge distillation to transfer the decision-making logics of the original NN to GRN (similar to Appendix Fig.

Original: 8 (b) (Ge et al., 2021)).
Revised: 8 (b) (Ge et al., 2021)).

Original: Using SCG to effectively reveal the reasoning logic of original NN has been validated with extensive experiments in (Ge et al., 2021), in which the authors evaluated the logical consistency and faithfulness between SCG explanation and original NN.
Revised: Using SCG to effectively reveal the reasoning logic of original NN has been validated with extensive experiments in (Ge et al., 2021), in which the authors evaluated the logical consistency and faithfulness between SCG explanation and original NN.

Original: (More discussion in Appendix Sec.
Revised: (More discussion in Appendix Sec.

Original: A.2.2).
Revised: A.2.2).

Original: 3. H UMAN - TO -NN
Revised: 3. H UMAN - TO -NN

Original: Human-to-NN path transfers human’s knowledge to NN, in order to improve original NN’s performance and generalizability.
Revised: Human-to-NN path transfers human’s knowledge to NN, in order to improve original NN’s performance and generalizability.

Original: There are three main steps: (1) User modifies c-SCG: after understanding the NN’s reasoning logic with NN-to-human path, users can verify whether the decision logic is reasonable or consistent with their understandings.
Revised: There are three main steps: (1) User modifies c-SCG: after understanding the NN’s reasoning logic with NN-to-human path, users can verify whether the decision logic is reasonable or consistent with their understandings.

Original: If not, human users are able to actively correct the decision logic by updating the c-SCG (e.g., deleting a visual concept and changing the structural relationship between concepts) efficiently (about 13s for each class in our experiments).
Revised: If not, human users are able to actively correct the decision logic by updating the c-SCG (e.g., deleting a visual concept and changing the structural relationship between concepts) efficiently (about 13s for each class in our experiments).

Original: (2) To represent human-modified logic, we use the modified c-SCG as template to automatically rebuild
Revised: (2) To represent human-modified logic, we use the modified c-SCG as template to automatically rebuild

Original: I-SCGs for images and train a new Graph Reasoning Network (GRN), with ground truth image labels.
Revised: I-SCGs for images and train a new Graph Reasoning Network (GRN), with ground truth image labels.

Original: (3) To let the original NN learn human-modified logic, we propose partial knowledge distillation to transfer the logic of GRN, which has incorporated the knowledge and prior from human users, back to the original NN.
Revised: (3) To let the original NN learn human-modified logic, we propose partial knowledge distillation to transfer the logic of GRN, which has incorporated the knowledge and prior from human users, back to the original NN.

Original: We describe the three steps in details in the following subsections.
Revised: We describe the three steps in details in the following subsections.

Original: 3.2. H UMAN MODIFY C -SCG
Revised: 3.2. H UMAN MODIFY C -SCG

Original: NN’s understanding of any specific class can be shown as a single c-SCG: the nodes (visual concepts) represent the crucial visual evidence or clue for NN to identify this class; the edges encode the
Revised: NN’s understanding of any specific class can be shown as a single c-SCG: the nodes (visual concepts) represent the crucial visual evidence or clue for NN to identify this class; the edges encode the

Original: [Figure]
Revised: [Figure]

Original: structure relationships and dependency between concepts.
Revised: structure relationships and dependency between concepts.

Original: After understanding the meaning of c-SCG, human users can then intuitively make modifications to c-SCGs (e.g., removing the incorrect nodes/edges) based on their knowledge or other priors, in order to improve the NN’s performance.
Revised: After understanding the meaning of c-SCG, human users can then intuitively make modifications to c-SCGs (e.g., removing the incorrect nodes/edges) based on their knowledge or other priors, in order to improve the NN’s performance.

Original: There are two main types of modification: nodes and edges, corresponding to changing the visual concepts and the relationships between concepts respectively.
Revised: There are two main types of modification: nodes and edges, corresponding to changing the visual concepts and the relationships between concepts respectively.

Original: Node (concept) modification: human users can easily identify non-casual concepts in c-SCG.
Revised: Node (concept) modification: human users can easily identify non-casual concepts in c-SCG.

Original: Fig. 3(a) shows an example of node modification.
Revised: Fig. 3(a) shows an example of node modification.

Original: In some cases, nodes may be irrelevant to the classof-interest (e.g., a background objects always appear together with object of interest; e.g., see building window and fire truck), or not representative/unique (e.g., an object part that is common among many classes; see wheel and red dot).
Revised: In some cases, nodes may be irrelevant to the classof-interest (e.g., a background objects always appear together with object of interest; e.g., see building window and fire truck), or not representative/unique (e.g., an object part that is common among many classes; see wheel and red dot).

Original: To substitute these two concepts with more representative and discriminative ones, human users can go back to the concept pool extracted by the VCE in the
Revised: To substitute these two concepts with more representative and discriminative ones, human users can go back to the concept pool extracted by the VCE in the

Original: NN-to-human path and select better visual concepts to improve the c-SCG (Fig. 3(a)).
Revised: NN-to-human path and select better visual concepts to improve the c-SCG (Fig. 3(a)).

Original: Edge (concept relationship) modification: Edges shown by c-SCG are the important dependencies selected based on the values of e ji .
Revised: Edge (concept relationship) modification: Edges shown by c-SCG are the important dependencies selected based on the values of e ji .

Original: Humans can modify them to remove non-stable or independent relationships between concepts.
Revised: Humans can modify them to remove non-stable or independent relationships between concepts.

Original: This may happen when substantial biases exist in training, when
Revised: This may happen when substantial biases exist in training, when

Original: NN may discover stable and dependent relationships between concepts which in fact do not always hold in real-world scenarios (e.g., relative position of a tiger’s leg and tail in Fig. 3(b)).
Revised: NN may discover stable and dependent relationships between concepts which in fact do not always hold in real-world scenarios (e.g., relative position of a tiger’s leg and tail in Fig. 3(b)).

Original: Human users can simply remove this edge on the c-SCG to correct the bias.
Revised: Human users can simply remove this edge on the c-SCG to correct the bias.

Original: In practice, modifications of nodes and edges can happen simultaneously to handle more complex situations.
Revised: In practice, modifications of nodes and edges can happen simultaneously to handle more complex situations.

Original: Note that to modify the decision reasoning logic for one class, human users only need to modify the corresponding c-SCG once : e.g., they can substitute one concept with another by changing the mean vector, or add/delete edges.
Revised: Note that to modify the decision reasoning logic for one class, human users only need to modify the corresponding c-SCG once : e.g., they can substitute one concept with another by changing the mean vector, or add/delete edges.

Original: They do not need to modify image-level I-SCG for each training image.
Revised: They do not need to modify image-level I-SCG for each training image.

Original: After updating the c-SCG, our framework automatically applies this modification to all image-level ISCGs.
Revised: After updating the c-SCG, our framework automatically applies this modification to all image-level ISCGs.

Original: Based on our human study with 11 human subjects, the average time to finish one class logic modification is 13s.
Revised: Based on our human study with 11 human subjects, the average time to finish one class logic modification is 13s.

Original: Human modification of c-SCGs is the first step in the human-to-NN path, where our framework provides an intuitive way for human users convert their knowledge, common sense and priors into a description in the same language that our framework uses.
Revised: Human modification of c-SCGs is the first step in the human-to-NN path, where our framework provides an intuitive way for human users convert their knowledge, common sense and priors into a description in the same language that our framework uses.

Original: Next we will show how the knowledge of human users can be transferred back to NN in the following subsections.
Revised: Next we will show how the knowledge of human users can be transferred back to NN in the following subsections.

Original: 3.2. T RAINING G RAPH R EASONING N ETWORK (GRN) WITH H UMAN ’ S L OGIC
Revised: 3.2. T RAINING G RAPH R EASONING N ETWORK (GRN) WITH H UMAN ’ S L OGIC

Original: Typically, the set S I of classes that require human intervention is a subset of the set of all S classes ( S I ⊂ S ).
Revised: Typically, the set S I of classes that require human intervention is a subset of the set of all S classes ( S I ⊂ S ).

Original: This setting is flexible and efficient: no matter how many classes the original NN can predict (e.g., 1000 classes in ImageNet), user may only want to modify the logic of a small subset of classes in question (e.g., some vehicles are easily confused with each other).
Revised: This setting is flexible and efficient: no matter how many classes the original NN can predict (e.g., 1000 classes in ImageNet), user may only want to modify the logic of a small subset of classes in question (e.g., some vehicles are easily confused with each other).

Original: In this case, we build a
Revised: In this case, we build a

Original: GRN that targets the logics of these classes only, which is more efficient to users.
Revised: GRN that targets the logics of these classes only, which is more efficient to users.

Original: For each class of interest c ∈ S I , NN-to-human path reveals its reasoning logic c-
Revised: For each class of interest c ∈ S I , NN-to-human path reveals its reasoning logic c-

Original: SCG cNN .
Revised: SCG cNN .

Original: After human’s analysis and modification, some class may have updated c-SCGs, c-
Revised: After human’s analysis and modification, some class may have updated c-SCGs, c-

Original: SCG cH after incorporating human user’s knowledge.
Revised: SCG cH after incorporating human user’s knowledge.

Original: c-SCG as a class representation cannot produce final decisions by itsel hence we reuse the GRN from NN-to-Human path to infer a prediction.
Revised: c-SCG as a class representation cannot produce final decisions by itsel hence we reuse the GRN from NN-to-Human path to infer a prediction.

Original: Fig.
Revised: Fig.

Original: 4 shows the pipeline of training GRN with the c-SCG updated by human users.
Revised: 4 shows the pipeline of training GRN with the c-SCG updated by human users.

Original: Given an input image I , the first two steps of the processing(i.e.
Revised: Given an input image I , the first two steps of the processing(i.e.

Original: Multi-resolution segmentation and Concept match to build I-SCG) are same as the GRN training in NN-to-human path while the objective is different.
Revised: Multi-resolution segmentation and Concept match to build I-SCG) are same as the GRN training in NN-to-human path while the objective is different.

Original: Here our goal is to obtain better performance by incorporating human user knowledge.
Revised: Here our goal is to obtain better performance by incorporating human user knowledge.

Original: the matched
Revised: the matched

Original: I-SCGs go through the graph convolution backbone and MLP in GRN and finally predict the image label with cross-entropy loss as the objective function.
Revised: I-SCGs go through the graph convolution backbone and MLP in GRN and finally predict the image label with cross-entropy loss as the objective function.

Original: The trained GRN can then produce decisions based on human-corrected reasoning logic because the input I-SCGs are derived from c- SCG cH .
Revised: The trained GRN can then produce decisions based on human-corrected reasoning logic because the input I-SCGs are derived from c- SCG cH .

Original: (See
Revised: (See

Original: Appendix Sec.
Revised: Appendix Sec.

Original: B.5 for more implementation details).
Revised: B.5 for more implementation details).

Original: 3.2. T RANSFERRING R EASONING L OGIC TO NN WITH P ARTIAL K NOWLEDGE D ISTILLATION
Revised: 3.2. T RANSFERRING R EASONING L OGIC TO NN WITH P ARTIAL K NOWLEDGE D ISTILLATION

Original: The last step to is to transfer the user-updated reasoning logic and knowledge in GRN back to the original NN.
Revised: The last step to is to transfer the user-updated reasoning logic and knowledge in GRN back to the original NN.

Original: To avoid catastrophic forgetting and negative impact on the classification performance of other classes, we proposed partial knowledge distillation as knowledge transfer method.
Revised: To avoid catastrophic forgetting and negative impact on the classification performance of other classes, we proposed partial knowledge distillation as knowledge transfer method.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 5 illustrates the process of partial knowledge distillation transferring human user’s knowledge from GRN back to the original NN.
Revised: 5 illustrates the process of partial knowledge distillation transferring human user’s knowledge from GRN back to the original NN.

Original: As described in Sec.
Revised: As described in Sec.

Original: 3.2.2, modified classes S I is a subset of all class set S .
Revised: 3.2.2, modified classes S I is a subset of all class set S .

Original: For unmodified classes set S U
Revised: For unmodified classes set S U

Original: =
Revised: =

Original: S \ S
Revised: S \ S

Original: I , we want to maintain their reasoning logic while we update those of of S I in original NN.
Revised: I , we want to maintain their reasoning logic while we update those of of S I in original NN.

Original: Hence two teacher models provide soft labels together.
Revised: Hence two teacher models provide soft labels together.

Original: GRN
Revised: GRN

Original: Net T 1 provides the probabilities of modified classes, the original NN Net T 2 with fixed parameters provide the probabilities of unmodified classes.
Revised: Net T 1 provides the probabilities of modified classes, the original NN Net T 2 with fixed parameters provide the probabilities of unmodified classes.

Original: Student model Net S shares the same architecture of the original NN and is initialized with the weights of original NN.
Revised: Student model Net S shares the same architecture of the original NN and is initialized with the weights of original NN.

Original: Formally, the overall loss during partial knowledge distillation is as follows:
Revised: Formally, the overall loss during partial knowledge distillation is as follows:

Original: [Equation]
Revised: [Equation]

Original: where α and β are the weighting of the two terms during distillation.
Revised: where α and β are the weighting of the two terms during distillation.

Original: For the soft label term:
Revised: For the soft label term:

Original: [Equation]
Revised: [Equation]

Original: where ˆ p T T c denotes the probability value of class c in the combined soft label with temperature T T from two teacher models.
Revised: where ˆ p T T c denotes the probability value of class c in the combined soft label with temperature T T from two teacher models.

Original: q T S c denotes the probability value of class c in the student prediction vector with temperature T S .
Revised: q T S c denotes the probability value of class c in the student prediction vector with temperature T S .

Original: N = | S | denotes the number of classes in the original NN.
Revised: N = | S | denotes the number of classes in the original NN.

Original: For q T S c , z c denotes the logits of Net S , which is the unnormalised predictions.
Revised: For q T S c , z c denotes the logits of Net S , which is the unnormalised predictions.

Original: The combined soft label ˆ p T T is the combination of two soft labels p T T 1 and p T T 2 from two teacher models Net T 1 and Net T 2 .
Revised: The combined soft label ˆ p T T is the combination of two soft labels p T T 1 and p T T 2 from two teacher models Net T 1 and Net T 2 .

Original: p T T 2 is a vector with length N , p T T 2 ∈ R N , while p T T 1 is a vector with length n , p T T 1 ∈ R n , where n = | S I | denotes the number of classes in the GRN, which is also the number of modified classes, v 1 c and v 2 c denote the logits of the teacher models Net T 1 and Net T 2 respectively:
Revised: p T T 2 is a vector with length N , p T T 2 ∈ R N , while p T T 1 is a vector with length n , p T T 1 ∈ R n , where n = | S I | denotes the number of classes in the GRN, which is also the number of modified classes, v 1 c and v 2 c denote the logits of the teacher models Net T 1 and Net T 2 respectively:

Original: [Equation]
Revised: [Equation]

Original: in which Pr ∈ (0 , 1] .
Revised: in which Pr ∈ (0 , 1] .

Original: To obtain the combined soft label ˆ p cT T , we first compute the sum of the probability of all classes of interest in p T T 2 .
Revised: To obtain the combined soft label ˆ p cT T , we first compute the sum of the probability of all classes of interest in p T T 2 .

Original: Pr = (cid:80) p T T 2 c for all { c ∈ S I } , which represents the probability proportion of the n modified classes w.r.t.
Revised: Pr = (cid:80) p T T 2 c for all { c ∈ S I } , which represents the probability proportion of the n modified classes w.r.t.

Original: all classes N in the original NN Net T 2 .
Revised: all classes N in the original NN Net T 2 .

Original: We then replace the value of class of interest in p T T 2 with the scaled value in p T T 1 to form the combined soft label.
Revised: We then replace the value of class of interest in p T T 2 with the scaled value in p T T 1 to form the combined soft label.

Original: The prediction of teachers can be erroneous, and we use ground-truth labels as hard labels to provide stronger constraint to Net S correcting these errors from teacher models.
Revised: The prediction of teachers can be erroneous, and we use ground-truth labels as hard labels to provide stronger constraint to Net S correcting these errors from teacher models.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where g c denotes the ground truth label for class c , q 1 c is the probability value of class c in the student prediction vector under temperature 1.
Revised: where g c denotes the ground truth label for class c , q 1 c is the probability value of class c in the student prediction vector under temperature 1.

Original: Eq.
Revised: Eq.

Original: 1 transfers knowledge from GRN back to the original NN (see Appendix for more implementation details).
Revised: 1 transfers knowledge from GRN back to the original NN (see Appendix for more implementation details).

Original: To summarize, we use Graph Reasoning Network (GRN) in both the NN-to-Human path and Humanto-NN path respectively (Fig. 2) with the same structure.
Revised: To summarize, we use Graph Reasoning Network (GRN) in both the NN-to-Human path and Humanto-NN path respectively (Fig. 2) with the same structure.

Original: In the NN-to-Human path, GRN simulates the reasoning logic of the original NN with knowledge distillation (detailed workflow see Appendix
Revised: In the NN-to-Human path, GRN simulates the reasoning logic of the original NN with knowledge distillation (detailed workflow see Appendix

Original: Fig.
Revised: Fig.

Original: 8 (b)(Ge et al., 2021)).
Revised: 8 (b)(Ge et al., 2021)).

Original: In the Human-to-NN path, after human users modifying the c-SCG for classes of interest, GRN uses the modified c-SCG to automatically derive I-SCGs for each image which will be used to train the GRN using ground truth labels (Fig. 4).
Revised: In the Human-to-NN path, after human users modifying the c-SCG for classes of interest, GRN uses the modified c-SCG to automatically derive I-SCGs for each image which will be used to train the GRN using ground truth labels (Fig. 4).

Original: Once the training of GRN is done, we transfer the knowledge of GRN back to the original NN, fow which we proposed partial knowledge distillation, where the newly-trained GRN becomes the teacher model to train the original
Revised: Once the training of GRN is done, we transfer the knowledge of GRN back to the original NN, fow which we proposed partial knowledge distillation, where the newly-trained GRN becomes the teacher model to train the original

Original: NN using GRN’s outputs as soft labels (Fig. 5).
Revised: NN using GRN’s outputs as soft labels (Fig. 5).

Original: We summarize the input/output details of HNI pipeline and each module/process in Appendix Table.
Revised: We summarize the input/output details of HNI pipeline and each module/process in Appendix Table.

Original:  E XPERIMENTS ON H UMAN -NN I NTERFACE A PPLICATIONS
Revised:  E XPERIMENTS ON H UMAN -NN I NTERFACE A PPLICATIONS

Original: HNI can be used as a generic interface for knowledge exchange between human users and NN.
Revised: HNI can be used as a generic interface for knowledge exchange between human users and NN.

Original: We conduct experiments to demonstrate several applications of HNI: (1) Human users improving NN’s performance by updating the logic of important concepts (Sec. 4.1.1) and relationships between them (Sec. 4.1.2), (2) Human users guiding NN in Zero-shot Learning(Sec. 4.2).
Revised: We conduct experiments to demonstrate several applications of HNI: (1) Human users improving NN’s performance by updating the logic of important concepts (Sec. 4.1.1) and relationships between them (Sec. 4.1.2), (2) Human users guiding NN in Zero-shot Learning(Sec. 4.2).

Original: 4. H UMAN IMPROVE NN’ S PERFORMANCE WITH HNI
Revised: 4. H UMAN IMPROVE NN’ S PERFORMANCE WITH HNI

Original: Through the Human-to-NN path (Sec. 3.2), humans can modify c-SCGs with their knowledge and then transfer the knowledge back to original NN.
Revised: Through the Human-to-NN path (Sec. 3.2), humans can modify c-SCGs with their knowledge and then transfer the knowledge back to original NN.

Original: This process can improve NN’s performance when
Revised: This process can improve NN’s performance when

Original: NN can not learn generalizable and robust logic during training.
Revised: NN can not learn generalizable and robust logic during training.

Original: This is quite common especially when training data is scarce or biased.
Revised: This is quite common especially when training data is scarce or biased.

Original: Scarce data can cause distribution mismatch between the training and test set, preventing the NN from really ’understanding’ the classes-of-interest.
Revised: Scarce data can cause distribution mismatch between the training and test set, preventing the NN from really ’understanding’ the classes-of-interest.

Original: Bias in training data can misled the learner to focus on the “wrong” patterns irrelevant to task objectives.
Revised: Bias in training data can misled the learner to focus on the “wrong” patterns irrelevant to task objectives.

Original: We will show how human users can use the human-to-NN path to improve original NN’s performance.
Revised: We will show how human users can use the human-to-NN path to improve original NN’s performance.

Original: 4.1. N ODES ( CONCEPTS ) MODIFICATION .
Revised: 4.1. N ODES ( CONCEPTS ) MODIFICATION .

Original: We use a subset of the ILSVRC2012 (ImageNet) (Deng et al., 2009) to train an 20-class classifier with ResNet-18 (Test performance see Table 1), each category consists of 200 images for training and 200 images for testing.
Revised: We use a subset of the ILSVRC2012 (ImageNet) (Deng et al., 2009) to train an 20-class classifier with ResNet-18 (Test performance see Table 1), each category consists of 200 images for training and 200 images for testing.

Original: Because the accuracy of the first six vehicles (bold) is low, we ask human users to help improving their performance with our framework.
Revised: Because the accuracy of the first six vehicles (bold) is low, we ask human users to help improving their performance with our framework.

Original: We first use the NN-to-Human path of HNI to visualize the reasoning logic of the original NN.
Revised: We first use the NN-to-Human path of HNI to visualize the reasoning logic of the original NN.

Original: A human user can modify the concepts in question.
Revised: A human user can modify the concepts in question.

Original: Appendix Fig.
Revised: Appendix Fig.

Original: 11 shows the c-SCG comparison before and after modification.
Revised: 11 shows the c-SCG comparison before and after modification.

Original: We then train GRN with the updated c-SCG (Sec. 3.2.2).
Revised: We then train GRN with the updated c-SCG (Sec. 3.2.2).

Original: To transfer human logic back to the original
Revised: To transfer human logic back to the original

Original: NN, we use partial knowledge distillation (Sec. 3.2.3).
Revised: NN, we use partial knowledge distillation (Sec. 3.2.3).

Original: The test results of modified NN is shown in
Revised: The test results of modified NN is shown in

Original: Table.
Revised: Table.

Original: The results suggest that by incorporating human user inputs, the original NN becomes more robust and generalizes better on the test set.
Revised: The results suggest that by incorporating human user inputs, the original NN becomes more robust and generalizes better on the test set.

Original: We also conducted a larger experiments of node and edge modification with 120 ImageNet classes (Appendix Sec. D).
Revised: We also conducted a larger experiments of node and edge modification with 120 ImageNet classes (Appendix Sec. D).

Original: While the performance on the six vehicles classes improved, the performance of other classes is almost not impacted, which shows
Revised: While the performance on the six vehicles classes improved, the performance of other classes is almost not impacted, which shows

Original: HNI can precisely modify the reasoning logic of the target classes while preserving the reasoning
Revised: HNI can precisely modify the reasoning logic of the target classes while preserving the reasoning

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: logic of others.
Revised: logic of others.

Original: This makes more flexible and
Revised: This makes more flexible and

Original: Baseline methods .
Revised: Baseline methods .

Original: We design the following methods as our baselines: (1) To evaluate how much human intervention boosts the NN performance, we conduct partial knowledge distillation without human intervention by directly supplying unmodified c-SCG to the human-to-NN path.
Revised: We design the following methods as our baselines: (1) To evaluate how much human intervention boosts the NN performance, we conduct partial knowledge distillation without human intervention by directly supplying unmodified c-SCG to the human-to-NN path.

Original: (2) based on the time it takes for human users to modify class-specific SCG (13s/class), we asked volunteers to annotate additional images with similar effort (30 additional labeled images for each of theclasses) and evaluate the performance.
Revised: (2) based on the time it takes for human users to modify class-specific SCG (13s/class), we asked volunteers to annotate additional images with similar effort (30 additional labeled images for each of theclasses) and evaluate the performance.

Original: All baseline results are shown in Table.
Revised: All baseline results are shown in Table.

Original: 2 (we only compare the modified 6 vehicle class; unmodified class have similar results before and after the modification).
Revised: 2 (we only compare the modified 6 vehicle class; unmodified class have similar results before and after the modification).

Original: 4.1. E DGES ( CONCEPTS RELATIONSHIP ) MODIFICATION .
Revised: 4.1. E DGES ( CONCEPTS RELATIONSHIP ) MODIFICATION .

Original: We conduct an edges (concept relationships) modification experiment on the iLab-20M (Borji et al., 2016) dataset, which has a controllable pose to introduce bias on concept relationships.
Revised: We conduct an edges (concept relationships) modification experiment on the iLab-20M (Borji et al., 2016) dataset, which has a controllable pose to introduce bias on concept relationships.

Original: iLab-20M contains images of toy vehicles placed on a turntable using 11 cameras at different viewpoints.
Revised: iLab-20M contains images of toy vehicles placed on a turntable using 11 cameras at different viewpoints.

Original: We tailor a subset of iLab-20M to train a three-class vehicle classifier with ResNet-18: bus, military, and tank.
Revised: We tailor a subset of iLab-20M to train a three-class vehicle classifier with ResNet-18: bus, military, and tank.

Original: In the training set, each class has 400 images.
Revised: In the training set, each class has 400 images.

Original: We manually introduce biases in pose of each class: all buses are with pose 1, all military vehicles are with pose 2 and all tanks are with pose(Fig.
Revised: We manually introduce biases in pose of each class: all buses are with pose 1, all military vehicles are with pose 2 and all tanks are with pose(Fig.

Original: 6 (a)).
Revised: 6 (a)).

Original: We construct an unbiased test set where each kind of vehicle has all the 3 poses.
Revised: We construct an unbiased test set where each kind of vehicle has all the 3 poses.

Original: After training, we use the NN-to-human path to reveal the reasoning logic of original NN to humans by visualizing the c-SCG and explaining the reasoning logic on misclassified images using I-SCG (see Appendix Sec.
Revised: After training, we use the NN-to-human path to reveal the reasoning logic of original NN to humans by visualizing the c-SCG and explaining the reasoning logic on misclassified images using I-SCG (see Appendix Sec.

Original: A.1 for instance-wise explanation)).
Revised: A.1 for instance-wise explanation)).

Original: Human users found that important concepts in c-SCG are mostly in the foreground and are consistent with huamen user’s own understanding of the class of interest.
Revised: Human users found that important concepts in c-SCG are mostly in the foreground and are consistent with huamen user’s own understanding of the class of interest.

Original: Explanation of incorrectly predicted images also shows similar results that most of the detected visual concepts had a positive contribution to the correct class.
Revised: Explanation of incorrectly predicted images also shows similar results that most of the detected visual concepts had a positive contribution to the correct class.

Original: Thus, no node modification was needed.
Revised: Thus, no node modification was needed.

Original: However, the structural relationship between concepts contributed mostly negatively, which caused incorrect predictions (Fig.
Revised: However, the structural relationship between concepts contributed mostly negatively, which caused incorrect predictions (Fig.

Original: 6 (b)).
Revised: 6 (b)).

Original: To eliminate the unstable/independent concept relationship, we delete all edges in SCG and train a GRN with the new c-SCG.
Revised: To eliminate the unstable/independent concept relationship, we delete all edges in SCG and train a GRN with the new c-SCG.

Original: The new
Revised: The new

Original: GRN focus only on the existence of visual concepts during decision making, while it does not pay attention to the relationship between them.
Revised: GRN focus only on the existence of visual concepts during decision making, while it does not pay attention to the relationship between them.

Original: Although in general deleting all edges may be too extreme, in this case it is one reasonable way to correct the problem as most errors come from pose bias in the training set, and edge features in SCG relies heavily on the pose of target objects in this case.
Revised: Although in general deleting all edges may be too extreme, in this case it is one reasonable way to correct the problem as most errors come from pose bias in the training set, and edge features in SCG relies heavily on the pose of target objects in this case.

Original: After the modification, we use partial knowledge distillation to modify the decision logic of NN.
Revised: After the modification, we use partial knowledge distillation to modify the decision logic of NN.

Original: The final performance of the modified NN shows improvement compared with the original NN (Fig. 6(c)), demonstrating that HNI can help human to improve original NN’s performance with an intuitive interface and effective mechanism in modifying the concept relationship and exchanging knowledge.
Revised: The final performance of the modified NN shows improvement compared with the original NN (Fig. 6(c)), demonstrating that HNI can help human to improve original NN’s performance with an intuitive interface and effective mechanism in modifying the concept relationship and exchanging knowledge.

Original: (More details in Appendix).
Revised: (More details in Appendix).

Original: In other applications, humans may modify nodes and edges simultaneously to improve the performance of NN.
Revised: In other applications, humans may modify nodes and edges simultaneously to improve the performance of NN.

Original: To evaluate the user feedback of HNI.
Revised: To evaluate the user feedback of HNI.

Original: We conducted a human user study (in Appx.
Revised: We conducted a human user study (in Appx.

Original: Sec. F) with responses from ML practitioners and students.
Revised: Sec. F) with responses from ML practitioners and students.

Original: 4. Z ERO - SHOT LEARNING : H UMAN TEACH NN TO LEARN NEW OBJECT THROUGH HNI
Revised: 4. Z ERO - SHOT LEARNING : H UMAN TEACH NN TO LEARN NEW OBJECT THROUGH HNI

Original: Zero-shot learning (Xian et al., 2018) is a popular and challenging task, where, at test time, a learner needs to classify samples from classes not seen during training.
Revised: Zero-shot learning (Xian et al., 2018) is a popular and challenging task, where, at test time, a learner needs to classify samples from classes not seen during training.

Original: We introduce a novel zero-shot learning pipeline with the proposed Human-NN-Interface (HNI).
Revised: We introduce a novel zero-shot learning pipeline with the proposed Human-NN-Interface (HNI).

Original: The high-level idea is that the understanding of a new object can be represented as class-specific c-SCG, which consists of visual concepts (nodes) and concept relationships (edges).
Revised: The high-level idea is that the understanding of a new object can be represented as class-specific c-SCG, which consists of visual concepts (nodes) and concept relationships (edges).

Original: Our HNI allow humans to design new c-SCG for new object category, with existing primitive visual concepts (nodes) or relationships (edges) discovered from other classes.
Revised: Our HNI allow humans to design new c-SCG for new object category, with existing primitive visual concepts (nodes) or relationships (edges) discovered from other classes.

Original: The new SCG can then be distilled back to the original NN, thereby guiding the original NN to encode new object category (i.e. zero-shot learning.
Revised: The new SCG can then be distilled back to the original NN, thereby guiding the original NN to encode new object category (i.e. zero-shot learning.

Original: While learning to recognize new class, the original NN will not “forget” the old classes.
Revised: While learning to recognize new class, the original NN will not “forget” the old classes.

Original: In our experiment, each learned 8 objects A, B, C, D, E, F, G and H has 300 training images and 216 testing images, while the new object I, J, K has only 216 testing images.
Revised: In our experiment, each learned 8 objects A, B, C, D, E, F, G and H has 300 training images and 216 testing images, while the new object I, J, K has only 216 testing images.

Original: Please refer to Fig.
Revised: Please refer to Fig.

Original: 7 for the detailed workflow:
Revised: 7 for the detailed workflow:

Original: [Figure]
Revised: [Figure]

Original: ABCDE
Revised: ABCDE

Original: F GH
Revised: F GH

Original: IJK
Revised: IJK

Original: ABCDEFGH I J K
Revised: ABCDEFGH I J K

Original: ABC D EF GH IJK
Revised: ABC D EF GH IJK

Original: [Table]
Revised: [Table]

Original: Step 1: NN-to-Human: We train a classifier for 8 objects A to H. We use VCE to discover the visual concepts and find the shared concepts.
Revised: Step 1: NN-to-Human: We train a classifier for 8 objects A to H. We use VCE to discover the visual concepts and find the shared concepts.

Original: We match the shared visual concepts from training images and form image-level SCGs (I-SCG) for all training images of objects A to H.
Revised: We match the shared visual concepts from training images and form image-level SCGs (I-SCG) for all training images of objects A to H.

Original: Step 2: Human-to-NN: Building new c-SCG for new object E and train new GRN: It is straightforward for Human to learn about a new class as they can relate the patterns and components on the new class to those they have seen in the past.
Revised: Step 2: Human-to-NN: Building new c-SCG for new object E and train new GRN: It is straightforward for Human to learn about a new class as they can relate the patterns and components on the new class to those they have seen in the past.

Original: We try to implement a similar mechanism here in describing the new class with SCG to GRN.
Revised: We try to implement a similar mechanism here in describing the new class with SCG to GRN.

Original: We construct novel I-SCG training instances with visual concepts and relationship from learned classes, in an automated fashion.
Revised: We construct novel I-SCG training instances with visual concepts and relationship from learned classes, in an automated fashion.

Original: For instance, to form a
Revised: For instance, to form a

Original: I-SCG for new object
Revised: I-SCG for new object

Original: I, we know some of its components are overlapping with object A to H. Hence we randomly sample one I-SCG from object A and use its concept 1 as the node of concept 1 in E’s
Revised: I, we know some of its components are overlapping with object A to H. Hence we randomly sample one I-SCG from object A and use its concept 1 as the node of concept 1 in E’s

Original: I-SCG.
Revised: I-SCG.

Original: Similarly, we obtain I’s nodes of concept 2 and 3 by randomly sampling I-SCGs of objects E and C. To construct I-SCG edges for E, we sample I-SCGs of D and form the edges of I based on their similar structures.
Revised: Similarly, we obtain I’s nodes of concept 2 and 3 by randomly sampling I-SCGs of objects E and C. To construct I-SCG edges for E, we sample I-SCGs of D and form the edges of I based on their similar structures.

Original: Building I-SCG for new objects J
Revised: Building I-SCG for new objects J

Original: and K are similar.
Revised: and K are similar.

Original: We form a new I-SCG training set by adding the novel I-SCGs of I,J,K into the original training set and then train a GRN that can classify objects A to H and I, J, K.
Revised: We form a new I-SCG training set by adding the novel I-SCGs of I,J,K into the original training set and then train a GRN that can classify objects A to H and I, J, K.

Original: Step 3: Transfer knowledge from GRN back to original NN: we use knowledge distillation to transfer the knowledge about new object E from GRN back to the original NN.
Revised: Step 3: Transfer knowledge from GRN back to original NN: we use knowledge distillation to transfer the knowledge about new object E from GRN back to the original NN.

Original: In this process, we only use the images of A to H as training set, and only use the soft label form GRN without any hard label to avoid bias toward old classes.
Revised: In this process, we only use the images of A to H as training set, and only use the soft label form GRN without any hard label to avoid bias toward old classes.

Original: Step 4: NN learn the knowledge to encode new objects without forgetting the knowledge about old classes.
Revised: Step 4: NN learn the knowledge to encode new objects without forgetting the knowledge about old classes.

Original: Table.
Revised: Table.

Original: 3 shows the performance of zero-shot learning with our HNI (see Appx.
Revised: 3 shows the performance of zero-shot learning with our HNI (see Appx.

Original: for more details).
Revised: for more details).

Original: We made the OBJECT dataset which we created and used here public.
Revised: We made the OBJECT dataset which we created and used here public.

Original: (Appx.
Revised: (Appx.

Original: Sec. C).
Revised: Sec. C).

Original: Comparing with mainstream zero-shot learning settings , where attribute descriptions for the images are given (e.g., stripes, horse-like shape, big four-leg animal), our method considers a more general settings and we make no assumption on the availability of attribute labels.
Revised: Comparing with mainstream zero-shot learning settings (Geng et al. Fu et al. ; Xian et al. (2018)), where attribute descriptions for the images are given (e.g., stripes, horse-like shape, big four-leg animal), our method considers a more general settings and we make no assumption on the availability of attribute labels.

Original: Instead, we rely on the unsupervised mining of primitive concepts from the training dataset (without any attribute or concept labels).
Revised: Instead, we rely on the unsupervised mining of primitive concepts from the training dataset (without any attribute or concept labels).

Original: With different combinations of any subset of these learned primitive concepts and different structural relationships between them, we can use GCN to represent novel classes and eventually guide the NN to learn to encode them by HNI.
Revised: With different combinations of any subset of these learned primitive concepts and different structural relationships between them, we can use GCN to represent novel classes and eventually guide the NN to learn to encode them by HNI.

Original: Our method has its limitations especially when the new class can not be easily represented by the learned primitive visual concepts.
Revised: Our method has its limitations especially when the new class can not be easily represented by the learned primitive visual concepts.

Original: However, our method is flexible in the sense that it can define new structural/spatial relationships based on learned relationships.
Revised: However, our method is flexible in the sense that it can define new structural/spatial relationships based on learned relationships.

Original: We argue that this is an important strength of our method as it is more extensible and capable of encoding novel objects with fewer assumptions, i.e., not limited by the given list of attributes in describing relationships/structures of components/parts of novel objects.
Revised: We argue that this is an important strength of our method as it is more extensible and capable of encoding novel objects with fewer assumptions, i.e., not limited by the given list of attributes in describing relationships/structures of components/parts of novel objects.

Original: A PPENDIX A F URTHER DISCUSSION AND EXPLORATION OF V ISUAL R EASONING E XPLANATION (VRX) FRAMEWORK (G E ET AL ., 2021) A. I NTRODUCTION
Revised: A PPENDIX A F URTHER DISCUSSION AND EXPLORATION OF V ISUAL R EASONING E XPLANATION (VRX) FRAMEWORK (G E ET AL ., 2021) A. I NTRODUCTION

Original: Visual Reasoning Explaination (VRX) (Ge et al., 2021), which for the first time uses GNN to mimic the reasoning logic of original NN and demonstrate the effectiveness of using Structural concept graph (SCG) to provide logical, easy-to-understand explanations for final decisions.
Revised: Visual Reasoning Explaination (VRX) (Ge et al., 2021), which for the first time uses GNN to mimic the reasoning logic of original NN and demonstrate the effectiveness of using Structural concept graph (SCG) to provide logical, easy-to-understand explanations for final decisions.

Original: Our NN-to-human path is based on two important module of VRX: Visual Concept Extractor (VCE) and Graph Reasoning
Revised: Our NN-to-human path is based on two important module of VRX: Visual Concept Extractor (VCE) and Graph Reasoning

Original: Network (GRN).
Revised: Network (GRN).

Original: We recap some important figures (Fig. 8, Fig. 9, Fig. 10) in this section which may help understand the details of VRX framework (Ge et al., 2021)
Revised: We recap some important figures (Fig. 8, Fig. 9, Fig. 10) in this section which may help understand the details of VRX framework (Ge et al., 2021)

Original: [Figure]
Revised: [Figure]

Original: The Visual Concept Extractor (VCE) discovers the class-specific important visual concepts.
Revised: The Visual Concept Extractor (VCE) discovers the class-specific important visual concepts.

Original: (b) In original NN, the representation of the top N concepts is distributed throughout the network (colored discs and rectangles).
Revised: (b) In original NN, the representation of the top N concepts is distributed throughout the network (colored discs and rectangles).

Original: (c) Using Visual Concept Graphs that are specific to each image class,
Revised: (c) Using Visual Concept Graphs that are specific to each image class,

Original: our
Revised: our

Original: VRX learns the respective contributions from visual concepts and from their spatial relationships, through distillation, to explain the network’s decision.
Revised: VRX learns the respective contributions from visual concepts and from their spatial relationships, through distillation, to explain the network’s decision.

Original: (d) In this example, the concept graphs colored according to contributions from concepts and relations towards each class explain why the network decides that this input is a Jeep and not others.
Revised: (d) In this example, the concept graphs colored according to contributions from concepts and relations towards each class explain why the network decides that this input is a Jeep and not others.

Original: For instance-wise explanation , with an important module Visual Decision Interpreter (VDI), VRX can meaningfully answer “why” and “why not” questions about the prediction, providing easy-tounderstand insights about the reasoning process.
Revised: For instance-wise explanation , with an important module Visual Decision Interpreter (VDI), VRX can meaningfully answer “why” and “why not” questions about the prediction, providing easy-tounderstand insights about the reasoning process.

Original: Fig.
Revised: Fig.

Original: 8 (d) and Fig.
Revised: 8 (d) and Fig.

Original: 9 give more detailed explanation.
Revised: 9 give more detailed explanation.

Original: A. F URTHER DISCUSSION A.2. U NDERSTANDING OF R EASONING LOGIC
Revised: A. F URTHER DISCUSSION A.2. U NDERSTANDING OF R EASONING LOGIC

Original: In Visual Reasoning Explaination (VRX) paper (Ge et al., 2021), they define the Reasoning logic as using the structured visual concept as tools/languages to answer the question of why and why not (e.g., why the input image is an ambulance?
Revised: In Visual Reasoning Explaination (VRX) paper (Ge et al., 2021), they define the Reasoning logic as using the structured visual concept as tools/languages to answer the question of why and why not (e.g., why the input image is an ambulance?

Original: why not fire engine?), which is a human-friendly way to understand the decision clue and logic of original NN.
Revised: why not fire engine?), which is a human-friendly way to understand the decision clue and logic of original NN.

Original: Thus, here we use the term “reasoning” to refer to the correlations between primitive patterns and specific classes of interest learned/encoded by the original neural network.
Revised: Thus, here we use the term “reasoning” to refer to the correlations between primitive patterns and specific classes of interest learned/encoded by the original neural network.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 1          （ a ） （ b ） （ c ）
Revised: 1          （ a ） （ b ） （ c ）

Original: A.2. F IDELITY OF SCG EXPLANATION
Revised: A.2. F IDELITY OF SCG EXPLANATION

Original: In terms of the fidelity of using Structural Concept Graph (SCG) to explain the reasoning logic of original NN, here are more details: While we agree that our current pipeline does not provide a guarantee that SCG will be 100% faithful to the original NN’s reasoning logic, we would like to point out a few considerations in our design:
Revised: In terms of the fidelity of using Structural Concept Graph (SCG) to explain the reasoning logic of original NN, here are more details: While we agree that our current pipeline does not provide a guarantee that SCG will be 100% faithful to the original NN’s reasoning logic, we would like to point out a few considerations in our design:

Original: (1) The faithfulness of our SCG to the original NN depends on several factors, including the concept selection, the number of concepts used to construct the SCG, the sampling of data used in distillation from the original NN to SCG.
Revised: (1) The faithfulness of our SCG to the original NN depends on several factors, including the concept selection, the number of concepts used to construct the SCG, the sampling of data used in distillation from the original NN to SCG.

Original: Since the focus of this paper is more of a feasibility study, rather than a ready-to-deploy system, each component inevitably has errors to a certain extent.
Revised: Since the focus of this paper is more of a feasibility study, rather than a ready-to-deploy system, each component inevitably has errors to a certain extent.

Original: (2) We use top-down attention (e.g. Grad-CAM) to identify regions in the input image which contributed most to the prediction of each class-of-interest, which can limit the
Revised: (2) We use top-down attention (e.g. Grad-CAM) to identify regions in the input image which contributed most to the prediction of each class-of-interest, which can limit the

Original: region in which we conduct concepts discovery.
Revised: region in which we conduct concepts discovery.

Original: By doing this, we can filter irrelevant concepts and focus on those considered important in the original NN’s reasoning/inference.
Revised: By doing this, we can filter irrelevant concepts and focus on those considered important in the original NN’s reasoning/inference.

Original: (3) In concept selection, we use concept activation vectors (TCAV)
Revised: (3) In concept selection, we use concept activation vectors (TCAV)

Original: (Kim et al., 2018) to compute the importance score of each concept w.r.t.
Revised: (Kim et al., 2018) to compute the importance score of each concept w.r.t.

Original: the prediction of a specific class-of-interest.
Revised: the prediction of a specific class-of-interest.

Original: With sufficient sampling of data, this gradient-based method can faithfully reflect the overall correlation between each visual concept and each class-of-interest in the original NN.
Revised: With sufficient sampling of data, this gradient-based method can faithfully reflect the overall correlation between each visual concept and each class-of-interest in the original NN.

Original: (4) Knowledge distillation is widely used in transferring knowledge (encapsulating the reasoning logics) between neural networks.
Revised: (4) Knowledge distillation is widely used in transferring knowledge (encapsulating the reasoning logics) between neural networks.

Original: In other words, it is transferring the input-output mapping from one neural network to another.
Revised: In other words, it is transferring the input-output mapping from one neural network to another.

Original: In our framework, we adapted knowledge distillation to imitate such mapping (instead of mapping from the input image to output logits, we learn the mapping between the visual concepts and the output logits).
Revised: In our framework, we adapted knowledge distillation to imitate such mapping (instead of mapping from the input image to output logits, we learn the mapping between the visual concepts and the output logits).

Original: We also add perturbation during distillation by randomly masking out one of the detected visual concepts in the input image while making sure GRN and original NN produce the same outputs.
Revised: We also add perturbation during distillation by randomly masking out one of the detected visual concepts in the input image while making sure GRN and original NN produce the same outputs.

Original: Given enough training data, GRN is expected to faithfully mimic the original NN and reveal the reasoning logics.
Revised: Given enough training data, GRN is expected to faithfully mimic the original NN and reveal the reasoning logics.

Original: (5) Experimental validation.
Revised: (5) Experimental validation.

Original: The faithfulness of using Structural Concept Graph (SCG) to explain the original NN is one key contribution of Visual Reasoning Explanation (VRX) ref [2], and in the paper, the authors conducted several experiments to prove the fidelity of SCG explanation, such as logical consistency between the explanation using SCG and the original NN.
Revised: The faithfulness of using Structural Concept Graph (SCG) to explain the original NN is one key contribution of Visual Reasoning Explanation (VRX) ref [2], and in the paper, the authors conducted several experiments to prove the fidelity of SCG explanation, such as logical consistency between the explanation using SCG and the original NN.

Original: To briefly recap: 1) In Sec.4.2 of VRX paper (Ge et al., 2021), they qualitatively and quantitatively verified the logical consistency between explanation using SCG and original NN by showing that VRX can correctly locate the reason that causes the original model’s wrong prediction and successfully correct the errors with the guidance of the explanation (Table 1 of VRX paper (Ge et al., 2021) shows 114/ errors were successfully corrected).
Revised: To briefly recap: 1) In Sec.4.2 of VRX paper (Ge et al., 2021), they qualitatively and quantitatively verified the logical consistency between explanation using SCG and original NN by showing that VRX can correctly locate the reason that causes the original model’s wrong prediction and successfully correct the errors with the guidance of the explanation (Table 1 of VRX paper (Ge et al., 2021) shows 114/ errors were successfully corrected).

Original: The logical consistency verification experiment shows that the explanation with SCG is faithful to the original NN.
Revised: The logical consistency verification experiment shows that the explanation with SCG is faithful to the original NN.

Original: 2) Explanation sensitivity of visual and structural perturbations.
Revised: 2) Explanation sensitivity of visual and structural perturbations.

Original: The authors also conducted two experiments with the control variate method to verify the explanations of VRX are faithfully and sensitive to visual perturbation and structural perturbation respectively (Sec.4.3 and Figure 7 of VRX paper (Ge et al., 2021) ).
Revised: The authors also conducted two experiments with the control variate method to verify the explanations of VRX are faithfully and sensitive to visual perturbation and structural perturbation respectively (Sec.4.3 and Figure 7 of VRX paper (Ge et al., 2021) ).

Original: Based on the above-mentioned designs and experimental validations on the fidelity of SCG explanations, we argue that the explanation with SCG can be faithful to the original NN, in ideal scenarios (assuming every component’s error is minimized).
Revised: Based on the above-mentioned designs and experimental validations on the fidelity of SCG explanations, we argue that the explanation with SCG can be faithful to the original NN, in ideal scenarios (assuming every component’s error is minimized).

Original: As part of future work, we plan to investigate the performance influence of the non-perfect fidelity of SCG.
Revised: As part of future work, we plan to investigate the performance influence of the non-perfect fidelity of SCG.

Original: However with our current pipeline, the users can rely on the knowledge distillation loss, which can serve as a good indication of the faithfulness of the SCG to the original NN, to gain a better understanding of whether the SCG needs further improvement (e.g. with more data used in knowledge distillation, more visual concepts should be selected, etc.)
Revised: However with our current pipeline, the users can rely on the knowledge distillation loss, which can serve as a good indication of the faithfulness of the SCG to the original NN, to gain a better understanding of whether the SCG needs further improvement (e.g. with more data used in knowledge distillation, more visual concepts should be selected, etc.)

Original: B I MPLEMENTATION D ETAIL B. P IPELINE SUMMARIZATION
Revised: B I MPLEMENTATION D ETAIL B. P IPELINE SUMMARIZATION

Original: Table.
Revised: Table.

Original: 4 is the input and output summarization of the whole Human-NN Interface pipeline and each module/process.
Revised: 4 is the input and output summarization of the whole Human-NN Interface pipeline and each module/process.

Original: [Table]
Revised: [Table]

Original: B. C ONCEPT MATCHING
Revised: B. C ONCEPT MATCHING

Original: In multi-resolution segmentation step
Revised: In multi-resolution segmentation step

Original: [Table]
Revised: [Table]

Original: to extract features for each patch resulting from image multiresolution segmentation, we resize the patch and use the original NN to compute features after a specific layer, e.g. “layer4.1.conv2” layer of ResNet-18.
Revised: to extract features for each patch resulting from image multiresolution segmentation, we resize the patch and use the original NN to compute features after a specific layer, e.g. “layer4.1.conv2” layer of ResNet-18.

Original: For each discovered concept, we store the mean vector of all patches belonging to this concept as an anchor for future concept matching given any image.
Revised: For each discovered concept, we store the mean vector of all patches belonging to this concept as an anchor for future concept matching given any image.

Original: In the concept match step (same as Fig.
Revised: In the concept match step (same as Fig.

Original: 4 step 2), for each class of interest c , we match candidate features to the stored anchors (mean concept vectors) of top k concepts in the concept pool
Revised: 4 step 2), for each class of interest c , we match candidate features to the stored anchors (mean concept vectors) of top k concepts in the concept pool

Original: [Table]
Revised: [Table]

Original: and we construct an I-SCG for image I and class c , based on similarity (Euclidean distance) between image patch feature and concept anchor feature.
Revised: and we construct an I-SCG for image I and class c , based on similarity (Euclidean distance) between image patch feature and concept anchor feature.

Original: Specifically, if the nearest image patch regards the euclidean distance between the concept anchor feature and the image patch feature is smaller than a threshold t , then we will identify this patch as a detected concept.
Revised: Specifically, if the nearest image patch regards the euclidean distance between the concept anchor feature and the image patch feature is smaller than a threshold t , then we will identify this patch as a detected concept.

Original: Otherwise, we will use dummy nodes (all feature values equal to a small constant ϵ ) to represent that undetected concept.
Revised: Otherwise, we will use dummy nodes (all feature values equal to a small constant ϵ ) to represent that undetected concept.

Original: We empirically choose the threshold t from observation when matching concepts mean vector to patches segmented from images and form image-level SCG (I-SCG).
Revised: We empirically choose the threshold t from observation when matching concepts mean vector to patches segmented from images and form image-level SCG (I-SCG).

Original: Specifically, the distance in the positive match and negative match have different orders of magnitudes in latent space, and the performance is not sensitive to the selection of t .
Revised: Specifically, the distance in the positive match and negative match have different orders of magnitudes in latent space, and the performance is not sensitive to the selection of t .

Original: We plan to explore more in this direction as to future work.
Revised: We plan to explore more in this direction as to future work.

Original: B. N ODE ( CONCEPT ) MODIFICATION
Revised: B. N ODE ( CONCEPT ) MODIFICATION

Original: Fig.
Revised: Fig.

Original: 11(a)
Revised: 11(a)

Original: visualize the class-specific SCG (c-SCG) of three example classes of interest.
Revised: visualize the class-specific SCG (c-SCG) of three example classes of interest.

Original: A human can modify the anti-causality and anti-common sense concepts by new concepts extracted by VCE.
Revised: A human can modify the anti-causality and anti-common sense concepts by new concepts extracted by VCE.

Original: The new c-SCG, which merged human’s reasoning logic, are shown in Fig.
Revised: The new c-SCG, which merged human’s reasoning logic, are shown in Fig.

Original: 11(b).
Revised: 11(b).

Original: B. P ARTIAL K NOWLEDGE D ISTILLATION
Revised: B. P ARTIAL K NOWLEDGE D ISTILLATION

Original: For the all partial knowledge distillation experiments in the Human-to-NN path, we use SGD with initial learning rate = 0.1, and it will multiply 0.1 every 30 epoch.
Revised: For the all partial knowledge distillation experiments in the Human-to-NN path, we use SGD with initial learning rate = 0.1, and it will multiply 0.1 every 30 epoch.

Original: Batch size = 4, the coefficient of L soft , α = 2.5, the coefficient of L hard , β = 1.
Revised: Batch size = 4, the coefficient of L soft , α = 2.5, the coefficient of L hard , β = 1.

Original: For Nodes (concepts) modification experiment (4.1.1 in the main paper), we use T s = 1 , T T 1 = 1 . 5 , T T 2 = 1 , For edges (concepts relationship) modification (4.1.2 in the main paper), we use T s = 2 , T T 1 = 2 .
Revised: For Nodes (concepts) modification experiment (4.1.1 in the main paper), we use T s = 1 , T T 1 = 1 . 5 , T T 2 = 1 , For edges (concepts relationship) modification (4.1.2 in the main paper), we use T s = 2 , T T 1 = 2 .

Original: For zero-shot learning experiments (4.2 in the main paper), we use T s = 1 . 5 , T T 1 = 1 . 5 .
Revised: For zero-shot learning experiments (4.2 in the main paper), we use T s = 1 . 5 , T T 1 = 1 . 5 .

Original: We use knowledge distillation to train original model for 100 epochs.
Revised: We use knowledge distillation to train original model for 100 epochs.

Original: For the performance influence on unmodified classes in Table.
Revised: For the performance influence on unmodified classes in Table.

Original: We agree that if we cannot find the best partial knowledge distillation temperature hyperparameter, a slight performance fluctuation of
Revised: We agree that if we cannot find the best partial knowledge distillation temperature hyperparameter, a slight performance fluctuation of

Original: [Figure]
Revised: [Figure]

Original: unmodified classes in Table.
Revised: unmodified classes in Table.

Original: 1 may occur.
Revised: 1 may occur.

Original: We call it fluctuation because we also find some unmodified classes slightly improved after distillation.
Revised: We call it fluctuation because we also find some unmodified classes slightly improved after distillation.

Original: We will conduct more experiments on this direction.
Revised: We will conduct more experiments on this direction.

Original: B. G RAPH R EASONING N ETWORK
Revised: B. G RAPH R EASONING N ETWORK

Original: Network Structure
Revised: Network Structure

Original: The network architectures of Graph Reasoning Network ( GRN ) are shown in
Revised: The network architectures of Graph Reasoning Network ( GRN ) are shown in

Original: Table.
Revised: Table.

Original: GRN consists of two parts, Graph Neural Network G and Embedding Network E .
Revised: GRN consists of two parts, Graph Neural Network G and Embedding Network E .

Original: G takes n hypotheses
Revised: G takes n hypotheses

Original: h
Revised: h

Original: = { h 1 , h 2 , ...h n } , (each hypothesis h
Revised: = { h 1 , h 2 , ...h n } , (each hypothesis h

Original: i is in the form of Structural Concept Graph (SCG), and each SCG has m n nodes as well as m e edges) as input, and output n feature vectors ( G ( h i ) ) which concatenate all updated node and edge features of h i .
Revised: i is in the form of Structural Concept Graph (SCG), and each SCG has m n nodes as well as m e edges) as input, and output n feature vectors ( G ( h i ) ) which concatenate all updated node and edge features of h i .

Original: In G , we use class-specific e cji for different hypotheses in each GraphConv layer.
Revised: In G , we use class-specific e cji for different hypotheses in each GraphConv layer.

Original: E concatenates all n feature vectors from all the hypotheses into a long vector and maps it into n dimensional vector (1 × n ) with a 4 layer
Revised: E concatenates all n feature vectors from all the hypotheses into a long vector and maps it into n dimensional vector (1 × n ) with a 4 layer

Original: MLP, where n is the number of classes of interest. ”
Revised: MLP, where n is the number of classes of interest. ”

Original: node” denotes node feature, ”edge” denotes edge feature, ”GraphConv” is graph convolutional layer, ”ReLU” denotes ReLU activation function, ”BN” denotes batch normalization, and
Revised: node” denotes node feature, ”edge” denotes edge feature, ”GraphConv” is graph convolutional layer, ”ReLU” denotes ReLU activation function, ”BN” denotes batch normalization, and

Original: ”FC” denotes fully connected layer.
Revised: ”FC” denotes fully connected layer.

Original: [Table]
Revised: [Table]

Original: Training details We train GRN in an end-to-end manner.
Revised: Training details We train GRN in an end-to-end manner.

Original: Below are the details: we use Adam with β 1 =0.9 and β 2 =0.999, batch size 32, learning rate 0.01 for the first 25 epochs and use a decay rate of 0.8 for every 25 epochs.
Revised: Below are the details: we use Adam with β 1 =0.9 and β 2 =0.999, batch size 32, learning rate 0.01 for the first 25 epochs and use a decay rate of 0.8 for every 25 epochs.

Original: We train GRN for 100 epochs.
Revised: We train GRN for 100 epochs.

Original: B. C OMPUTE R ESOURCES
Revised: B. C OMPUTE R ESOURCES

Original: We use 2 RTX-2080 and 2 Telsa V100.
Revised: We use 2 RTX-2080 and 2 Telsa V100.

Original: We train knowledge distillation on RTX-2080, which costs around 2 hours for 100 epochs on RTX-2080, and we train GRN on Telsa V100, which costs 1 hour for 100 epochs.
Revised: We train knowledge distillation on RTX-2080, which costs around 2 hours for 100 epochs on RTX-2080, and we train GRN on Telsa V100, which costs 1 hour for 100 epochs.

Original: C M ORE DETAILS OF OBJECT D ATASET
Revised: C M ORE DETAILS OF OBJECT D ATASET

Original: OBJECT is a computer-generated 3D object image dataset using Blender Community (2018) as the rendering engine.
Revised: OBJECT is a computer-generated 3D object image dataset using Blender Community (2018) as the rendering engine.

Original: Each image contains a 3D main object which consists of multiple basic object parts: ball, cylinder cubic, etc.
Revised: Each image contains a 3D main object which consists of multiple basic object parts: ball, cylinder cubic, etc.

Original: Each main object is rendered using 4 independent generating attributes: object size, background color, rotation angle, sub-object material.
Revised: Each main object is rendered using 4 independent generating attributes: object size, background color, rotation angle, sub-object material.

Original: The image size is 512 x 512.
Revised: The image size is 512 x 512.

Original: The first version of the dataset contains 13 different main objects and each one has more than 400 images.
Revised: The first version of the dataset contains 13 different main objects and each one has more than 400 images.

Original: We also publicly distribute the source code, which allows one to render new data with custom main objects.
Revised: We also publicly distribute the source code, which allows one to render new data with custom main objects.

Original: The domain of attributes and resolution depends on the needed dataset size.
Revised: The domain of attributes and resolution depends on the needed dataset size.

Original: Fig.
Revised: Fig.

Original: 12 shows an example that also illustrates the workflow of new object images generation: User can select the basic objects to form the main object, we will output the corresponding three-view drawing helping people to make sure the main object configuration and parameters are correct.
Revised: 12 shows an example that also illustrates the workflow of new object images generation: User can select the basic objects to form the main object, we will output the corresponding three-view drawing helping people to make sure the main object configuration and parameters are correct.

Original: After the user confirms the main object configuration, we can automatically synthesize over 400 images for user defined main object with different rendering attributes mentioned above.
Revised: After the user confirms the main object configuration, we can automatically synthesize over 400 images for user defined main object with different rendering attributes mentioned above.

Original: The output datasets will contain all possible combinations of the attributes.
Revised: The output datasets will contain all possible combinations of the attributes.

Original: Our primary motivation for creating the OBJECT datasets is that it allows fast testing and idea iteration, on zero-shot learning and 3D object recognition.
Revised: Our primary motivation for creating the OBJECT datasets is that it allows fast testing and idea iteration, on zero-shot learning and 3D object recognition.

Original: As there are two experiments of zero-shot learning with HNI, we upload their related datasets.
Revised: As there are two experiments of zero-shot learning with HNI, we upload their related datasets.

Original: There are four folders in our dataset zip file.
Revised: There are four folders in our dataset zip file.

Original: Folders with the ”main paper” prefix mean they are the data for the main paper experiment, and folders with the ”appendix” prefix mean they are the data for the appendix experiment.
Revised: Folders with the ”main paper” prefix mean they are the data for the main paper experiment, and folders with the ”appendix” prefix mean they are the data for the appendix experiment.

Original: You can download the dataset and its generating code from http:// anon-blind-submission , which we plan to keep up-to-date with contributions from ourselves and the community.
Revised: You can download the dataset and its generating code from http:// anon-blind-submission , which we plan to keep up-to-date with contributions from ourselves and the community.

Original: D L ARGER E XPERIMENTS ON N ODE AND E DGE M ODIFICATION
Revised: D L ARGER E XPERIMENTS ON N ODE AND E DGE M ODIFICATION

Original: We conducted a larger experiment with ImageNet dataset images.
Revised: We conducted a larger experiment with ImageNet dataset images.

Original: In addition to the originalclasses, we randomly selected another 100 classes and constructed a 120 classes dataset.
Revised: In addition to the originalclasses, we randomly selected another 100 classes and constructed a 120 classes dataset.

Original: For each class, we randomly selected 300 images for training and 200 images for testing.
Revised: For each class, we randomly selected 300 images for training and 200 images for testing.

Original: The dataset contains a total of 60000 images.
Revised: The dataset contains a total of 60000 images.

Original: After training, human users chose 16 classes of interest, and used the
Revised: After training, human users chose 16 classes of interest, and used the

Original: NN-to-Human path to visualize the reasoning logic (c-SCG) of NN, while the remaining classes were not modified.
Revised: NN-to-Human path to visualize the reasoning logic (c-SCG) of NN, while the remaining classes were not modified.

Original: Then the human users analyzed the c-SCG and modified both nodes (visual concepts) and edges (Concept relationships), one human intervention per class.
Revised: Then the human users analyzed the c-SCG and modified both nodes (visual concepts) and edges (Concept relationships), one human intervention per class.

Original: The performance of modified classes is shown in Table.
Revised: The performance of modified classes is shown in Table.

Original: 6 below:
Revised: 6 below:

Original: [Table]
Revised: [Table]

Original: For those 106 unmodified classes, their accuracy before and after distillation is consistent (with difference < 0.03), their mean accuracy are: (42.6% before distillation and 42.9% after distillation)
Revised: For those 106 unmodified classes, their accuracy before and after distillation is consistent (with difference < 0.03), their mean accuracy are: (42.6% before distillation and 42.9% after distillation)

Original: As shown in Fig.
Revised: As shown in Fig.

Original: In our larger experiments of 120 class ImageNet experiements, for the bird “white stork” (Fig. 13(a)), we delete the edges between head and feathers because they don’t have a very stable relationship, i.e. when the bird is in different poses: sit, stand, fly (Fig. 13(b)), its head and feathers may have different spatial relationships (no strong structure relationship between the two concepts).
Revised: In our larger experiments of 120 class ImageNet experiements, for the bird “white stork” (Fig. 13(a)), we delete the edges between head and feathers because they don’t have a very stable relationship, i.e. when the bird is in different poses: sit, stand, fly (Fig. 13(b)), its head and feathers may have different spatial relationships (no strong structure relationship between the two concepts).

Original: Instead, we add edges between the head and neck since it was not initially captured by the
Revised: Instead, we add edges between the head and neck since it was not initially captured by the

Original: NN and they actually share reliable structural relationships.
Revised: NN and they actually share reliable structural relationships.

Original: Similarly, for the “leopard”(Fig. 13(c)(d)), we delete the edges between the tail and legs and add the edges between head and back.
Revised: Similarly, for the “leopard”(Fig. 13(c)(d)), we delete the edges between the tail and legs and add the edges between head and back.

Original: [Figure]
Revised: [Figure]

Original: E M ORE E XPERIMENTS ON Z ERO -S HOT L EARNING THROUGH HNI
Revised: E M ORE E XPERIMENTS ON Z ERO -S HOT L EARNING THROUGH HNI

Original: We conduct two zero-shot learning experiments with HNI (Fig.
Revised: We conduct two zero-shot learning experiments with HNI (Fig.

Original: 15and Fig.
Revised: 15and Fig.

Original: 14), which is similar to the Sec.
Revised: 14), which is similar to the Sec.

Original: 4.2 experiment of zero-shot learning in the main paper.
Revised: 4.2 experiment of zero-shot learning in the main paper.

Original: Here, each learned object A, B, C, and D has 192 training images and 216 testing images while the new object E (we want to learn) has only 216 testing images.
Revised: Here, each learned object A, B, C, and D has 192 training images and 216 testing images while the new object E (we want to learn) has only 216 testing images.

Original: We use a ResNet-18 with no pretrain as classification model.
Revised: We use a ResNet-18 with no pretrain as classification model.

Original: Fig.
Revised: Fig.

Original: 14 and
Revised: 14 and

Original: Table.
Revised: Table.

Original: 7 shows the first experiments.
Revised: 7 shows the first experiments.

Original: Different from the previous experiment, in the following experiments (Fig. 15), we have 4 different shared concepts in this experiment, and each object uses 3 of them.
Revised: Different from the previous experiment, in the following experiments (Fig. 15), we have 4 different shared concepts in this experiment, and each object uses 3 of them.

Original: This setting enlarges the variance
Revised: This setting enlarges the variance

Original: [Figure]
Revised: [Figure]

Original: Given images of objects A, B, C, and D (object E is a new class with no images).
Revised: Given images of objects A, B, C, and D (object E is a new class with no images).

Original: Step 1: train a
Revised: Step 1: train a

Original: ResNet-18 with images of objects A, B, C, and D; then use VCE to discover the shared concepts; we can use concept match to obtain image-level SCGs (I-SCG) for learned objects.
Revised: ResNet-18 with images of objects A, B, C, and D; then use VCE to discover the shared concepts; we can use concept match to obtain image-level SCGs (I-SCG) for learned objects.

Original: Step 2: humans first analyze the relationship of new object E with shared concepts and learned objects (ABCD), then build custom I-SCG for object E by recombining the nodes and edges.
Revised: Step 2: humans first analyze the relationship of new object E with shared concepts and learned objects (ABCD), then build custom I-SCG for object E by recombining the nodes and edges.

Original: Then train a GRN that can classify both ABCD and E. Step 3, we use knowledge distillation to transfer the knowledge from
Revised: Then train a GRN that can classify both ABCD and E. Step 3, we use knowledge distillation to transfer the knowledge from

Original: SCG back to the original NN.
Revised: SCG back to the original NN.

Original: Step 4: The original NN obtain the ability to classify the new object E
Revised: Step 4: The original NN obtain the ability to classify the new object E

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: between main objects, which is similar to more general real-life settings that there may have large inter-object variance.
Revised: between main objects, which is similar to more general real-life settings that there may have large inter-object variance.

Original: The new object E does not need to have totally the same concepts as the training objects, such as object A, they only have two same concepts.
Revised: The new object E does not need to have totally the same concepts as the training objects, such as object A, they only have two same concepts.

Original: This means that knowledge can be transferred between large variance objects which is more flexible.
Revised: This means that knowledge can be transferred between large variance objects which is more flexible.

Original: Fig. 15 and Table.
Revised: Fig. 15 and Table.

Original: 8 shows the performance of zero-shot learning with our HNI.
Revised: 8 shows the performance of zero-shot learning with our HNI.

Original: [Table]
Revised: [Table]

Original: F H UMAN - IN - THE -L OOP U SER S TUDY
Revised: F H UMAN - IN - THE -L OOP U SER S TUDY

Original: We conducted a user study with responses from 43 ML practitioners and students giving us the following data.
Revised: We conducted a user study with responses from 43 ML practitioners and students giving us the following data.

Original: For the practitioners: (1) they are volunteers to join the human-in-the-loop User Study and it is unpaid.
Revised: For the practitioners: (1) they are volunteers to join the human-in-the-loop User Study and it is unpaid.

Original: (2) the age range is 23 to 30 years old and there are 23 males and 20 females among allparticipants.
Revised: (2) the age range is 23 to 30 years old and there are 23 males and 20 females among allparticipants.

Original: The average time to finish this questionnaire is about 5 minutes.
Revised: The average time to finish this questionnaire is about 5 minutes.

Original: Over 95% of the participants found ”how to modify the Structural Concept Graph” is easy to understand.
Revised: Over 95% of the participants found ”how to modify the Structural Concept Graph” is easy to understand.

Original: Over 93% of the participants found our method is ”more helpful” than traditional ways to modify the logic of the neural network.
Revised: Over 93% of the participants found our method is ”more helpful” than traditional ways to modify the logic of the neural network.

Original: Given these visualizations, 91% ”expressed interest” in potentially using HNI to modify their models.
Revised: Given these visualizations, 91% ”expressed interest” in potentially using HNI to modify their models.

Original: Table.
Revised: Table.

Original: 9 shows the details of our questionnaire.
Revised: 9 shows the details of our questionnaire.

Original: As for the ”explanation” referred to in question 1, we directly use Fig.
Revised: As for the ”explanation” referred to in question 1, we directly use Fig.

Original: 3 in the main paper as the example.
Revised: 3 in the main paper as the example.

Original: [Table]
Revised: [Table]

Original: G M ORE DISCUSSION G. T OWARDS G ENERIC I NTERFACE
Revised: G M ORE DISCUSSION G. T OWARDS G ENERIC I NTERFACE

Original: As this work is the first attempt towards a generic interface between human and neural networks, we primarily focus on image-related applications in our experiments.
Revised: As this work is the first attempt towards a generic interface between human and neural networks, we primarily focus on image-related applications in our experiments.

Original: However, we want to point out that the proposed framework can be easily extended to other data modalities and tasks with minor adaptations.
Revised: However, we want to point out that the proposed framework can be easily extended to other data modalities and tasks with minor adaptations.

Original: The input modality is only related to the concept extraction where the concepts can be extracted from other modalities such as texts and other structured data.
Revised: The input modality is only related to the concept extraction where the concepts can be extracted from other modalities such as texts and other structured data.

Original: For regression tasks, our pipeline can easily be adapted because we do not have assumptions or constraints on the output format, the Graph reasoning network can also do regression tasks if the original NN is used to solve the regression problem.
Revised: For regression tasks, our pipeline can easily be adapted because we do not have assumptions or constraints on the output format, the Graph reasoning network can also do regression tasks if the original NN is used to solve the regression problem.

Original: In both classification and regression tasks, humans only need to be involved with class-specific SCG (c-SCG) to modify the reasoning logic, which is agnostic to the output format.
Revised: In both classification and regression tasks, humans only need to be involved with class-specific SCG (c-SCG) to modify the reasoning logic, which is agnostic to the output format.

Original: For textual input, only adaptation required is in the concept extraction, one will need to discover the concepts with different representations (e.g., important tokens).
Revised: For textual input, only adaptation required is in the concept extraction, one will need to discover the concepts with different representations (e.g., important tokens).

Original: We do have the plan to apply the proposed framework to tackle more applications such as few-shot learning, continual learning, active learning, more humanoid Neural networks, and fine-grained classification
Revised: We do have the plan to apply the proposed framework to tackle more applications such as few-shot learning, continual learning, active learning, more humanoid Neural networks, and fine-grained classification

Original: G. LIMITED DATASET SETTING
Revised: G. LIMITED DATASET SETTING

Original: The reason we use a limited dataset is to simulate the scenario where NN fails to generalize from limited training data, hence it is easier to demonstrate how human guidance can help to improve the original NN in this case.
Revised: The reason we use a limited dataset is to simulate the scenario where NN fails to generalize from limited training data, hence it is easier to demonstrate how human guidance can help to improve the original NN in this case.

Original: However, we want to emphasize that the proposed framework is not limited by or making any assumption on, the training data size.
Revised: However, we want to emphasize that the proposed framework is not limited by or making any assumption on, the training data size.

Original: First, the size of GRN depends on the classes of interest whose logic we want to modify, and it is compatible with any scale original NN with a large number of unmodified classes.
Revised: First, the size of GRN depends on the classes of interest whose logic we want to modify, and it is compatible with any scale original NN with a large number of unmodified classes.

Original: As is shown in Table.
Revised: As is shown in Table.

Original: 1, where we modify the logic of 6 classes of interest among 20 classes, the number of modified classes is independent of the total number of classes.
Revised: 1, where we modify the logic of 6 classes of interest among 20 classes, the number of modified classes is independent of the total number of classes.

Original: We only create a Graph reasoning network for the classes of interest, and change the “local logic” where these classes may be confused with each other.
Revised: We only create a Graph reasoning network for the classes of interest, and change the “local logic” where these classes may be confused with each other.

Original: On the other hand, when sufficient sampling of a large variety of classes (e.g., ImageNet) is available in training, NN is likely to be able to learn a good representation and reliable reasoning logic.
Revised: On the other hand, when sufficient sampling of a large variety of classes (e.g., ImageNet) is available in training, NN is likely to be able to learn a good representation and reliable reasoning logic.

Original: In this case, NN can help humans to better understand and learn to differentiate specific classes by NN-to-Human path via the visualization of the decision logic (e.g., the most discriminative concepts and relationships).
Revised: In this case, NN can help humans to better understand and learn to differentiate specific classes by NN-to-Human path via the visualization of the decision logic (e.g., the most discriminative concepts and relationships).

Original: However, in real-world scenarios, if we have very limited data and a limited number of very specific classes of interest, NN may learn some hard-to-decouple bias and may fail to generalize.
Revised: However, in real-world scenarios, if we have very limited data and a limited number of very specific classes of interest, NN may learn some hard-to-decouple bias and may fail to generalize.

Original: Human users can easily spot the bias (irrelevant concepts and incorrect relationships) and help modify the reasoning logic to improve the original NN’s performance.
Revised: Human users can easily spot the bias (irrelevant concepts and incorrect relationships) and help modify the reasoning logic to improve the original NN’s performance.

Original: That is the reason why, to demonstrate the efficacy of our framework, we chose to use a limited dataset to simulate the scenarios with data scarcity.
Revised: That is the reason why, to demonstrate the efficacy of our framework, we chose to use a limited dataset to simulate the scenarios with data scarcity.

Original: G. P ERFORMANCE DROP AFTER PARTIAL KNOWLEDGE DISTILLATION
Revised: G. P ERFORMANCE DROP AFTER PARTIAL KNOWLEDGE DISTILLATION

Original: GRN accuracy on the modified classes is better than modified NN after knowledge distillation.
Revised: GRN accuracy on the modified classes is better than modified NN after knowledge distillation.

Original: Here are the analysis and reasons:
Revised: Here are the analysis and reasons:

Original: (1) GRN is
Revised: (1) GRN is

Original: a specialist model which is only trained on modified classes of interest and only needs to consider the local logic.
Revised: a specialist model which is only trained on modified classes of interest and only needs to consider the local logic.

Original: It cannot predict the unmodified classes.
Revised: It cannot predict the unmodified classes.

Original: However, the original NN is a larger and general model which can classify more categories.
Revised: However, the original NN is a larger and general model which can classify more categories.

Original: (2) Our proposed “Partial knowledge distillation” method (Fig. 5) tries to transfer knowledge between
Revised: (2) Our proposed “Partial knowledge distillation” method (Fig. 5) tries to transfer knowledge between

Original: Graph neural network and Convolutional Neural network with different input modalities, we also have two teacher models and combine them together as soft labels.
Revised: Graph neural network and Convolutional Neural network with different input modalities, we also have two teacher models and combine them together as soft labels.

Original: It is known that knowledge distillation is sensitive to the hyperparameter temperature ( T ).
Revised: It is known that knowledge distillation is sensitive to the hyperparameter temperature ( T ).

Original: We have three different choices of T s in our case which makes it even harder to find the best one.
Revised: We have three different choices of T s in our case which makes it even harder to find the best one.

Original: Due to the goal of verifying the key idea, we do not take too much effort to conduct a hyperparameter search which may further improve the distillation performance.
Revised: Due to the goal of verifying the key idea, we do not take too much effort to conduct a hyperparameter search which may further improve the distillation performance.

Original: Moreover, we need to keep the high performance of the unmodified classes, which may also influence the performance of modified classes during knowledge distillation.
Revised: Moreover, we need to keep the high performance of the unmodified classes, which may also influence the performance of modified classes during knowledge distillation.

Original: (3) Combining both GRN and original NN may have the best performance, but at an additional cost (both in parameters and run time).
Revised: (3) Combining both GRN and original NN may have the best performance, but at an additional cost (both in parameters and run time).

Original: The reason we want to distill the knowledge from GRN back to the original NN is to try to avoid modifying the structure, size, and inference time of the original NN and instead serve as a tool to improve its performance.
Revised: The reason we want to distill the knowledge from GRN back to the original NN is to try to avoid modifying the structure, size, and inference time of the original NN and instead serve as a tool to improve its performance.

Original: This way it does not affect the deployment of the original NN (e.g., the original NN may be a tiny network to be deployed on an edge platform).
Revised: This way it does not affect the deployment of the original NN (e.g., the original NN may be a tiny network to be deployed on an edge platform).

Original: Our main goal is to use the proposed Human-NN Interface, to exchange knowledge between human users and original NN.
Revised: Our main goal is to use the proposed Human-NN Interface, to exchange knowledge between human users and original NN.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G. L IMITATIONS
Revised: G. L IMITATIONS

Original: (1) The zero-shot learning experiments use a synthesized dataset which is an ideal situation to show the idea.
Revised: (1) The zero-shot learning experiments use a synthesized dataset which is an ideal situation to show the idea.

Original: But in the real world, the shared concepts between objects may have larger variance, which may hinder the performance.
Revised: But in the real world, the shared concepts between objects may have larger variance, which may hinder the performance.

Original: The knowledge distillation result is sensitive to the temperature of knowledge distillation, finding a suitable temperature is a time-consuming process.
Revised: The knowledge distillation result is sensitive to the temperature of knowledge distillation, finding a suitable temperature is a time-consuming process.

Original: The consistency of discovered concepts may not be good.
Revised: The consistency of discovered concepts may not be good.

Original: However, our work highly relies on the result of discovered concepts.
Revised: However, our work highly relies on the result of discovered concepts.

Original: In order to solve this, we need to use different segmentation methods to find consistent concepts, which takes a longer time.
Revised: In order to solve this, we need to use different segmentation methods to find consistent concepts, which takes a longer time.

Original: We plan to use better methods for segmentation and match during the concept discovery and matching stage.
Revised: We plan to use better methods for segmentation and match during the concept discovery and matching stage.

Original: G. B ROADER I MPACT
Revised: G. B ROADER I MPACT

Original: HNI allows humans and NN to understand each other using SCG as a ”language”.
Revised: HNI allows humans and NN to understand each other using SCG as a ”language”.

Original: A human can directly modify NN with human prior knowledge.
Revised: A human can directly modify NN with human prior knowledge.

Original: It can also help us ”teach” NN to learn new objects.
Revised: It can also help us ”teach” NN to learn new objects.

Original: This can have a positive impact on: (1) We can teach NN not to learn some knowledge we do not want it to learn, e.g., not learn some social bias; (2) we can help NN learn something it has never seen before.
Revised: This can have a positive impact on: (1) We can teach NN not to learn some knowledge we do not want it to learn, e.g., not learn some social bias; (2) we can help NN learn something it has never seen before.

Original: This will be useful when we lack the data.
Revised: This will be useful when we lack the data.

Original: We can use our prior knowledge to build
Revised: We can use our prior knowledge to build

Original: SCG and help NN to learn the knowledge of this object.
Revised: SCG and help NN to learn the knowledge of this object.

Original: However, those who have bad intentions may be able to figure out a way to use it maliciously.
Revised: However, those who have bad intentions may be able to figure out a way to use it maliciously.

Original: In our case, one might find a way to teach NN something unsuitable.
Revised: In our case, one might find a way to teach NN something unsuitable.

Original: There are more and more scenarios using NN to help humans to make decisions.
Revised: There are more and more scenarios using NN to help humans to make decisions.

Original: If we teach NN unsuitable knowledge, it may make wrong decisions, but that could be done through standard training as well.
Revised: If we teach NN unsuitable knowledge, it may make wrong decisions, but that could be done through standard training as well.

Original: G. A SSET
Revised: G. A SSET

Original: Imagenet license: Researcher shall use the Database only for non-commercial research and educational purposes
Revised: Imagenet license: Researcher shall use the Database only for non-commercial research and educational purposes

Original: We use a subset of the ILSVRC2012 dataset (ImageNet) and iLab-20M.
Revised: We use a subset of the ILSVRC2012 dataset (ImageNet) and iLab-20M.

Original: The objects in ImageNet are vehicles and animals that can be found on the Internet.
Revised: The objects in ImageNet are vehicles and animals that can be found on the Internet.

Original: The object in iLab-20M is toy vehicles, under
Revised: The object in iLab-20M is toy vehicles, under

Original: Creative Commons CC-BY license.
Revised: Creative Commons CC-BY license.

Original: They do not contain any personally identifiable information offensive content.
Revised: They do not contain any personally identifiable information offensive content.