Original: R EL V I T: C ONCEPT - GUIDED V ISION T RANSFORMER FOR V ISUAL R ELATIONAL R EASONING
Revised: R EL V I T: C ONCEPT - GUIDED V ISION T RANSFORMER FOR V ISUAL R ELATIONAL R EASONING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reasoning about visual relationships is central to how humans interpret the visual world.
Revised: Reasoning about visual relationships is central to how humans interpret the visual world.

Original: This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i.e. systematic generalization.
Revised: This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i.e. systematic generalization.

Original: In this work, we use vision transformers (ViTs) as our base model for visual reasoning and make better use of concepts deﬁned as object entities and their relations to improve the performances.
Revised: In this work, we use vision transformers (ViTs) as our base model for visual reasoning and make better use of concepts deﬁned as object entities and their relations to improve the reasoning ability of ViTs.

Original: Speciﬁcally, we introduce a novel concept-feature dictionary to allow ﬂexible image feature retrieval at training time with concept keys.
Revised: Speciﬁcally, we introduce a novel concept-feature dictionary to allow ﬂexible image feature retrieval at training time with concept keys.

Original: This dictionary enables two new conceptguided auxiliary tasks: 1) a global task for promoting relational reasoning, and 2) a local task for facilitating semantic object-centric correspondence learning.
Revised: This dictionary enables two new conceptguided auxiliary tasks: 1) a global task for promoting relational reasoning, and 2) a local task for facilitating semantic object-centric correspondence learning.

Original: To examine the systematic generalization of visual reasoning models, we introduce systematic splits for the standard HICO and GQA benchmarks.
Revised: To examine the systematic generalization of visual reasoning models, we introduce systematic splits for the standard HICO and GQA benchmarks.

Original: We show the resulting model, Concept-guided Vision Transformer (or RelViT for short) significantly outperforms prior approaches on HICO and GQA by 16% and 13% in the original split, and by 43% and 18% in the systematic split.
Revised: We show the resulting model, Concept-guided Vision Transformer (or RelViT for short) significantly outperforms prior approaches on HICO and GQA by 16% and 13% in the original split, and by 43% and 18% in the systematic split.

Original: Our ablation analyses also reveal our model’s compatibility with multiple ViT variants and robustness to hyper-parameters.
Revised: Our ablation analyses also reveal our model’s compatibility with multiple ViT variants and robustness to hyper-parameters.

Original: I NTRODUCTION
Revised: I NTRODUCTION

Original: Deep neural networks have achieved great success in visual recognition.
Revised: Deep neural networks have achieved great success in visual recognition.

Original: However, their ability for visual relational reasoning, i.e. reasoning with entities and their relationships in a visual scene, still falls short of human-level performances, especially in real-world domains.
Revised: However, their ability for visual relational reasoning, i.e. reasoning with entities and their relationships in a visual scene, still falls short of human-level performances, especially in real-world domains.

Original: The challenges of common visual relational reasoning tasks, e.g. HICO and GQA benchmarks (Chao et al., 2015;
Revised: The challenges of common visual relational reasoning tasks, e.g. HICO and GQA benchmarks (Chao et al., 2015;

Original: Hudson & Manning, 2019) are manifested in three aspects: 1) object-centric learning: to identify objects (including humans) as well as their visual properties; 2) relational reasoning: to infer all pairwise relationships between the object entities; and 3) systematic generalization: to reason with visual entities and relations on novel object-relation combinations and extrapolate to longer reasoning hops (Bahdanau et al., 2018; Hupkes et al., 2020).
Revised: Hudson & Manning, 2019) are manifested in three aspects: 1) object-centric learning to identify objects (including humans) as well as their visual properties; 2) relational reasoning to infer all pairwise relationships between the object entities; and 3) systematic generalization to reason with visual entities and relations on novel object-relation combinations and extrapolate to longer reasoning hops (Bahdanau et al., 2018; Hupkes et al., 2020).

Original: While existing models have leveraged pre-trained object detectors (Ren et al., 2015; Jiang et al., 2020) and/or explicit symbolic reasoning methods (Yi et al., 2018) to tackle these challenges, they leave ample space for improvement.
Revised: While existing models have leveraged pre-trained object detectors (Ren et al., 2015; Jiang et al., 2020) and/or explicit symbolic reasoning methods (Yi et al., 2018) to tackle these challenges, they leave ample space for improvement.

Original: More recently, vision transformers (ViTs) have become the new paradigm for visual recognition and have made great strides in a broad range of visual recognition tasks (Dosovitskiy et al., 2020;
Revised: More recently, vision transformers (ViTs) have become the new paradigm for visual recognition and have made great strides in a broad range of visual recognition tasks (Dosovitskiy et al., 2020;

Original: Wang et al., 2021a; Liu et al., 2021).
Revised: Wang et al., 2021a; Liu et al., 2021).

Original: Several properties of ViT make it a compelling model choice for visual relational reasoning.
Revised: Several properties of ViT make it a compelling model choice for visual relational reasoning.

Original: First, the self-attention mechanism in ViT offers a strong relational inductive bias, explicitly modeling the relations between input entities.
Revised: First, the self-attention mechanism in ViT offers a strong relational inductive bias, explicitly modeling the relations between input entities.

Original: Second, the design of image as patches facilitates the learning of object-centric representations, as evidenced by recent works, e.g. DINO and EsViT (Caron et al., 2021; Li et al., 2021), that demonstrate ViTs trained with self-supervised learning (SSL) capture objects in the image without label annotations.
Revised: Second, the design of image as patches facilitates the learning of object-centric representations, as evidenced by recent works, e.g. DINO and EsViT (Caron et al., 2021; Li et al., 2021), that demonstrate ViTs trained with self-supervised learning (SSL) capture objects in the image without label annotations.

Original: To investigate the efﬁcacy of the ViT backbone for visual relational reasoning, in particular on systematic generalization, we introduce new systematic splits to canonical benchmarks and compare the ViT backbone with the CNN backbone.
Revised: To investigate the efﬁcacy of the ViT backbone for visual relational reasoning, in particular on systematic generalization, we introduce new systematic splits to canonical benchmarks and compare the ViT backbone with the CNN backbone.

Original: Results on GQA show that switching to ViTs in MCAN model (Yu et al., 2019) brings an immediate 11% gain in accuracy.
Revised: Results on GQA show that switching to ViTs in MCAN model (Yu et al., 2019) brings an immediate 11% gain in accuracy.

Original: However, the performance gap between the original GQA testing split and the new systematic split remains considerable (15% in accuracy) for both backbones.
Revised: However, the performance gap between the original GQA testing split and the new systematic split remains considerable (15% in accuracy) for both backbones.

Original: It  that generic ViTscan still  be improved to tackle the reasoning task, especially on systematic generalization.
Revised: It suggests that generic ViTs still need to be improved to tackle the reasoning task, especially on systematic generalization.

Original: Recent works have shown that neural networks can learn representations with better generalization, by learning certain auxiliary tasks of predicting human-speciﬁed concepts (Hill et al., 2020; Koh et al., 2020).
Revised: Recent works have shown that neural networks can learn representations with better generalization, by learning certain auxiliary tasks of predicting human-speciﬁed concepts (Hill et al., 2020; Koh et al., 2020).

Original: A natural question emerges: can we exploit these concepts to improve the reasoning ability of ViTs?
Revised: A natural question emerges: can we exploit these concepts to improve the reasoning ability of ViTs?

Original: Our approach is to make better use of concepts (e.g. the labels in the original training dataset) in the
Revised: Our approach is to make better use of concepts (e.g. the labels in the original training dataset) in the

Original: ViT training for better relational reasoning.
Revised: ViT training for better relational reasoning.

Original: To this end, we ﬁrst introduce a novel concept-feature dictionary , where each key is a concept and its value is a queue of image features with the same concept, as shown in Figure 1.
Revised: To this end, we ﬁrst introduce a novel concept-feature dictionary , where each key is a concept and its value is a queue of image features with the same concept, as shown in Figure 1.

Original: It allows dynamic and ﬂexible training-time image feature retrieval during training.
Revised: It allows dynamic and ﬂexible training-time image feature retrieval during training.

Original: Based on this dictionary, we then augment the canonical ViT training pipeline with two auxiliary tasks: To facilitate high-level reasoning about relationships, we design a global task that helps cluster images with the same concept together to produce semantically consistent relational representations.
Revised: Based on this dictionary, we then augment the canonical ViT training pipeline with two auxiliary tasks: First, to facilitate high-level reasoning about relationships, we design a global task that helps cluster images with the same concept together to produce semantically consistent relational representations.

Original: To learn better object-centric representations, we develop a local task that guides the model to discover object-centric semantic correspondence across images (Liu et al., 2010).
Revised: Second, to learn better object-centric representations, we develop a local task that guides the model to discover object-centric semantic correspondence across images (Liu et al., 2010).

Original: Thanks to the plug-and-play feature of our concept-feature dictionary, our auxiliary tasks can be easily incorporated intothe existing ViT training pipeline without additional input pre-processing.
Revised: Thanks to the plug-and-play feature of our concept-feature dictionary, our auxiliary tasks can be easily incorporated into existing ViT training pipelines without additional input preprocessing.

Original: We term the resulting model concept-guided vision transformer (or RelViT for short).
Revised: We term the resulting model concept-guided vision transformer (or RelViT for short).

Original: by 16%, 43%, and 7% on the original non-systematic and two new systematic splits.

Original: Sys. : systematic.

Original: We evaluate our method on two standard visual relational reasoning benchmarks: HICO and GQA.
Revised: We evaluate our method on two standard visual relational reasoning benchmarks: HICO and GQA.

Original: Beyond the original independent and identically distributed (I.I.D.) training-testing split, we introduce a systematic split for each dataset to examine the ability of systematic generalization, i.e. , recognizing novel object-relation combinations.
Revised: Beyond the original independent and identically distributed (I.I.D.) training-testing split, we introduce new systematic splits for each dataset to examine the ability of systematic generalization, i.e. , recognizing novel object-relation combinations.

Original: Our results show that
Revised: Our results show that

Original: RelViT signiﬁcantly outperforms previous approaches.
Revised: RelViT signiﬁcantly outperforms previous approaches.

Original: On
Revised: On

Original: HICO, it improves the best baseline by 16%, 43%, and 7% on the original non-systematic and two new systematic splits, respectively, as shown in Figure 2.
Revised: HICO, it improves the best baseline by 16%, 43%, and 7% on the original non-systematic and two new systematic splits, respectively, as shown in Figure 2.

Original: On GQA, it further closes the gap of overall accuracy between models using visual backbone feature only and models using additional bounding box features (obtained from pre-trained object detectors) by 13% and 18% on the two splits.
Revised: On GQA, it further closes the gap of overall accuracy between models using visual backbone feature only and models using additional bounding box features (obtained from pre-trained object detectors) by 13% and 18% on the two splits.

Original: We also show that our method is compatible with various ViT variants and robust to hyperparameters.
Revised: We also show that our method is compatible with various ViT variants and robust to hyperparameters.

Original: Finally, our qualitative inspection indicates that
Revised: Finally, our qualitative inspection indicates that

Original: RelViT does improve ViTs on learning relational and object-centric representations.
Revised: RelViT does improve ViTs on learning relational and object-centric representations

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We propose RelViT, by incorporating visual relational concepts to the ViT training with the newlyintroduced concept-guided global and local auxiliary tasks, where a concept-feature dictionary is proposed to enable dynamic and ﬂexible image feature retrieval with the concept keys.
Revised: • We propose RelViT, by incorporating visual relational concepts to the ViT training with the newlyintroduced concept-guided global and local auxiliary tasks, where a concept-feature dictionary is proposed to enable dynamic and ﬂexible image feature retrieval with the concept keys.

Original: • In extensive experiments on the original non-systematic and new systematic split of the HICO and GQA datasets, we demonstrate the advantages of RelViT over various strong baselines for visual relational reasoning.
Revised: • In extensive experiments on the original non-systematic and new systematic split of the HICO and GQA datasets, we demonstrate the advantages of RelViT over various strong baselines for visual relational reasoning.

Original: • We perform ablation studies on RelViT to show the contributions of its key components, its compatibility to various ViT architectures, and its robustness to hyper-parameters.
Revised: • We perform ablation studies on RelViT to show the contributions of its key components, its compatibility to various ViT architectures, and its robustness to hyper-parameters.

Original: We provide qualitative results to conﬁrm our improved learning of relational and object-centric representations.
Revised: We provide qualitative results to conﬁrm our improved learning of relational and object-centric representations.

Original:  M ETHODOLOGY 2. B ACKGROUND
Revised:  M ETHODOLOGY 2. B ACKGROUND

Original: Vision transformers.
Revised: Vision transformers.

Original: Here we brieﬂy review the architecture of multi-staged ViTs (Dosovitskiy et al., 2020).
Revised: Here we brieﬂy review the architecture of multi-staged ViTs (Dosovitskiy et al., 2020).

Original: Given an image I ∈
Revised: Given an image I ∈

Original: R H × W × C , a ViT model g ﬁrst tokenizes the input into N image tokens (patches) with a resolution of ( T, T ) : tokenize ( I ) =
Revised: R H × W × C , a ViT model g ﬁrst tokenizes the input into N image tokens (patches) with a resolution of ( T, T ) : tokenize ( I ) =

Original: [ t 1 , · · · , t N ] , t i ∈ R T 2 × C , N =
Revised: [ t 1 , · · · , t N ] , t i ∈ R T 2 × C , N =

Original: HW/T 2 , where ( H, W ) and C denotes the original resolution and number of channel of the image, respectively.
Revised: HW/T 2 , where ( H, W ) and C denotes the original resolution and number of channel of the image, respectively.

Original: Then in each stage, a patch embedding and a multi-head self attention (MHSA) module is applied to these tokens to produce input for the next stage.
Revised: Then in each stage, a patch embedding and a multi-head self attention (MHSA) module is applied to these tokens to produce input for the next stage.

Original: The ﬁnal output of ViT g
Revised: The ﬁnal output of ViT g

Original: ( I ) is a sequence of tokens [ z 1 , · · · , z N ] that correspond to the aforementioned input tokens.
Revised: ( I ) is a sequence of tokens [ z 1 , · · · , z N ] that correspond to the aforementioned input tokens.

Original: For global prediction tasks, e.g. image categorization, a summary of the input image can be obtained by either inserting an extra [CLS] token to the input sequence of image tokens or performing an extra pooling operation over the output tokens (Zhai et al., 2021).
Revised: For global prediction tasks, e.g. image categorization, a summary of the input image can be obtained by either inserting an extra [CLS] token to the input sequence of image tokens or performing an extra pooling operation over the output tokens (Zhai et al., 2021).

Original: Self-supervised learning with DINO and EsViT.
Revised: Self-supervised learning with DINO and EsViT.

Original: Our method is developed upon the recently proposed self-supervised learning (SSL) approach self-distillation with no labels (DINO) (Caron et al., 2021) and its follow-up EsViT (Li et al., 2021).
Revised: Our method is developed upon the recently proposed self-supervised learning (SSL) approach self-distillation with no labels (DINO) (Caron et al., 2021) and its follow-up EsViT (Li et al., 2021).

Original: As shown in Figure 1, their main idea is to encourage the output consistency between a teacher g t and a student network g s , parameterized by θ t and θ s , respectively.
Revised: As shown in Figure 1, their main idea is to encourage the output consistency between a teacher g t and a student network g s , parameterized by θ t and θ s , respectively.

Original: Given an input image I , both networks map it to a probability distribution
Revised: Given an input image I , both networks map it to a probability distribution

Original: P t ( I ) =
Revised: P t ( I ) =

Original: h t ( g t ( I )) and P s ( I ) =
Revised: h t ( g t ( I )) and P s ( I ) =

Original: h s ( g s ( I )) via an extra projection head h ( · ) .
Revised: h s ( g s ( I )) via an extra projection head h ( · ) .

Original: The teacher and student network will be updated alternatively by following these two rules: (1) For the student network: θ s ← arg min θ s L Global , where L Global = − P t ( I ) log P s ( I ) ; (2) For the teacher network
Revised: The teacher and student network will be updated alternatively by following these two rules: (1) For the student network: θ s ← arg min θ s L Global , where L Global = − P t ( I ) log P s ( I ) ; (2) For the teacher network

Original: , θ t is updated using an exponential moving average (EMA) on θ s : θ t ← λθ t + (1 − λ ) θ s , where λ controls the updating momentum.
Revised: , θ t is updated using an exponential moving average (EMA) on θ s : θ t ← λθ t + (1 − λ ) θ s , where λ controls the updating momentum.

Original: In practice, multiple views of the input image I will be generated via data augmentation and the teacher and student networks will receive different views, preventing the task from being trivialized.
Revised: In practice, multiple views of the input image I will be generated via data augmentation and the teacher and student networks will receive different views, preventing the task from being trivialized.

Original: EsViT further extends the image-level loss L Global to patch-level by applying dense SSL (Wang et al., 2021c) for learning correspondence between the different views, enhancing the performance on dense prediction.
Revised: EsViT further extends the image-level loss L Global to patch-level by applying dense SSL (Wang et al., 2021c) for learning correspondence between the different views, enhancing the performance on dense prediction.

Original: Readers are encouraged to refer to Caron et al.
Revised: Readers are encouraged to refer to Caron et al.

Original: (2021) and Li et al.
Revised: (2021) and Li et al.

Original: (2021) for more details about these two works.
Revised: (2021) for more details about these two works.

Original: 2. R EL V I T
Revised: 2. R EL V I T

Original: RelViT is a concept-guided ViT that makes better use of the concepts in the ViT training for better relational reasoning.
Revised: RelViT is a concept-guided ViT that makes better use of the concepts in the ViT training for the improved relational reasoning.

Original: In this section, we ﬁrst introduce a concept-feature dictionary to store and retrieve image features with their concept keys.
Revised: In this section, we ﬁrst introduce a concept-feature dictionary to store and retrieve image features with their concept keys.

Original: We then augment the canonical ViT training pipeline with two auxiliary tasks: a global level task and a local level task, both are concept-guided by resorting to the concept-feature dictionary.
Revised: We then augment the canonical ViT training pipeline with two auxiliary tasks: a global level task and a local level task, both are concept-guided by resorting to the concept-feature dictionary.

Original: Intuitively, the global task help cluster images with the same concept together to produce semantically consistent relational features, while the local task guides the model to discover object-centric semantic correspondence across images.
Revised: Intuitively, the global task help cluster images with the same concept together to produce semantically consistent relational features, while the local task guides the model to discover object-centric semantic correspondence across images.

Original: Concept-feature dictionary.
Revised: Concept-feature dictionary.

Original: We assume the total number of concepts is M , and the set of all concepts C
Revised: We assume the total number of concepts is M , and the set of all concepts C

Original: = { c 1 , · · · , c M } .
Revised: = { c 1 , · · · , c M } .

Original: A concept-feature dictionary is denoted by D
Revised: A concept-feature dictionary is denoted by D

Original: = { ( c 1 , Q 1 ) , · · · , ( c M , Q M ) } , where each concept c i is associated with a queue Q i of image features.
Revised: = { ( c 1 , Q 1 ) , · · · , ( c M , Q M ) } , where each concept c i is associated with a queue Q i of image features.

Original: During training, each image I may come with multiple concepts, which we denote by C I ⊂ C .
Revised: During training, each image I may come with multiple concepts, which we denote by C I ⊂ C .

Original: For instance, there may exist several human-object interactions in an image from the HICO dataset, each of which may correspond to a concept.
Revised: For instance, there may exist several human-object interactions in an image from the HICO dataset, each of which may correspond to a concept.

Original: As shown in Figure 1, whenever a new image-concept pair ( I , C I ) comes, we uniformly draw a concept code c from C I , pick up the queue Q from the dictionary that corresponds to c , and then retrieve the image feature f from Q .
Revised: As shown in Figure 1, whenever a new image-concept pair ( I , C I ) comes, we uniformly draw a concept code c from C I , pick up the queue Q from the dictionary that corresponds to c , and then retrieve the image feature f from Q .

Original: Meanwhile, we pass the input image I to the teacher network g t to get the new image feature f (cid:48) =
Revised: Meanwhile, we pass the input image I to the teacher network g t to get the new image feature f (cid:48) =

Original: g t ( I ) , and enqueue it to Q .
Revised: g t ( I ) , and enqueue it to Q .

Original: Note that if Q is full already, we ﬁrst need to dequeue the oldest image feature from Q .
Revised: Note that if Q is full already, we ﬁrst need to dequeue the oldest image feature from Q .

Original: During training, we use the retrieved image feature f for the two auxiliary tasks below, rather than the input image feature f (cid:48) .
Revised: During training, we use the retrieved image feature f for the two auxiliary tasks below, rather than the input image feature f (cid:48) .

Original: Furthermore, the sampling strategy, i.e. how to retrieve image feature f from Q , plays an important role in the overall performance of our method.
Revised: Furthermore, the sampling strategy, i.e. how to retrieve image feature f from Q , plays an important role in the overall performance of our method.

Original: We consider the following two sampling strategies:
Revised: We consider the following two sampling strategies:

Original: • Uniform sampling.
Revised: • Uniform sampling.

Original: Each image feature is drawn with equal probability from the queue, i
Revised: Each image feature is drawn with equal probability from the queue, i

Original: e. suppose we have N features in the queue, then the probability of each feature being sampled is 1 /N .
Revised: e. suppose we have N features in the queue, then the probability of each feature being sampled is 1 /N .

Original: This tactic encourages the diversity of the retrieved image features, beneﬁting the overall performance.
Revised: This tactic encourages the diversity of the retrieved image features, beneﬁting the overall performance.

Original: However, some older features in the queue may largely fall behind the current model if the teacher network g t is updated quickly, eliciting unstable training.
Revised: However, some older features in the queue may largely fall behind the current model if the teacher network g t is updated quickly, eliciting unstable training.

Original: • “Most-recent” sampling.
Revised: • “Most-recent” sampling.

Original: The sampling probability mass is allocated based on the freshness of image features, and the most recent feature has the highest chance to be retrieved.
Revised: The sampling probability mass is allocated based on the freshness of image features, and the most recent feature has the highest chance to be retrieved.

Original: Speciﬁcally, suppose we have N features in the queue Q ( | Q | > = N ).
Revised: Speciﬁcally, suppose we have N features in the queue Q ( | Q | > = N ).

Original: Then for the i -th newest feature f , we deﬁne its weight w i = N − i + 1 .
Revised: Then for the i -th newest feature f , we deﬁne its weight w i = N − i + 1 .

Original: Finally, the probability of the i -th newest feature being sampled is w i / (cid:80) Nj =1 w j .
Revised: Finally, the probability of the i -th newest feature being sampled is w i / (cid:80) Nj =1 w j .

Original: This tactic ensures we retrieve more up-to-date features and thereby stabilizes the learning.
Revised: This tactic ensures we retrieve more up-to-date features and thereby stabilizes the learning.

Original: But it may hurt the overall performance due to a lack of feature diversity, as the chance of older features being sampled is small.
Revised: But it may hurt the overall performance due to a lack of feature diversity, as the chance of older features being sampled is small.

Original: Note that the feature queue is empty at the beginning of training.
Revised: Note that the feature queue is empty at the beginning of training.

Original: In this case, we simply use the input image feature f (cid:48) for the auxiliary tasks, and also enqueue it to Q that corresponds to the concept of the input image, as shown in Algorithm 1.
Revised: In this case, we simply use the input image feature f (cid:48) for the auxiliary tasks, and also enqueue it to Q that corresponds to the concept of the input image.

Original: As we can show in the next, now our proposed global and local tasks reduce to DINO (Caron et al., 2021) and EsViT (Li et al., 2021), respectively.
Revised: As we can show in the next, now our proposed global and local tasks reduce to DINO (Caron et al., 2021) and EsViT (Li et al., 2021), respectively.

Original: Concept-guided global task.
Revised: Concept-guided global task.

Original: Suppose we have two views { I (1) , I (2) } of an image I , the main idea of our concept-guided global task is to replace I (1) in the DINO loss (Caron et al., 2021) with the image feature f sampled from the concept-feature dictionary, which becomes
Revised: Suppose we have two views { I (1) , I (2) } of an image I , the main idea of our concept-guided global task is to replace I (1) in the DINO loss (Caron et al., 2021) with the image feature f sampled from the concept-feature dictionary, which becomes

Original: [Equation]
Revised: [Equation]

Original: where h t and h s are the projection head of the teacher and student network, respectively, and g s is the student network.
Revised: where h t and h s are the projection head of the teacher and student network, respectively, and g s is the student network.

Original: Intuitively, minimizing the global loss is equivalent to encouraging the similarity of any two different image features with the same concept.
Revised: Intuitively, minimizing the global loss is equivalent to encouraging the similarity of any two different image features with the same concept.

Original: Hence, it can help produce more semantically consistent relational representations, in particular when the concepts stored in the concept-feature dictionary are themselves relational.
Revised: Hence, it can help produce more semantically consistent relational representations, in particular when the concepts stored in the concept-feature dictionary are themselves relational.

Original: Similar inter-class representation learning techniques have been explored before (Wang et al., 2017;
Revised: Similar inter-class representation learning techniques have been explored before (Wang et al., 2017;

Original: Caron et al., 2018).
Revised: Caron et al., 2018).

Original: However, these approaches require a rather complex pre-processing stage, e.g. the images have to be split in terms of the concept before training, making them not directly applicable to existing training pipelines.
Revised: However, these approaches require a rather complex pre-processing stage, e.g. the images have to be split in terms of the concept before training, making them not directly applicable to existing training pipelines.

Original: Rather, with our proposed concept-feature dictionary that dynamically saves & retrieves image features from the running storage, our concept-guided global task becomes a plug-n-play task to existing training pipelines.
Revised: Rather, with our proposed concept-feature dictionary that dynamically saves & retrieves image features from the running storage, our concept-guided global task becomes a plug-n-play task to existing training pipelines.

Original: Besides, our dictionary also allows replaying features relationally, which can help tackle the long-tailed data distribution issue (Shen et al., 2018) in many real-world visual reasoning tasks.

Original: Concept-guided local task.
Revised: Concept-guided local task.

Original: As we mentioned earlier, our concept-guided local task aims at facilitating object-centric learning, by the means of correspondence learning (Liu et al., 2010; Wang et al., 2019).
Revised: As we mentioned earlier, our concept-guided local task aims at facilitating object-centric learning, by the means of correspondence learning (Liu et al., 2010; Wang et al., 2019).

Original: Recent studies have unveiled the possibility of learning correspondence with SSL (Wang et al., 2021c; Li et al., 2021).
Revised: Recent studies have unveiled the possibility of learning correspondence with SSL (Wang et al., 2021c; Li et al., 2021).

Original: However, only low-level correspondence between two augmented ( e.g. rotated) views of an image can be discovered, while the semantic information of objects is missing.
Revised: However, only low-level correspondence between two augmented ( e.g. rotated) views of an image can be discovered, while the semantic information of objects is missing.

Original: To remedy this, we bring concepts to these methods, endowing them the capability of learning semantic correspondence from images.
Revised: To remedy this, we bring concepts to these methods, endowing them the capability of learning semantic correspondence from images.

Original: Speciﬁcally, suppose we have two views { I (1) , I (2) } of an image I , and we also tokenize the image feature into a sequence of N local image tokens.
Revised: Speciﬁcally, suppose we have two views { I (1) , I (2) } of an image I , and we also tokenize the image feature into a sequence of N local image tokens.

Original: Then at the output of ViT, we obtain g t ( I (1) ) =
Revised: Then at the output of ViT, we obtain g t ( I (1) ) =

Original: [ z (1)1 , · · · , z (1) N ] and g s ( I (2) )
Revised: [ z (1)1 , · · · , z (1) N ] and g s ( I (2) )

Original: =
Revised: =

Original: [ z (2)1 , · · · , z (2) N ] , where z denotes the local feature.
Revised: [ z (2)1 , · · · , z (2) N ] , where z denotes the local feature.

Original: Prior work, such as EsViT (Li et al., 2021), relies on the local features g t
Revised: Prior work, such as EsViT (Li et al., 2021), relies on the local features g t

Original: ( I (1) ) and g t ( I (2) ) for the local task.
Revised: ( I (1) ) and g t ( I (2) ) for the local task.

Original: Instead, we replace g t ( I (1) ) with the image feature f retrieved from the concept-feature dictionary with the concept of  I .
Revised: Instead, we replace g t ( I (1) ) with the image feature f retrieved from the concept-feature dictionary using the concept of the image I .

Original: We then split f into multiple local features, i.e. f =
Revised: We then split f into multiple local features, i.e. f =

Original: [ z ( f ) 1 , · · · , z ( f ) N ] and our conceptguided local loss becomes
Revised: [ z ( f ) 1 , · · · , z ( f ) N ] and our concept-guided local loss becomes

Original: [Equation]
Revised: [Equation]

Original: where h t ( · ) , h s ( · ) are the projection heads that map local features to probability distributions 1 .
Revised: where h t ( · ) , h s ( · ) are the projection heads that map local features to probability distributions 1 .

Original: Intuitively, it greedily matches the output between two local regions that have minimal feature distance – bootstrapping the object-level semantic correspondence among images with the same concept.
Revised: Intuitively, it greedily matches the output between two local regions that have minimal feature distance – bootstrapping the object-level semantic correspondence among images with the same concept.

Original: Overall loss.
Revised: Overall loss.

Original: By combining the global and local tasks, we add an auxiliary task loss L aux to the main loss L main ( e.g. cross-entropy loss of the reasoning task).
Revised: By combining the global and local tasks, we add an auxiliary task loss L aux to the main loss L main ( e.g. cross-entropy loss of the reasoning task).

Original: The eventual objective is
Revised: The eventual objective is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where a trade-off weight α is added for better ﬂexibility.
Revised: ) where a trade-off weight α is added for better ﬂexibility.

Original: As we mentioned above, our method will reduce to EsViT, a baseline with non-concept auxiliary tasks, when we use the current input features g t ( I (1) ) instead of f sampled from our dictionary for computing L Global and L Local .
Revised: As we mentioned above, our method will reduce to EsViT, a baseline without concept-guided auxiliary tasks, when we use the current input features g t ( I (1) ) instead of f retrieved from our dictionary for computing L Global and L Local .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments on two challenging visual relational reasoning datasets: HICO (Chao et al., 2015) and GQA (Hudson & Manning, 2019).
Revised: We conduct experiments on two challenging visual relational reasoning datasets: HICO (Chao et al., 2015) and GQA (Hudson & Manning, 2019).

Original: Besides their original non-systematic split, we introduce a systematic split of each dataset to evaluate the systematic generalization of our method.
Revised: Besides their original non-systematic split, we introduce the systematic splits of each dataset to evaluate the systematic generalization of our method.

Original: First, we compare our method against various strong baselines (Mallya & Lazebnik, 2016; Girdhar & Ramanan, 2017; Hudson & Manning, 2018a) on visual relational reasoning, as well as state-of-the-art
Revised: First, we compare our method against various strong baselines (Mallya & Lazebnik, 2016; Girdhar & Ramanan, 2017; Hudson & Manning, 2018a) on visual relational reasoning, as well as stateof-the-art ViTs.

Original: ViTs. Second, we perform the ablation analysis to examine the key components of our method: ViT backbones, concept-feature dictionaries, and auxiliary tasks.
Revised: Second, we perform the ablation analysis to examine the key components of our method: ViT backbones, concept-feature dictionaries, and auxiliary tasks.

Original: Finally, we provide qualitative results to justify the emerging image clustering in terms of concepts and the learned semantic correspondence.
Revised: Finally, we provide qualitative results to justify the emerging image clustering in terms of concepts and the learned semantic correspondence.

Original: Please see more details of all the evaluated tasks in the supplementary material.
Revised: Please see more details of all the evaluated tasks in the supplementary material.

Original: 3. M AIN RESULTS I: H UMAN - OBJECT I NTERACTION R ECOGNITION
Revised: 3. M AIN RESULTS I: H UMAN - OBJECT I NTERACTION R ECOGNITION

Original: Overview.
Revised: Overview.

Original: HICO (Chao et al., 2015) features the human-object interaction (HOI) recognition, i.e. predicting all the possible HOI categories of the input image.
Revised: HICO (Chao et al., 2015) features the human-object interaction (HOI) recognition, i.e. predicting all the possible HOI categories of the input image.

Original: It contains 600 HOI categories with 117 unique actions and 80 object classes.
Revised: It contains 600 HOI categories with 117 unique actions and 80 object classes.

Original: The training set includes 38116 images and the test set includes 9658 images.
Revised: The training set includes 38116 images and the test set includes 9658 images.

Original: For a fair comparison, we follow the standard practice and mainly focus on those previous methods that do not require extra supervision (Fang et al., 2018) or data (Li et al., 2020b; 2019b; Jin et al., 2021).
Revised: For a fair comparison, we follow the standard practice and mainly focus on those previous methods that do not require extra supervision (Fang et al., 2018) or data (Li et al., 2020b; 2019b; Jin et al., 2021).

Original: By default, we choose PVTv2-b2 (Wang et al., 2021b)
Revised: By default, we choose PVTv2-b2 (Wang et al., 2021b)

Original: as the ViT backbone.
Revised: as the ViT backbone.

Original: Regarding the concept-feature dictionary, we use the “most-recent” sampling and a queue length | Q | of 10.
Revised: Regarding the concept-feature dictionary, we use the “most-recent” sampling and a queue length | Q | of 10.

Original: The trade-off weight α in the overall loss is ﬁxed to 0.1.
Revised: The trade-off weight α in the overall loss is ﬁxed to 0.1.

Original: Other hyper-parameters are inherited from DINO (Caron et al., 2021).
Revised: Other hyper-parameters are inherited from DINO (Caron et al., 2021).

Original: Systematic split.
Revised: Systematic split.

Original: The systematic generalization in HICO has been studied before under the name “zero-shot HOI recognition”
Revised: The systematic generalization in HICO has been studied before under the name “zero-shot HOI recognition”

Original: (Shen et al., 2018).
Revised: (Shen et al., 2018).

Original: The main idea is to remove some HOI categories from the training set while ensuring all the single actions and objects can still be kept in the remaining HOI categories.
Revised: The main idea is to remove some HOI categories from the training set while ensuring all the single actions and objects can still be kept in the remaining HOI categories.

Original: We thereby reuse the systematic splits offered by Hou et al.
Revised: We thereby reuse the systematic splits offered by Hou et al.

Original: There are two splits: systematic-easy , where only the rare HOI classes are removed from the training set; systematic-hard , where only the non-rare HOI classes are removed besides the rare ones.
Revised: There are two splits: systematic-easy , where only the rare HOI classes are removed from the training set; systematic-hard , where only the non-rare HOI classes are removed besides the rare ones.

Original: The systematic-hard split contains much fewer training instances and thereby is more challenging.
Revised: The systematic-hard split contains much fewer training instances and thereby is more challenging.

Original: Concepts.
Revised: Concepts.

Original: In HICO, we simply use the 600 HOI categories as our default concepts.
Revised: In HICO, we simply use the 600 HOI categories as our default concepts.

Original: We also report results with other concepts ( e.g. actions, objects) in the ablation study.[Table] Mallya & Lazebnik [Table]
Revised: We also report results with other concepts ( e.g. actions, objects) in the ablation study.

Original: [Table]

Original: & Ramanan

Original: [Table]

Original: [Table]

Original: Results.
Revised: Results.

Original: In Table [Table] we compare our method with several counterparts.
Revised: In Table 1, we compare our method with several counterparts.

Original: The results read that even a simple model with PVTv2-b2 ( 25.4M parameters ) backbone can outperform many previous methods using ResNet-101 ( 44.7M parameters ) and lots of bell and whistles.
Revised: The results read that even a simple model with PVTv2-b2 ( 25.4M parameters ) backbone can outperform many previous methods using ResNet-101 ( 44.7M parameters ) and lots of bell and whistles.

Original: This conﬁrms the great potentials of ViTs in visual relation reasoning.
Revised: This conﬁrms the great potentials of ViTs in visual relation reasoning.

Original: By further adding our global and local tasks, we attain 4-6 mAP gain on original and systematic splits.
Revised: By further adding our global and local tasks, we attain 4-6 mAP gain on original and systematic splits.

Original: We also observe that EsViT (Li et al., 2021), a recently proposed SSL approach, also outperforms the ViT-only baseline.
Revised: We also observe that EsViT (Li et al., 2021), a recently proposed SSL approach, also outperforms the ViT-only baseline.

Original: Therefore, we combine their SSL task and our concept-guided tasks and reach the peak performance (40.12 mAP) on the original HICO split.
Revised: Therefore, we combine their SSL task and our concept-guided tasks and reach the peak performance (40.12 mAP) on the original HICO split.

Original: Although we do not utilize any extra supervison , RelViT+EsViT beats the current state-of-the-art Fang et al.
Revised: Although we do not utilize any extra supervison , RelViT+EsViT beats the current state-of-the-art Fang et al.

Original: (2018) that uses the additional “pose” supervision that does not exist in the HICO dataset.
Revised: (2018) that uses the additional “pose” supervision that does not exist in the HICO dataset.

Original: Overall, we raise the results of a fair counterpart (Girdhar & Ramanan, 2017) that only exploits extra bbox supervision (which is included in HICO) by 16% (34.6 → 40.12) on the original split.
Revised: Overall, we raise the results of a fair counterpart (Girdhar & Ramanan, 2017) that only exploits extra bbox supervision (which is included in HICO) by 16% (34.6 → 40.12) on the original split.

Original: For systematic splits, we raise the results of Hou et al.
Revised: For systematic splits, we raise the results of Hou et al.

Original: (2020) by 43% (26.
Revised: (2020) by 43% (26.

Revised: [Table]

Original: → 37.21) on the systematic-easy split and 7% (21.76 → 23.06) on the systematic-hard split.
Revised: Mallya & Lazebnik (2016) ∗ [Table] Girdhar & Ramanan (2017) ∗ [Table]

Revised: Fang et al.

Revised: (2018) ∗

Revised: [Table]

Revised: Hou et al.

Revised: [Table]

Revised: ViT-only

Revised: [Table]

Revised: EsViT (2021)

Original: Finally, although the gap between systematic and non-systematic split still exists (partly due to the much smaller training set for systematic splits), our method makes signiﬁcant progress, especially on unseen classes (+12.3 mAP on systematic-hard).
Revised: [Table] RelViT (Ours) [Table] RelViT + EsViT (Ours)

Revised: [Table]

Original: This further demonstrates the advantages of our concept-guided ViT in systematic generalization.
Revised: This further demonstrates the advantages of our -guided in systematic generalization.

Original: 3. M AIN RESULTS II: V ISUAL Q UESTION A NSWERING
Revised: 3. M AIN RESULTS II: V ISUAL Q UESTION A NSWERING

Original: Overview.
Revised: Overview.

Original: GQA (Hudson & Manning, 2019) is a recent visual question answering (VQA) dataset with a focus on relational reasoning.
Revised: GQA (Hudson & Manning, 2019) is a recent visual question answering (VQA) dataset with a focus on relational reasoning.

Original: Each question is also labeled with semantics.
Revised: Each question is also labeled with semantics.

Original: By default, it offers both pretrained-CNN grid features and region features obtained through Faster R-CNN (Ren et al., 2015).
Revised: By default, it offers both pretrained-CNN grid features and region features obtained through Faster R-CNN (Ren et al., 2015).

Original: For counterparts, we focus on fair comparisons and therefore exclude those that require massive vision-language pretraining (Li et al., 2019a).
Revised: For counterparts, we focus on fair comparisons and therefore exclude those that require massive vision-language pretraining (Li et al., 2019a).

Original: Notably, we do not use extra supervision, such as scene graph (Krishna et al., 2016).
Revised: Notably, we do not use extra supervision, such as scene graph (Krishna et al., 2016).

Original: The RelViT conﬁguration is almost the same as in
Revised: The RelViT conﬁguration is almost the same as in

Original: HICO, except that we apply the uniform sampling instead as we ﬁnd it empirically works better.
Revised: HICO, except that we apply the uniform sampling instead as we ﬁnd it empirically works better.

Original: We employ MCAN-Small (Yu et al., 2019) as our VQA model and the ImageNet1K-pretrained PVTv2b2 as our vision backbone.
Revised: We employ MCAN-Small (Yu et al., 2019) as our VQA model and the ImageNet1K-pretrained PVTv2b2 as our vision backbone.

Original: The results are reported on the full validation set of GQA.
Revised: The results are reported on the full validation set of GQA.

Original: Systematic split.
Revised: Systematic split.

Original: In GQA, we especially examine the facet of productivity in systematic generalization, i.e. the ability of reasoning with longer reasoning hops (Hupkes et al., 2020).
Revised: In GQA, we especially examine the facet of productivity in systematic generalization, i.e. the ability of reasoning with longer reasoning hops (Hupkes et al., 2020).

Original: To this end, we exploit the extra semantics label associated with the GQA questions.
Revised: To this end, we exploit the extra semantics label associated with the GQA questions.

Original: We observe that the semantics in GQA break down each question into a sequence of “reasoning hops”, where a distribution of reasoning hops can be found in Figure 3.
Revised: We observe that the semantics in GQA break down each question into a sequence of “reasoning hops”, where a distribution of reasoning hops can be found in Figure 3.

Original: See the supplementary material for examples.
Revised: See the supplementary material for examples.

Original: Therefore, our idea is to exclude questions with longer reasoning hops from the training set.
Revised: Therefore, our idea is to exclude questions with longer reasoning hops from the training set.

Original: We end up only keeping questions with less than 5 reasoning hops in the training set.
Revised: We end up only keeping questions with less than 5 reasoning hops in the training set.

Original: We refer to this setting as the systematic split (“Sys.”) in the results.
Revised: We refer to this setting as the systematic split (“Sys.”) in the results.

Original: Concepts.
Revised: Concepts.

Original: Inspired by recent research on vision-language pretraining (Tan & Bansal, 2019; Li et al., 2019a; 2020a), we obtain concepts by parsing the questions into keywords.
Revised: Inspired by recent research on vision-language pretraining (Tan & Bansal, 2019; Li et al., 2019a; 2020a), we obtain concepts by parsing the questions into keywords.

Original: Speciﬁcally, we only keep certain verbs, nouns, and adjectives that contain signiﬁcant meanings (e.g. actions, objects, characteristics, etc), ending up with 1615 concepts.
Revised: Speciﬁcally, we only keep certain verbs, nouns, and adjectives that contain signiﬁcant meanings (e.g. actions, objects, characteristics, etc), ending up with 1615 concepts.

Original: Due to the space limit, readers may ﬁnd more details on concept parsing in the supplementary material.
Revised: Due to the space limit, readers may ﬁnd more details on concept parsing in the supplementary material.

Original: Results.
Revised: Results.

Original: We report the comparison results on the original and systematic splits in Table 2.
Revised: We report the comparison results on the original and systematic splits in Table 2.

Original: The main goal of our experiments on GQA mainly is to verify if our method can help reduce the gap[Table] between models using backbone features only and models using additional bbox features (with dense object detectors).
Revised: The main goal of our experiments on GQA mainly is to verify if our method can help reduce the gap between models using backbone features only and models using additional bbox features (with dense object detectors).

Original: Besides, we also examine to which extent our method can improve systematic generalization.
Revised: Besides, we also examine to which extent our method can improve systematic generalization.

Original: Firstly, we observe that using ViT can largely alleviate the aforementioned gap (51.
Revised: Firstly, we observe that using ViT can largely alleviate the aforementioned gap (51.

Original: → 56.62), suggesting that the object-centric representations emerge in ViTs.
Revised: → 56.62), suggesting that the object-centric representations emerge in ViTs.

Original: It implies the potential of using ViTs in eliminating the need for external object detectors.
Revised: It implies the potential of using ViTs in eliminating the need for external object detectors.

Original: By further adding our proposed auxiliary tasks, we achieve the peak performance and raise the results of MCAN-Small w/o bbox features by 13% (51.1 → 57.87) on the original split and 18% (30.12 → 35.48) on the systematic split.
Revised: By further adding our proposed auxiliary tasks, we achieve the peak performance and raise the results of MCAN-Small w/o bbox features by 13% (51.1 → 57.87) on the original split and 18% (30.12 → 35.48) on the systematic split.

Original: Without any detection pretraining or bbox features , our method achieves very close results to MCAN-Small w/ bbox features on both two splits.
Revised: Without any detection pretraining or bbox features , our method achieves very close results to MCAN-Small w/ bbox features on both two splits.

Revised: The additional results in appendix demonstrate

Original: We hypothesize that the marginal gap could be further eliminated if we apply larger backbone models (PVTv2-b2 has much fewer parameters than ResNet-101).
Revised: [Table] Testing Training that the marginal gap could be further eliminated if we apply larger backbone models (PVTv2-b has much fewer parameters than ResNet-101)

Original: We leave this exploration as future work.
Revised: [Table]

Original: 3. W HY DO OUR AUXILIARY TASKS WORK ?
Revised: 3. W HY DO OUR AUXILIARY TASKS WORK ?

Original: The results in the previous section suggest that RelViT outperforms its counterparts on the challenging relational reasoning tasks.
Revised: The results in the previous section suggest that RelViT outperforms its counterparts on the challenging relational reasoning tasks.

Original: Now we would like to provide more insights into our method design by answering the question: why do our auxiliary tasks work?
Revised: Now we would like to provide more insights into our method design by answering the question: why do our auxiliary tasks work?

Original: To this end, we perform a diverse set of analyses on accessing the impact of key components in RelViT .
Revised: To this end, we perform a diverse set of analyses on accessing the impact of key components in RelViT .

Original: We also qualitatively justify the intuitions of two auxiliary tasks.
Revised: We also qualitatively justify the intuitions of two auxiliary tasks.

Original: These results are reported on the HICO dataset.
Revised: These results are reported on the HICO dataset.

Original: 3.3. A BLATION STUDY
Revised: 3.3. A BLATION STUDY

Original: Different ViT architectures.
Revised: Different ViT architectures.

Original: The ﬁrst aspect we examine is the ViT architecture.
Revised: The ﬁrst aspect we examine is the ViT architecture.

Original: Besides the default choice on PVTv2-b2, we test our method with the original ViT-S/16 (Dosovitskiy et al., 2020) and another prevalent architecture Swin-Small (Liu et al., 2021).
Revised: Besides the default choice on PVTv2-b2, we test our method with the original ViT-S/16 (Dosovitskiy et al., 2020) and another prevalent architecture Swin-Small (Liu et al., 2021).

Original: The results are presented in Figure 4a and Figure 4b, respectively.
Revised: The results are presented in Figure 4a and Figure 4b, respectively.

Original: These two architectures can both beneﬁt from our auxiliary tasks and we have similar advantages over counterparts as in the default setting, which conﬁrms our compatibility to various ViT variants.
Revised: These two architectures can both beneﬁt from our auxiliary tasks and we have similar advantages over counterparts as in the default setting, which conﬁrms our compatibility to various ViT variants.

Original: Full quantitative results are provided in the supplementary.
Revised: Full quantitative results are provided in the supplementary.

Original: Implementation of concept-feature dictionary.
Revised: Implementation of concept-feature dictionary.

Original: We conduct ablations on three facets of conceptfeature dictionary: the choice of concepts, sampling tactics, and the size of queue | Q | .
Revised: We conduct ablations on three facets of conceptfeature dictionary: the choice of concepts, sampling tactics, and the size of queue | Q | .

Original: In Figure 4c, we compare three different concept choices: actions, objects, and HOIs with our best model.
Revised: In Figure 4c, we compare three different concept choices: actions, objects, and HOIs with our best model.

Original: The results suggest that all three choices can bring improvement to the baseline without any feature queue (denoted as “None”) while using HOIs and objects brings larger improvement.
Revised: The results suggest that all three choices can bring improvement to the baseline without any feature queue (denoted as “None”) while using HOIs and objects brings larger improvement.

Original: We hypothesize that the proposed auxiliary tasks need more “delicate” concepts to guide the ViT training but actions in
Revised: We hypothesize that the proposed auxiliary tasks need more “delicate” concepts to guide the ViT training but actions in

Original: HICO tend to be vague and even ambiguous (Shen et al., 2018).
Revised: HICO tend to be vague and even ambiguous (Shen et al., 2018).

Original: Therefore, albeit the consistent advantages of our method in terms of different concept selections, a careful design of concept space could still be pivotal to achieve the peak performance in relational reasoning.
Revised: Therefore, albeit the consistent advantages of our method in terms of different concept selections, a careful design of concept space could still be pivotal to achieve the peak performance in relational reasoning.

Original: Furthermore, we show the interplay between sampling strategies and queue size | Q | in Figure 4d.
Revised: Furthermore, we show the interplay between sampling strategies and queue size | Q | in Figure 4d.

Original: Interestingly, | Q | has a much smaller impact on the performance with the “most-recent” sampling than that with the uniform sampling .
Revised: Interestingly, | Q | has a much smaller impact on the performance with the “most-recent” sampling than that with the uniform sampling .

Original: As we mentioned in Section 2.2, the uniform sampling could help with more diverse features but could also elicit unstable training.
Revised: As we mentioned in Section 2.2, the uniform sampling could help with more diverse features but could also elicit unstable training.

Original: Larger | Q | makes the two consequences in the uniform sampling more prominent, thus causing worse performance when stable training is the bottleneck ( e.g. in a small dataset like HICO).
Revised: Larger | Q | makes the two consequences in the uniform sampling more prominent, thus causing worse performance when stable training is the bottleneck ( e.g. in a small dataset like HICO).

Original: Rather, the “most-recent” sampling can be less sensitive to | Q | as only the recent features could be sampled even when | Q | is large.
Revised: Rather, the “most-recent” sampling can be less sensitive to | Q | as only the recent features could be sampled even when | Q | is large.

Original: Auxiliary tasks.
Revised: Auxiliary tasks.

Original: In Figure 4e, we show the results of only adding our global or local task in L aux .
Revised: In Figure 4e, we show the results of only adding our global or local task in L aux .

Original: Surprisingly, just using the local task is enough to deliver competitive results in the HICO task.
Revised: Surprisingly, just using the local task is enough to deliver competitive results in the HICO task.

Original: This suggests that the real bottleneck in ViTs seems to be better object-centric representations, as our local task is designed for this.
Revised: This suggests that the real bottleneck in ViTs seems to be better object-centric representations, as our local task is designed for this.

Original: Nonetheless, adding our global task can still elicit clear advantages over other counterparts that do not exploit concept-guided learning.[Figure] [Figure]
Revised: Nonetheless, adding our global task can still elicit clear advantages over other counterparts that do not exploit concept-guided learning.

Original: Robustness to α .
Revised: Robustness to α .

Original: We sweep the trade-off weight α from 0.02 to 0.5 and report the results in Figure 4f, where solid and dash represent our method and the baseline, respectively.
Revised: We sweep the trade-off weight α from 0.02 to 0.5 and report the results in Figure 4f, where solid and dash represent our method and the baseline, respectively.

Original: It is observed that adding the proposed auxiliary tasks always achieves better performances than the baseline, indicating our method is robust to hyper-parameters.
Revised: It is observed that adding the proposed auxiliary tasks always achieves better performances than the baseline, indicating our method is robust to hyper-parameters.

Original: Moreover, the improvements become slightly more signiﬁcant when α is relatively large (but not too large).
Revised: Moreover, the improvements become slightly more signiﬁcant when α is relatively large (but not too large).

Original: The peak performances in different splits all appear around α = 0 .
Revised: The peak performances in different splits all appear around α = 0 .

Original: 1 , which we thus use as our default choice.
Revised: 1 , which we thus use as our default choice.

Revised: [Figure]

Revised: [Figure]

Original: 3.3. Q UALITATIVE INSPECTION
Revised: 3.3. Q UALITATIVE INSPECTION

Original: Features vs. concepts.
Revised: Features vs. concepts.

Original: To further justify whether our global task can truly facilitate the learned representation to be more relational, we illustrate the learned output features (max-pooling on all the output tokens) by t-SNE visualization in Figure 5.
Revised: To further justify whether our global task can truly facilitate the learned representation to be more relational, we illustrate the learned output features (max-pooling on all the output tokens) by t-SNE visualization in Figure 5.

Original: Different colors correspond to different HOI categories, i.e. the concepts we used in RelViT.
Revised: Different colors correspond to different HOI categories, i.e. the concepts we used in RelViT.

Original: The results read that more clusters can be identiﬁed over the image features extracted by RelViT; therefore our concept-guided global task can encourage the learned features to be more discriminative regarding the relational concepts than the baselines.
Revised: The results read that more clusters can be identiﬁed over the image features extracted by RelViT; therefore our concept-guided global task can encourage the learned features to be more discriminative regarding the relational concepts than the baselines.

Original: Semantic correspondence.
Revised: Semantic correspondence.

Original: We also probe the learned semantic correspondence that could be encouraged by our local task, by intuition.
Revised: We also probe the learned semantic correspondence that could be encouraged by our local task, by intuition.

Original: We aim at comparing the correspondence extracted from a model trained with different auxiliary tasks, i.e. no auxiliary task, no-concept auxiliary tasks, and our auxiliary tasks.
Revised: We aim at comparing the correspondence extracted from a model trained with different auxiliary tasks, i.e. no auxiliary task, no-concept auxiliary tasks, and our auxiliary tasks.

Original: We consider two settings: 1) semantic setting (two images that belong to the same concept, e.g. both contains a cat), and 2) non-semantic setting (two views of the same image).
Revised: We consider two settings: 1) semantic setting (two images that belong to the same concept, e.g. both contains a cat), and 2) non-semantic setting (two views of the same image).

Original: Results in Figure 6 highlight the high-similarity matches.
Revised: Results in Figure 6 highlight the high-similarity matches.

Original: Although our method and non-concept baseline (EsViT) both work ﬁne in the non-semantic setting, our method can identify the semantic correspondence on more objects thanks to the concept guidance.
Revised: Although our method and non-concept baseline (EsViT) both work ﬁne in the non-semantic setting, our method can identify the semantic correspondence on more objects thanks to the concept guidance.

Original: Not surprisingly, baseline w/o any auxiliary task (ViT-only) performs the worst as it may suffer from over-smoothness (Gong et al., 2021) and lose all the meaningful spatial information after ﬁne-tuning on the target task.
Revised: Not surprisingly, baseline w/o any auxiliary task (ViT-only) performs the worst as it may suffer from over-smoothness (Gong et al., 2021) and lose all the meaningful spatial information after ﬁne-tuning on the target task.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Systematic generalization in visual reasoning.
Revised: Systematic generalization in visual reasoning.

Original: Systematic generalization (Hupkes et al., 2020;
Revised: Systematic generalization (Hupkes et al., 2020;

Original: Bahdanau et al., 2018) characterizes to which extent a learning agent can identify and exploit the underlying entities and relations of the training data, and generalize to novel combinations and longer reasoning hops.
Revised: Bahdanau et al., 2018) characterizes to which extent a learning agent can identify and exploit the underlying entities and relations of the training data, and generalize to novel combinations and longer reasoning hops.

Original: There has been extensive research on inspecting and tackling systematic generalization in visual reasoning (Johnson et al., 2017; Kim & Mnih, 2018; Higgins et al., 2016; Kuhnle & Copestake, 2017).
Revised: There has been extensive research on inspecting and tackling systematic generalization in visual reasoning (Johnson et al., 2017; Kim & Mnih, 2018; Higgins et al., 2016; Kuhnle & Copestake, 2017).

Original: However, most of them only focus on controlled and synthetic domains (Ruis et al., 2020;
Revised: However, most of them only focus on controlled and synthetic domains (Ruis et al., 2020;

Original: Zhang et al., 2019; Barrett et al., 2018; Xie et al., 2021; Nie et al., 2020), while the open-ended real-world domains are largely neglected with very few exceptions (Shen et al., 2018; Teney et al., 2020).
Revised: Zhang et al., 2019; Barrett et al., 2018; Xie et al., 2021; Nie et al., 2020; Hong et al., 2021), while the open-ended real-world domains are largely neglected with very few exceptions (Shen et al., 2018; Teney et al., 2020).

Original: In this paper, we tackle systematic generalization in visual relational reasoning with natural images, thereby ﬁlling the gap between synthetic and real domains.
Revised: In this paper, we tackle systematic generalization in visual relational reasoning with natural images, thereby ﬁlling the gap between synthetic and real domains.

Original: Object-centric and relational representations.
Revised: Object-centric and relational representations.

Original: Many seminal research reveals that ML models can beneﬁt from object-centric and relational representations with better sample efﬁciency and generalization (Farhadi & Sadeghi, 2013;
Revised: Many seminal research reveals that ML models can beneﬁt from object-centric and relational representations with better sample efﬁciency and generalization (Farhadi & Sadeghi, 2013;

Original: Ding et al., 2020; Mrowca et al., 2018).
Revised: Ding et al., 2020; Mrowca et al., 2018).

Original: However, obtaining such representations from unstructured inputs, i.e. raw images, still remains challenging (Greff et al., 2019; Locatello et al., 2020; 2019).
Revised: However, obtaining such representations from unstructured inputs, i.e. raw images, still remains challenging (Greff et al., 2019; Locatello et al., 2020; 2019; Yu et al., 2021).

Original: Prevalent approaches adopt a latent variable model to explicitly infer the foreground-background split as well as objects & relations (Eslami et al., 2016; Lin et al., 2020; Zhu & Mumford, 2007), while recent ﬁndings suggest that they can be an emerging property of transformers trained with self-supervised objectives (Caron et al., 2021; Li et al., 2021).
Revised: Prevalent approaches adopt a latent variable model to explicitly infer the foreground-background split as well as objects & relations (Eslami et al., 2016; Lin et al., 2020; Zhu & Mumford, 2007), while recent ﬁndings suggest that they can be an emerging property of transformers trained with self-supervised objectives (Caron et al., 2021; Li et al., 2021).

Original: Our goal aligns better with the later regime, as it enables implicit representations and thus could be more versatile and efﬁcient.
Revised: Our goal aligns better with the later regime, as it enables implicit representations and thus could be more versatile and efﬁcient.

Original: A key difference is that these methods do not exploit concepts in reasoning benchmarks, making them less capable of learning semantic representations.
Revised: A key difference is that these methods do not exploit concepts in reasoning benchmarks, making them less capable of learning semantic representations.

Original: Self-supervised learning for ViTs.
Revised: Self-supervised learning for ViTs.

Original: The recent resurgence on self-supervised learning (SSL) of image models has delivered impressive results on many few-shot or zero-shot tasks (Oord et al., 2018).
Revised: The recent resurgence on self-supervised learning (SSL) of image models has delivered impressive results on many few-shot or zero-shot tasks (Oord et al., 2018).

Original: From a high-level view, these approaches can be categorized into contrastive (He et al., 2020; Chen et al., 2020) and non-contrastive (Chen & He, 2021).
Revised: From a high-level view, these approaches can be categorized into contrastive (He et al., 2020; Chen et al., 2020) and non-contrastive (Chen & He, 2021).

Original: However, not all SSL avenues work well with vision transformers (ViTs) and some delicate design may be needed.
Revised: However, not all SSL avenues work well with vision transformers (ViTs) and some delicate design may be needed.

Original: Caron et al.
Revised: Caron et al.

Original: (2021) found their non-contrastive learning objective (DINO) manifested better quantitative results and emerging properties on ViTs.
Revised: (2021) found their non-contrastive learning objective (DINO) manifested better quantitative results and emerging properties on ViTs.

Original: Chen et al.
Revised: Chen et al.

Original: (2021) brought similar results on contrastive SSL.
Revised: (2021) brought similar results on contrastive SSL.

Original: Li et al.
Revised: Li et al.

Original: (2021) further introduced patch-level SSL objective to ViTs for dense prediction tasks.
Revised: (2021) further introduced patch-level SSL objective to ViTs for dense prediction tasks.

Original: In this paper, instead of proposing a new SSL approach, we make better use of concepts for ViT training, which can be directly applied to the existing SSL objectives for better visual reasoning.
Revised: In this paper, instead of proposing a new SSL approach, we make better use of concepts for ViT training, which can be directly applied to the existing SSL objectives for the improved visual reasoning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, our goal is to seek a better inductive bias for visual relational reasoning, especially on real-world data.
Revised: In this paper, our goal is to seek a better inductive bias for visual relational reasoning, especially on real-world data.

Original: We found ViTs to be a promising candidate due to their potential on relational reasoning, object-centric learning, and systematic generalization.
Revised: We found ViTs to be a promising candidate due to their potential on relational reasoning, object-centric learning, and systematic generalization.

Original: We further presented RelViT, a simple yet efﬁcient method for exploiting concepts in the visual relational reasoning tasks to boost the performances of ViTs.
Revised: We further presented RelViT, a simple yet efﬁcient method for exploiting concepts in the visual relational reasoning tasks to boost the performances of ViTs.

Original: In speciﬁc, we proposed two auxiliary tasks in RelViT : a global task for semantically consistent relational representation, and a local task for learning object-centric semantic correspondence.
Revised: In speciﬁc, we proposed two auxiliary tasks in RelViT : a global task for semantically consistent relational representation, and a local task for learning object-centric semantic correspondence.

Original: These two tasks are made possible through the use of our proposed concept-feature dictionary.
Revised: These two tasks are made possible through the use of our proposed concept-feature dictionary.

Original: RelViT largely outperforms other counterparts on two challenging visual relational reasoning benchmarks.
Revised: RelViT largely outperforms other counterparts on two challenging visual relational reasoning benchmarks.

Original: While we mainly focus on extending ViTs to visual reasoning using auxiliary tasks, further exploration of combining our work with architectural modiﬁcation over ViTs to enable better generalization could be a new direction for future work. 
Revised: While we mainly focus on extending ViTs to visual reasoning using auxiliary tasks, further exploration of combining our work with architectural modiﬁcation over ViTs to enable better generalization could be a new direction for future work. A A FORMAL DESCRIPTION OF LEARNING IN R EL V I T Algorithm 1 formally depicts the execution ﬂow of RelViT. Algorithm

Original: A E XTRA DETAILS ON R EL V I T A. I NPUT PIPELINE
Revised: B E XTRA DETAILS ON R EL V I T B. I NPUT PIPELINE

Original: We adopt the following data augmentation pipeline for the generating the additional views for our two auxiliary tasks
Revised: We adopt the following data augmentation pipeline for the generating the additional views for our two auxiliary tasks

Original: Randomly crop and resize the image into (224 , 224) with scale ratio (0 . 2 , 1 . 0) ; 2.
Revised: Randomly crop and resize the image into (224 , 224) with scale ratio (0 . 2 , 1 . 0) ; 2.

Original: Randomly jitter the color of the image on brightness, contrast saturation and hue with probability of (0 . 4 , 0 . 4 , 0 . 4 , 0 . 1) , respectively; 3.
Revised: Randomly jitter the color of the image on brightness, contrast saturation and hue with probability of (0 . 4 , 0 . 4 , 0 . 4 , 0 . 1) , respectively; 3.

Original: Randomly turn the image into gray scale with probability 0 . 2 ; 4.
Revised: Randomly turn the image into gray scale with probability 0 . 2 ; 4.

Original: Randomly apply Gaussian blur with kernel size 23 and sigma (0 . 1 , 2 . 0) and probability 0 .
Revised: Randomly apply Gaussian blur with kernel size 23 and sigma (0 . 1 , 2 . 0) and probability 0 .

Original: Randomly ﬂip the image horizontally.
Revised: Randomly ﬂip the image horizontally.

Original: Notably, we apply a random crop operation to ensure that all the input images for our auxiliary tasks contain the same number of patches.
Revised: Notably, we apply a random crop operation to ensure that all the input images for our auxiliary tasks contain the same number of patches.

Original: A. H YPER - PARAMETERS AND BASELINES
Revised: B. H YPER - PARAMETERS AND BASELINES

Original: Table 3 summarizes the hyper-parameters used by RelViT.
Revised: Table 3 summarizes the hyper-parameters used by RelViT.

Original: We inherit most of the parameters from
Revised: We inherit most of the parameters from

Original: DINO (Caron et al., 2021)
Revised: DINO (Caron et al., 2021).

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Table 4 summarizes the key details about the loss implementation of different baselines and RelViT .L Global
Revised: Table 4 summarizes the key details about the loss implementation of different baselines and RelViT .

Original: L Local
Revised: [Table]

Original: Compare student(aug(img)) with
Revised: Compare student(aug(img)) with

Original: DINO x teacher(aug(img)) EsViT x x teacher(aug(img)) RelViT x x queues[concept(img)].pop() RelViT + EsViT x x teacher(aug(img)) and queues[concept(img)].pop()
Revised: [Table]

Original: B E XTRA DETAILS ON THE DATASETS B. HICO B.1. O RIGINAL AND SYSTEMATIC SPLITS
Revised: C E XTRA DETAILS ON THE DATASETS C. HICO C.1. O RIGINAL AND SYSTEMATIC SPLITS

Original: Besides the ofﬁcial training/testing split, we adopt the splits for systematic generalization presented in (Hou et al., 2020).
Revised: Besides the ofﬁcial training/testing split [Table] we adopt the splits for systematic generalization presented in (Hou et al., 2020).

Original: It offers two splits that follow different strategies to select held-out HOI categories.
Revised: It offers two splits that follow different strategies to select held-out HOI categories.

Original: Systematic-easy only select rare HOI categories (with less than 10 training samples), while Systematic-hard select non-rare categories instead.
Revised: Systematic-easy only select rare HOI categories (with less than 10 training samples), while Systematic-hard select non-rare categories instead.

Original: Therefore, the training set of Systematichard will contain much fewer samples and become more challenging.
Revised: Therefore, the training set of Systematichard will contain much fewer samples and become more challenging.

Original: Some basic statistics of these training/testing splits can be found in Table 5.[Table] [Table] [Table]
Revised: Some basic statistics of these training/testing splits can be found in Table 5.

Original: [Table]
Revised: [Table]

Original:  I MPLEMENTATION OF L main
Revised: C.1. I MPLEMENTATION OF L main

Original: In HICO, there might be multiple HOIs for a single image.
Revised: In HICO, there might be multiple HOIs for a single image.

Original: We, therefore, formulate the HOI prediction task as a multi-class classiﬁcation problem.
Revised: We, therefore, formulate the HOI prediction task as a multi-class classiﬁcation problem.

Original: Speciﬁcally, the model makes 600 binary classiﬁcations and L main in equation 3 is a binary cross-entropy loss.
Revised: Speciﬁcally, the model makes 600 binary classiﬁcations and L main in equation 3 is a binary cross-entropy loss.

Original: B. GQA B.2. O RIGINAL AND SYSTEMATIC SPLITS
Revised: C. GQA C.2. O RIGINAL AND SYSTEMATIC SPLITS

Original: We introduce a systematic split for the GQA dataset that is based on reasoning hops.
Revised: We introduce a systematic split for the GQA dataset that is based on reasoning hops.

Original: Speciﬁcally, we remove those questions that have more than 4 reasoning hops from the training set.
Revised: Speciﬁcally, we remove those questions that have more than 4 reasoning hops from the training set.

Original: Some basic statistics of these training/testing splits can be found in Table 6.
Revised: Some basic statistics of these training/testing splits can be found in Table 6.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B.2. R EASONING HOPS
Revised: .2. R EASONING HOPS

Original: Since all the questions and answers in the GQA dataset are synthetic, it additionally provides “semantics” that characterizes the reasoning procedure that generates the answer from a question and a visual scene.
Revised: Since all the questions and answers in the GQA dataset are synthetic, it additionally provides “semantics” that characterizes the reasoning procedure that generates the answer from a question and a visual scene.

Original: These semantics are composed of multiple “reasoning primitives” that act like functions: receiving arguments and generating output for the next reasoning step.
Revised: These semantics are composed of multiple “reasoning primitives” that act like functions: receiving arguments and generating output for the next reasoning step.

Original: It is believed they can reﬂect whether a question will require complex multi-hop reasoning – a pivotal angle of systematic generalization.
Revised: It is believed they can reﬂect whether a question will require complex multi-hop reasoning – a pivotal angle of systematic generalization.

Original: Therefore we develop our systematic split with it.
Revised: Therefore we develop our systematic split with it.

Original: Table 7 provides a few examples on semantics.
Revised: Table 7 provides a few examples on semantics.

Original: B.2. C ONCEPT PARSING

Original: We obtain the concepts in the GQA dataset by parsing the questions into word tokens.

Original: Speciﬁcally, we construct a set of concepts that contain nouns, verbs, and adjectives that are with signiﬁcant meaning.

Original: We also manually ﬁlter some ambiguous words from this set.

Original: The resulting concept set contains 1615 concepts.

Original: Question
Revised: Question

Original: Semantics (Reasoning hops)
Revised: Semantics (Reasoning hops)

Original: Is the pizza with the pepper small and covered?
Revised: Is the pizza with the pepper small and covered?

Original: relate([0], pizza, with, s(1130674)); filter([1], pizza); verify([2], covered); verify size([2], small); and([3,4]);
Revised: relate([0], pizza, with, s(1130674)); filter([1], pizza); verify([2], covered); verify size([2], small); and([3,4]);

Original: Do you see any tablecloths or dressers?
Revised: Do you see any tablecloths or dressers?

Original: select([], dreser); exist([0], ?); select([], tablecloth); exist([2], ?); or([1, 3], ?);
Revised: select([], dreser); exist([0], ?); select([], tablecloth); exist([2], ?); or([1, 3], ?);

Original: Are there microwave ovens to the right of the appliance near the window?
Revised: Are there microwave ovens to the right of the appliance near the window?

Original: select([], window); relate([0], appliance, near, s(1297947)) relate([1], microwave, right, s(1297947)); exist([2], ?);
Revised: select([], window); relate([0], appliance, near, s(1297947)) relate([1], microwave, right, s(1297947)); exist([2], ?);

Revised: (reasoning hops) in GQA dataset.

Revised: C.2. C ONCEPT PARSING

Revised: We obtain the concepts in the GQA dataset by parsing the questions into word tokens.

Revised: Speciﬁcally, we construct a set of concepts that contain nouns, verbs, and adjectives that are with signiﬁcant meaning.

Revised: We also manually ﬁlter some ambiguous words from this set.

Original:  We use the python nltk package to process the question.
Revised: The resulting concept set contains 1615 concepts. We use the python nltk package to process the question.

Original: The parsing procedure starts with part-ofspeech tagging, where we only keep nouns ( NN ), verbs ( VB ) and adjectives ( JJ ).
Revised: The parsing procedure starts with part-ofspeech tagging, where we only keep nouns ( NN ), verbs ( VB ) and adjectives ( JJ ).

Original: Then we lemmatize the remaining words to obtain the minimal form of them.
Revised: Then we lemmatize the remaining words to obtain the minimal form of them.

Original: Finally, we remove those that do not present in the pre-selected concept list.
Revised: Finally, we remove those that do not present in the pre-selected concept list.

Original: Additionally, we skip questions with “No” as the answer as the question may be unrelated to the image.
Revised: Additionally, we skip questions with “No” as the answer as the question may be unrelated to the image.

Original: We provide the statistics of the concepts in GQA in Table 8.
Revised: We provide the statistics of the concepts in GQA in Table 8.

Original: The number of associated questions of all the 1615 concepts and a histogram on the number of concepts for each question is presented in Figure 7a and Figure 7b, respectively.
Revised: The number of associated questions of all the 1615 concepts and a histogram on the number of concepts for each question is presented in Figure 7a and Figure 7b, respectively.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: B.2. I MPLEMENTATION OF L main
Revised: C.2. I MPLEMENTATION OF L main

Original: GQA is formulated as a classiﬁcation problem, i.e. the learner needs to select an answer from the pre-deﬁned answer set; thus L main in equation 3 is a cross-entropy loss.
Revised: GQA is formulated as a classiﬁcation problem, i.e. the learner needs to select an answer from the pre-deﬁned answer set; thus L main in equation 3 is a cross-entropy loss.

Original: C F ULL QUANTITATIVE RESULTS ON ABLATION STUDIES
Revised: D F ULL QUANTITATIVE RESULTS ON ABLATION STUDIES

Original: We provide the full quantitative results of the ablation studies in Table 9.
Revised: We provide the full quantitative results of the ablation studies in Table 9.

Original: Method Orig. Sys.-easy Sys.-hard
Revised: [Table]

Original: ViT-only 27. EsViT 30. RelViT 31. EsViT+RelViT 33.
Revised: [Table]

Original: (a) RelViT with ViT-S/ Method Orig. Sys.-easy Sys.-hard
Revised: [Table]

Original: ViT-only 36. EsViT 38. RelViT 39. EsViT+RelViT 39.
Revised: [Table]

Original: (b) RelViT with Siwn-Small Concepts Orig. Sys.-easy Sys.-hard None 35.
Revised: [Table]

Original: Actions 38. Objects 39. HOIs 40. (c) Different choice of concepts | Q | Most-recent Uniform 39.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: As we discussed in Section 2.2, the use of concept-feature dictionary can also allow feature replay, which could help mitigate the long-tail distribution issue in a dataset like HICO. As shown in Table 10, our method makes signiﬁcant progress on rare classes, which supports our insight.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: One insight into the design of RelViT is that it can offer better object-centric representations without explicit object-detection supervision or RoI pooling, thanks to the patch-based input of ViT and our concept-guided local task. To further justify this point, we provide results on combing ResNet/ViT/RelViT with explicit RoI pooling operations using ground truth bounding boxes.
Revised: Local

Original: As shown in Table 11, the performance gap between using RoI features and not using them is much smaller for ViTs (ViT, EsViT, RelViT, RelViT+EsViT), compared to ResNet.
Revised: [Table] [Table]

Original: These observations are consistent with Kim et al. (2021), where a ViT-based vision-language model could perform comparably to previous approaches without region features. Furthermore, methods with RelViT have the smallest gap between using and not using RoI features, conﬁrming that our method can better learn object-centric representations and thus eliminate the need for external object detectors.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: It is clear that with a better and larger ViT backbone, our method largely outperforms the state-ofthe-art methods on the HICO and GQA datasets in both original and systematic settings
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D. R EL V I T WITH LARGER BACKBONE MODELS
Revised: E A DDITIONAL RESULTS E. R EL V I T WITH LARGER BACKBONE MODELS

Original: As we mentioned in Section 3.
Revised: As we mentioned in Section 3.

Original: [Table]
Revised: [Table]

Original: the ViT backbone we use (PVTv2-b2) only has 25.4M parameters, even less than the commonly-used ResNet-101 ( 44.7M parameters).
Revised: the ViT backbone we use (PVTv2-b2) only has 25.4M parameters, even less than the commonly-used ResNet-101 ( 44.7M parameters).

Original: Therefore
Revised: Therefore

Original: [Table]
Revised: [Table]

Original: we testify RelViT with larger state-of-the-art ViT backbones: PVTv2-b3 ( 45.2M parameters) and Swin-base ( 88M parameters) (Liu et al., 2021) and provide the results on HICO and GQA below:
Revised: we testify RelViT with larger state-of-the-art ViT backbones: PVTv2-b3 ( 45.2M parameters) and Swin-base ( 88M parameters) (Liu et al., 2021) and provide the results on HICO and GQA below:

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: E E XTRA DISCUSSION ON THE RELATED WORK E. D ISCUSSION ON THE MEMORY BANK MECHANISM
Revised: F E XTRA DISCUSSION ON THE RELATED WORK F. D ISCUSSION ON THE MEMORY BANK MECHANISM

Original: Intuitively, the idea of using the concept-feature dictionary to the ViT training could be similar to the memory bank mechanism in MoCo (He et al., 2020), where the features are stored in a queue for replaying later.
Revised: Intuitively, the idea of using the concept-feature dictionary to the ViT training could be similar to the memory bank mechanism in MoCo (He et al., 2020), where the features are stored in a queue for replaying later.

Original: However, the difference is also clear: we have multiple queues that are indexed by concept codes while MoCo only has a single queue.
Revised: However, the difference is also clear: we have multiple queues that are indexed by concept codes while MoCo only has a single queue.

Original: Similar use of memory bank can also be found in Wu et al.
Revised: Similar use of memory bank can also be found in Wu et al.

Original: ; Tian et al.
Revised: ; Tian et al.

Original: but they follow MoCo, and therefore it is used for providing negative samples when computing the self-supervised contrastive learning loss.
Revised: but they follow MoCo, and therefore it is used for providing negative samples when computing the self-supervised contrastive learning loss.

Original: Rather, our conceptfeature dictionary is designed to make better use of the concept supervision via our concept-guided global and local losses to improve the performance on visual relational reasoning.
Revised: Rather, our conceptfeature dictionary is designed to make better use of the concept supervision via our concept-guided global and local losses to improve the performance on visual relational reasoning.

Original: E. C OMPARISON TO MAC (H UDSON & M ANNING , 2018 B )
Revised: F. C OMPARISON TO MAC (H UDSON & M ANNING , 2018 B )

Original: Here we highlight the difference between our concept-feature dictionary and the knowledge base in MAC (Hudson & Manning, 2018b): The knowledge base in MAC is used during a single VQA reasoning pass (i.e. it will be cleared & initialized with the new image features (visual knowledge) whenever a new <image, question> pair comes), and thus is used in both the training and testing time for the VQA reasoning.
Revised: Here we highlight the difference between our concept-feature dictionary and the knowledge base in MAC (Hudson & Manning, 2018b): The knowledge base in MAC is used during a single VQA reasoning pass (i.e. it will be cleared & initialized with the new image features (visual knowledge) whenever a new <image, question> pair comes), and thus is used in both the training and testing time for the VQA reasoning.

Original: However, the concept-feature dictionary in RelViT is used to store & retrieve features according to the concept of the current input image and help compute our local and global losses that encourage learning better representations.
Revised: However, the concept-feature dictionary in RelViT is used to store & retrieve features according to the concept of the current input image and help compute our local and global losses that encourage learning better representations.

Original: Therefore, we use it in the training time only as these two losses won’t be computed & optimized during testing.
Revised: Therefore, we use it in the training time only as these two losses won’t be computed & optimized during testing.