Original: L EARNING T RANSFERABLE R EWARD FOR Q UERY O BJECT L OCALIZATION WITH P OLICY A DAPTATION
Revised: L EARNING T RANSFERABLE R EWARD FOR Q UERY O BJECT L OCALIZATION WITH P OLICY A DAPTATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a reinforcement learning based approach to query object localization , for which an agent is trained to localize objects of interest speciﬁed by a small exemplary set.
Revised: We propose a reinforcement learning based approach to query object localization , for which an agent is trained to localize objects of interest specified by a small exemplary set.

Original: We learn a transferable reward signal formulated using the exemplary set by ordinal metric learning.
Revised: We learn a transferable reward signal formulated using the exemplary set by ordinal metric learning.

Original: Our proposed method enables test-time policy adaptation to new environments where the reward signals are not readily available, and outperforms ﬁne-tuning approaches that are limited to annotated images.
Revised: Our proposed method enables test-time policy adaptation to new environments where the reward signals are not readily available, and outperforms fine-tuning approaches that are limited to annotated images.

Original: In addition, the transferable reward allows repurposing the trained agent from one speciﬁc class to another class.
Revised: In addition, the transferable reward allows repurposing the trained agent from one specific class to another class.

Original: Experiments ona corrupted MNIST, the CU-Birds, andthe COCO datasets demonstrate the effectiveness of our approach.
Revised: Experiments on corrupted MNIST, CU-Birds and COCO datasets demonstrate the effectiveness of our approach

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is increasing interest in designing machine learning models that can be adapted to unseen environments or repurposed for new tasks, with only minimal human guidance (Vinyals et al., 2016;
Revised: There is increasing interest in designing machine learning models that can be adapted to unseen environments or repurposed for new tasks, with only minimal human guidance (Vinyals et al., 2016;

Original: Snell et al., 2017;
Revised: Snell et al., 2017;

Original: Finn et al., 2017; Sun et al., 2020;
Revised: Finn et al., 2017; Sun et al., 2020;

Original: Hansen et al., 2021).
Revised: Hansen et al., 2021).

Original: To this end, a small set of examples can not only serve the purpose of implicitly generalizing a trained model to a new environment during training time, but also enable the learner to update its learning objectives during test time.
Revised: To this end, a small set of examples can not only serve the purpose of implicitly generalizing a trained model to a new environment during training time, but also enable the learner to update its learning objectives during test time.

Original: In this paper, we focus on a reinforcement learning (RL) formulation to the problem of query object localization , in which an agent is trained to localize the target object speciﬁed by a small set of exemplary images.
Revised: In this paper, we focus on a reinforcement learning (RL) formulation to the problem of query object localization , in which an agent is trained to localize the target object specified by a small set of exemplary images.

Original: Instead of using ﬁxed bounding box proposals, our vision-based agent can be viewed as a proactive information gatherer (Guo, 2003), which actively interacts with an image environment; Furthermore, it follows a class-speciﬁc localization policy, and thus is more suitable for aerial imagery (Xia et al., 2018), robotic manipulation (Kalashnikov et al., 2018), or embodied AI (Savva et al., 2019) tasks.
Revised: Instead of using fixed bounding box proposals, our vision-based agent can be viewed as a proactive information gatherer (Guo, 2003), which actively interacts with an image environment; Furthermore, it follows a class-specific localization policy, and thus is more suitable for aerial imagery (Xia et al., 2018), robotic manipulation (Kalashnikov et al., 2018), or embodied AI (Savva et al., 2019) tasks.

Original: During the test time of query object localization, the queried object class to localize may be novel, or the background environment may undergo substantial changes, hindering the applicability of class-agnostic agents with a ﬁxed policy.
Revised: During the test time of query object localization, the queried object class to localize may be novel, or the background environment may undergo substantial changes, hindering the applicability of class-agnostic agents with a fixed policy.

Original: In addition, in standard RL settings for object localization, the reward signal is often available, so ﬁne-tuning methods (Julian et al., 2020) can effectively adapt agents to new environments and yield improved performance; On the contrary, in query object localization, the reward signal is not available at test time, because the bounding box annotations are to be found by the localization agent on test images.
Revised: In addition, in standard RL settings for object localization, the reward signal is often available, so fine-tuning methods (Julian et al., 2020) can effectively adapt agents to new environments and yield improved performance; On the contrary, in query object localization, the reward signal is not available at test time, because the bounding box annotations are to be found by the localization agent on test images.

Original: To address these problems, we propose an ordinal metric learning based framework for learning an implicitly transferable reward signal deﬁned with a small exemplary set.
Revised: To address these problems, we propose an ordinal metric learning based framework for learning an implicitly transferable reward signal defined with a small exemplary set.

Original: An ordinal embedding network is pre-trained with data augmentation under a loss function designed to be relevant to the RL task.
Revised: An ordinal embedding network is pre-trained with data augmentation under a loss function designed to be relevant to the RL task.

Original: The reward signal allows explicit updates of the controller in the policy network with continual training during test time.
Revised: The reward signal allows explicit updates of the controller in the policy network with continual training during test time.

Original: Compared to ﬁne-tuning approaches, the agents can get exposed to new environments more extensively with unlimited usage of test images.
Revised: Compared to fine-tuning approaches, the agent can get exposed to new environments more extensively with unlimited usage of test images.

Original: Informed by the exemplary set precisely, the agent is adapted to the changes of the localization target.
Revised: Informed by the exemplary set precisely, the agent is adapted to the changes of the localization target.

Original: Our contributions in this paper are summarized as follows: (1) We propose a novel ordinal metric learning based RL framework to learn a transferable reward signal deﬁned with a small exemplary set for query object localization; (2) Our RL framework enables test-time policy adaptation to new environments where there is no readily available reward signal; (3) Our learned transferable reward allows repurposing a trained agent from localizing one speciﬁc class to another; (4) Extensive experiments on several datasets demonstrate the effectiveness of our proposed approach.
Revised: Our contributions in this paper are summarized as follows: (1) We propose a novel ordinal metric learning based RL framework to learn a transferable reward signal defined with a small exemplary set for query object localization; (2) Our RL framework enables test-time policy adaptation to new environments where there is no readily available reward signal; (3) Our learned transferable reward allows repurposing a trained agent from localizing one specific class to another; (4) Extensive experiments on several datasets demonstrate the effectiveness of our proposed approach.

Original:  R ELATED W ORK 2. O BJECT L OCALIZATION
Revised:  R ELATED W ORK 2. O BJECT LOCALIZATION

Original: Compared to bounding-box regression approaches (Redmon et al., 2016; Ren et al., 2016), deep
Revised: Compared to bounding-box regression approaches (Redmon et al., 2016; Ren et al., 2016), deep

Original: RL based object localization approaches (Caicedo & Lazebnik, 2015; Jie et al., 2016) have the advantage of being region-proposal free, with customized search paths for each image environment.
Revised: RL based object localization approaches (Caicedo & Lazebnik, 2015; Jie et al., 2016) have the advantage of being region-proposal free, with customized search paths for each image environment.

Original: The speciﬁcity of an agent purely depends on the classes of bounding-boxes used in the reward.
Revised: The specificity of an agent purely depends on the classes of bounding-boxes used in the reward.

Original: They can be made class-speciﬁc (Caicedo & Lazebnik, 2015), but the agent for each class would need to be trained separately.
Revised: They can be made class-specific (Caicedo & Lazebnik, 2015), but the agent for each class would need to be trained separately.

Original: Despite the rise of crowdsourcing platforms, obtaining ample amount of bounding-box annotations remains costly and error-prone.
Revised: Despite the rise of crowdsourcing platforms, obtaining ample amount of bounding-box annotations remains costly and error-prone.

Original: Furthermore, the quality of annotations often varies, and precise annotations for certain object class may require special expertise from annotators.
Revised: Furthermore, the quality of annotations often varies, and precise annotations for certain object class may require special expertise from annotators.

Original: The emergence of weakly supervised object localization (WSOL) (Song et al., 2014; Oquab et al., 2015; Zhou et al., 2016) methods alleviates the situation, in which image class labels are used to derive bounding box annotations.
Revised: The emergence of weakly supervised object localization (WSOL) (Song et al., 2014; Oquab et al., 2015; Zhou et al., 2016) methods alleviates the situation, in which image class labels are used to derive bounding box annotations.

Original: It is known that WSOL methods have drawbacks of overly relying on inter-class discriminative features and failing to generalize to classes unseen during the training phase.
Revised: It is known that WSOL methods have drawbacks of overly relying on inter-class discriminative features and failing to generalize to classes unseen during the training phase.

Original: We argue that intra-class similarity is a more natural objective for the problem of localizing objects belonging to a target class. "
Revised: We argue that intra-class similarity is a more natural objective for the problem of localizing objects belonging to a target class. "

Original: Query object localization" emphasizes more about the class-speciﬁc nature of a localization task, with the relevance deﬁned by the similarity to query images.
Revised: Query object localization" emphasizes more about the class-specific nature of a localization task, with the relevance defined by the similarity to query images.

Original: A similar problem is image co-localization
Revised: A similar problem is image co-localization

Original: (Tang et al., 2014; Wei et al., 2017), in which the task is to identify the common objects within a set of images.
Revised: (Tang et al., 2014; Wei et al., 2017), in which the task is to identify the common objects within a set of images.

Original: Co-localization approaches exploit the common characteristics across images to localize objects.
Revised: Co-localization approaches exploit the common characteristics across images to localize objects.

Original: Being unsupervised, co-localization approaches could suffer from ambiguity if there exist multiple common objects or parts, e.g., bird head and body (Jaderberg et al., 2015), which may provide unwanted common objects as output.
Revised: Being unsupervised, co-localization approaches could suffer from ambiguity if there exist multiple common objects or parts, e.g., bird head and body (Jaderberg et al., 2015), which may provide unwanted common objects as output.

Original: 2. P OLICY A DAPTATION
Revised: 2. P OLICY ADAPTATION

Original: There seems to be a contradiction between the goals of training an agent with high task-speciﬁcity and better generalization performance to new situations at the same time.
Revised: There seems to be a contradiction between the goals of training an agent with high task-specificity and better generalization performance to new situations at the same time.

Original: The key to reconcile these two goals lies in the usage of a small set of examples.
Revised: The key to reconcile these two goals lies in the usage of a small set of examples.

Original: There has been a paradigm shift from training static models deﬁned with parameters to  ones deﬁned together with a support set (Vinyals et al., 2016; Snell et al., 2017), proved to be very effective in few-shot classiﬁcation.
Revised: There has been a paradigm shift from training static models defined with parameters to dynamic ones defined together with a support set (Vinyals et al., 2016; Snell et al., 2017), proved to be very effective in few-shot classification.

Original: Besides the efforts of employing meta learning to adjust models, ﬁne-tuning on a pretrained model (Oquab et al., 2014; Finn et al., 2017; Julian et al., 2020) has also been widely used in transferring knowledge from data-abundant to data-scarce tasks.
Revised: Besides the efforts of employing meta learning to adjust models, fine-tuning on a pre-trained model (Oquab et al., 2014; Finn et al., 2017; Julian et al., 2020) has also been widely used in transferring knowledge from data-abundant to data-scarce tasks.

Original: When reward signal is not available, Hansen et al. (2021) proposed a policy adaptation approach, in which the intermediate representation is ﬁne-tuned via optimizing a self-supervised auxiliary loss while the controller is kept ﬁxed.
Revised: When reward signal is not available, (Hansen et al., 2021) proposed a policy adaptation approach, in which the intermediate representation is fine-tuned via optimizing a self-supervised auxiliary loss while the controller is kept fixed.

Original: Our work shares the same motivation of test-time training (Sun et al., 2020), but we focus instead on the settings where the controller needs to be adapted or even repurposed for new tasks.
Revised: Our work shares the same motivation of test-time training (Sun et al., 2020), but we focus instead on the settings where the controller needs to be adapted or even repurposed for new tasks.

Original:  L EARNING T RANSFERABLE R EWARD FOR L OCALIZING Q UERY O BJECTS 3. RL FORMULATION FOR QUERY OBJECT LOCALIZATION
Revised:  L EARNING T RANSFERABLE R EWARD FOR L OCALIZING Q UERY O BJECTS 3. RL FORMULATION FOR QUERY OBJECT LOCALIZATION

Original: Given a set of images I with known class labels, the task of query object localization is to ﬁnd the location b of the bounding box most relevant to the query object, speciﬁed in a small set of exemplary images E .
Revised: Given a set of images I with known class labels, the task of query object localization is to find the location b of the bounding box most relevant to the query object, specified in a small set of exemplary images E .

Original: Following the Markov Decision Process (MDP) framework for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016), this task can be formulated as a RL problem.
Revised: Following the Markov Decision Process (MDP) framework for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016), this task can be formulated as a RL problem.

Original: In the training phase, a localization agent learns to take actions of moving the bounding box to maximize a reward function reﬂecting localization accuracy.
Revised: In the training phase, a localization agent learns to take actions of moving the bounding box to maximize a reward function reflecting localization accuracy.

Original: In the test phase, the agent is informed by a small test exemplary set about a new task objective.
Revised: In the test phase, the agent is informed by a small test exemplary set about a new task objective.

Original: Figure 1 shows an overview of the query object localization problem and the RL formulation, with components listed as follows.
Revised: Figure 1 shows an overview of the query object localization problem and the RL formulation, with components listed as follows.

Original: • Environment: Raw pixels of a single image
Revised: • Environment: Raw pixels of a single image

Original: I i , without candidate proposal boxes
Revised: I i , without candidate proposal boxes.

Original: Action Discrete actions facilitating a top-down search scaling scale towards top left
Revised: • Action: Discrete actions facilitating a top-down search, including five scaling: scale towards top left/right, bottom left/right, center; eight translation transformations: move left/right/up/- down, enlarge width/height, reduce width/height; plus one stay action.

Original: right left center eight translation transformations: move left
Revised: • State:

Original: right/up - down, enlarge width/height, reduce width/heigh plus one stay action • State: Pooled features from the current box and an internal state within RNN that encodes information from history observations.
Revised: Pooled features from the current box and an internal state within RNN that encodes information from history observations.

Original: • Reward:
Revised: • Reward:

Original: Improvements the agent made in localizing the query images (in different tasks).
Revised: Improvements the agent made in localizing the query images (in different tasks).

Original: 3. P RETRAINING O RDINAL R EWARD S IGNAL
Revised: 3. P RE - TRAINING ORDINAL REWARD SIGNAL

Original: For training an agent to localize different objects according to queries, the reward function needs to be transferable.
Revised: For training an agent to localize different objects according to queries, the reward function needs to be transferable.

Original: Existing deep RL approaches for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016) use its ground-truth object bounding box g i as the reward signal,
Revised: Existing deep RL approaches for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016) use its ground-truth object bounding box g i as the reward signal,

Original: [Equation]
Revised: [Equation]

Original: where IoU ( b t , g i ) denotes the Intersection-over-Union (IoU) between the current window b t and the corresponding ground-truth box g i 1 , and IoU ( b, g ) =
Revised: where IoU ( b t , g i ) denotes the Intersection-over-Union (IoU) between the current window b t and the corresponding ground-truth box g i 2 , and IoU ( b, g ) =

Original: area ( b ∩ g ) / area ( b ∪ g ) .
Revised: area ( b ∩ g ) / area ( b ∪ g ) .

Original: Similar to the bounding box regression approaches which learns a mapping φ : I (cid:55)→ g , the image and box must be paired.
Revised: Similar to the bounding box regression approaches which learn a mapping ϕ I (cid:55)→ g , the image and box must be paired.

Original: However, annotated image-box pairs ( I, g ) may be scarce in both the training and testing phases.
Revised: However, annotated image-box pairs ( I, g ) may be scarce in both the training and testing phases.

Original: The reward signal deﬁned in Eq.
Revised: The reward signal defined in Eq.

Original: 1 is instance-wise, which is not transferable across images.
Revised: 1 is instance-wise, which is not transferable across images.

Original: To address this problem, a natural idea is to deﬁne the reward signal based on learned representations of the cropped images by current window b t and the ground-truth window g .
Revised: To address this problem, a natural idea is to define the reward signal based on learned representations of the cropped images by current window b t and the ground-truth window g .

Original: Given their Mdimensional representations b t and g produced by an embedding function f : R D (cid:55)→ R M from
Revised: Given their Mdimensional representations b t and g produced by an embedding function f : R D (cid:55)→ R M from

Original: D - dimensional image feature vectors, a distance function d : R M × R M (cid:55)→ [0 , + ∞ ) returns the embedding distance d ( b t , g ) .
Revised: D - dimensional image feature vectors, a distance function d : R M × R M (cid:55)→ [0 , + ∞ ) returns the embedding distance d ( b t , g ) .

Original: However, an embedding distance based on off-the-shelf pretrained networks is not adequate, since it may not decrease monotonically as the agent approaches to the ground-truth box g .
Revised: However, an embedding distance based on off-the-shelf pre-trained networks is not adequate, since it may not decrease monotonically as the agent approaches to the ground-truth box g .

Original: As a result, the embedding distance based reward signal may be less effective than Eq.
Revised: As a result, the embedding distance based reward signal may be less effective than Eq.

Revised: 1 (as shown in Section 4).

Original: Therefore, an embedding function customized for the downstream RL-based localization task is needed.
Revised: Therefore, an embedding function customized for the downstream RL-based localization task is needed.

Original: Ordinal metric learning.
Revised: Ordinal metric learning.

Original: We propose to use an ordinal embedding based reward signal.
Revised: We propose to use an ordinal embedding based reward signal.

Original: For any two perturbed boxes b j , b k from ground truth g , embeddings b j , b k , g are learned, such that the relative preferences between any pair of boxes are preserved in the Euclidean space,
Revised: For any two perturbed boxes b j , b k from ground truth g , embeddings b j , b k , g are learned, such that the relative preferences between any pair of boxes are preserved in the Euclidean space,

Original: [Equation]
Revised: [Equation]

Original: where ρ j and ρ k denote the preference (derived from either IoU to ground-truth box or ordinal feedbacks from user), and C is an ordinal constraint set constructed by sampling box pairs around g .
Revised: where ρ j and ρ k denote the preference (derived from either IoU to ground-truth box or ordinal feedback from user), and C is an ordinal constraint set constructed by sampling box pairs around g .

Original: This problem is originally posed as non-metric multidimensional scaling (Agarwal et al., 2007;
Revised: This problem is originally posed as non-metric multidimensional scaling (Agarwal et al., 2007;

Original: Jamieson & Nowak, 2011).
Revised: Jamieson & Nowak, 2011).

Original: Although we apply a very simple pairwise-based approach, there exist other extensions such as the listwise-based (Cao et al., 2007), quadruplet-based (Terada & Luxburg, 2014) and landmark-based (Anderton & Aslam, 2019; Ghosh et al., 2019) approaches.
Revised: Although we apply a very simple pairwise-based approach, there exist other extensions such as the listwise-based (Cao et al., 2007), quadruplet-based (Terada & Luxburg, 2014) and landmark-based (Anderton & Aslam, 2019; Ghosh et al., 2019) approaches.

Original: Loss function.
Revised: Loss function.

Original: In this paper, we deﬁne preference ρ as the IoU of box b to the ground-truth box g , i.e., ρ = IoU ( b, g ) .
Revised: In this paper, we define preference ρ as the IoU of box b to the ground-truth box g , i.e., ρ = IoU ( b, g ) .

Original: We choose to optimize a triplet loss (Hermans et al., 2017) for learning the desired embeddings,
Revised: We choose to optimize a triplet loss (Hermans et al., 2017) for learning the desired embeddings,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where a is the “anchor" embedding that will be discussed in detail later, p , n are the “positive" and “negative" embeddings for boxes with larger and smaller IoUs with ground truth box g , respectively, m is a margin hyperparameter.
Revised: where a is the “anchor" embedding that will be discussed in detail later, p , n are the “positive" and “negative" embeddings for boxes with larger and smaller IoUs with ground truth box g , respectively, m is a margin hyper-parameter.

Original: We learn an embedding space consistent with local ordinal constraints on positive and negative data obtained via data augmentation such as box perturbation.
Revised: We learn an embedding space consistent with local ordinal constraints on positive and negative data obtained via data augmentation such as box perturbation.

Original: Box Perturbation Assuming the training exemplary set E train contains both image I and box g , we adopt a tailored data augmentation scheme - box perturbation , in which a pair of boxes are randomly sampled from I with different IoUs to g .
Revised: Box Perturbation Assuming the training exemplary set E train contains both image I and box g , we adopt a tailored data augmentation scheme - box perturbation , in which a pair of boxes are randomly sampled from I with different IoUs to g .

Original: We compare the efﬁciency of sampling schemes on generating augmented bounding box pairs, (a) Random sampling , where the pair of boxes are generated completely randomly, and (b) Group-based sampling , where dense boxes with variant scales are ﬁrst generated, and then divided into M = 10 groups according to the IoU with groundtruth box.
Revised: We compare the efficiency of sampling schemes on generating augmented bounding box pairs, (a) Random sampling , where the pair of boxes are generated completely randomly, and (b) Group-based sampling , where dense boxes with variant scales are first generated, and then divided into M = 10 groups according to the IoU with groundtruth box.

Original: Each group has an IoU interval of 1 /M .
Revised: Each group has an IoU interval of 1 /M .

Original: The sampling is ﬁrst done on the group level, i.e., 2 out of M , then sample one box from each group.
Revised: The sampling is first done on the group level, i.e., 2 out of M , then sample one box from each group.

Original: Thus, the sampled boxes are likely to cover more diverse cases comparing to random sampling.
Revised: Thus, the sampled boxes are likely to cover more diverse cases comparing to random sampling.

Original: We have found that using IoU-based partition scheme is more effective than random sampling.
Revised: We have found that using IoU-based partition scheme is more effective than random sampling.

Original: The choice of anchor.
Revised: The choice of anchor.

Original: The anchor embedding a in Eq.
Revised: The anchor embedding a in Eq.

Original: 3 is not restricted to the cropped image from the same image.
Revised: 3 is not restricted to the cropped image from the same image.

Original: For example, it could be replaced by the prototype embedding (Snell et al., 2017) of the exemplary set E , c
Revised: For example, it could be replaced by the prototype embedding (Snell et al., 2017) of the exemplary set E , c

Original: = 1 / |E| (cid:80) i ∈E g i , where g i is the embedding of the cropped image
Revised: = 1 / |E| (cid:80) i ∈E g i , where g i is the embedding of the cropped image

Original: I i by ground-truth box g i .
Revised: I i by ground-truth box g i .

Original: If images from multiple classes are available, the prototype can be further made to be class-dependent, or clustering-based.
Revised: If images from multiple classes are of interest, the prototype can be further made to be class-dependent, or clustering-based.

Original: A prototype as the anchor enables test-time policy adaptation without image-box pairs in the exemplary set E test .
Revised: A prototype as the anchor enables test-time policy adaptation without image-box pairs in the exemplary set E test .

Original: In comparison, supervised methods such as faster-RCNN would require image-box pairs for ﬁne-tuning on new tasks.
Revised: In comparison, supervised methods such as Faster RCNN would require image-box pairs for fine-tuning on new tasks.

Original: We ﬁnd that using prototype-based embedding as the anchor leads to better generalization performance than using g in some experiments.
Revised: In some experiments (Appendix B.1), we find that using prototype-based embedding as the anchor could lead to better generalization performance than using g .

Original: It is also interesting to explore other treatment on the exemplary set, such as permutation-invariant architectures (Zaheer et al., 2017; Ilse et al., 2018; Lee et al., 2019).
Revised: It is also interesting to explore other treatment on the exemplary set, such as permutation-invariant architectures (Zaheer et al., 2017; Ilse et al., 2018; Lee et al., 2019).

Original: 3. L OCALIZATION A GENT T RAINING
Revised: 3. L OCALIZATION AGENT TRAINING

Original: As discussed in Section 3.2, we use the ordinal embedding rather than the bounding box coordinates to quantify the improvement that an agent makes, and the continuous-valued reward for the agent moving from state s (cid:48) to s takes the following form,[Equation]
Revised: As discussed in Section 3.2, we use the ordinal embedding rather than the bounding box coordinates to quantify the improvement that an agent makes, and the continuous-valued reward for the agent moving from state s ′ to s takes the following form,

Original: [Equation]
Revised: [Equation]

Original: ) where c is the protoptype embedding, b t is the embedding of the cropped region at step t .
Revised: where c is the prototype embedding, b t is the embedding of the cropped region at step t .

Original: We expect positive reward if the embedding distance between cropped image b t and query image a is decreased.
Revised: We expect positive reward if the embedding distance between cropped image b t and query image a is decreased.

Original: State representation.
Revised: State representation.

Original: Note that a good representation for deﬁning reward may not necessarily be a good state representation at the same time - it may not contain enough information guiding the agent taking the right actions.
Revised: Note that a good representation for defining reward may not necessarily be a good state representation at the same time - it may not contain enough information guiding the agent taking the right actions.

Original: Chen et al.
Revised: Chen et al.

Original: (2020) suggests that adding a projection head between the representation and the contrastive loss substantially improves the quality of the learned representation.
Revised: (2020) suggests that adding a projection head between the representation and the contrastive loss substantially improves the quality of the learned representation.

Original: We ﬁnd that the use of projection head is crucial in balancing the two objectives in our task.
Revised: We find that the use of projection head is crucial in balancing the two objectives in our task.

Original: The network architecture is shown in Figure 2(a), in which an MLP projection head is attached after an (Region of Interest) RoI encoder.
Revised: The network architecture is shown in Figure 2(a), in which an MLP projection head is attached after an (Region of Interest) RoI encoder.

Original: The ROIAlign module handles boxes of different sizes.
Revised: The ROIAlign module handles boxes of different sizes.

Original: Policy network.
Revised: Policy network.

Original: Starting from a whole image as input, the agent is trained to select actions to transform the current box at each step, maximizing the total discounted reward.
Revised: Starting from a whole image as input, the agent is trained to select actions to transform the current box at each step, maximizing the total discounted reward.

Original: We use policy gradient based on a recurrent neural network (RNN)
Revised: We use policy gradient based on a recurrent neural network (RNN)

Original: (Mnih et al., 2014) with a vector of history actions and states.
Revised: (Mnih et al., 2014) with a vector of history actions and states.

Original: More detailed architecture can be found in Figure 2(b) and Appendix Figure 7.
Revised: More detailed architecture can be found in Figure 2(b) and Appendix Figure 6.

Original: For all experiments, we apply policy gradient (REINFORCE)
Revised: For all experiments, we apply policy gradient (REINFORCE)

Original: (Williams, 1992) with entropy loss regularization to encourage exploration.
Revised: (Williams, 1992) with entropy loss regularization to encourage exploration.

Original: The main algorithm is outlined in Algorithm 1.
Revised: The main algorithm is outlined in Algorithm 1.

Original: Require: initial policy π θ , batch size N , pretrained RoI encoder e and projection head h , learning rate α Output : policy network π θ for sampled minibatch { x k } Nk =1 do for k=1,...
Revised: Require: initial policy π θ , [Table] size N , [Table]

Original: N do
Revised: and [Table] , learning rate [Table]

Revised: Output

Revised: [Table]

Revised: π θ for

Revised: [Table]

Original: Construct an exemplary set E batch g j ← h ( e ( x k , g j )) // compute the ordinal embedding in Figure 2(a)
Revised: [Table] [Table] = [Table]

Original: c
Revised: do for [Table] = [Table]

Original: k ← 1 |E batch | (cid:80)
Revised: .N

Original: j ∈E batch g j// compute the prototype vector τ k
Revised: do Construct g j

Original: [Table]
Revised: [Equation]

Original: [Table] [Table] [Table]
Revised: , g j )) // compute the ordinal embedding in Figure 2(a)

Original: by
Revised: c

Original: [Table]
Revised: [Equation]

Original: // sample trajectory b
Revised: / compute the prototype vector τ k ← { s 1 , a 1 , ...

Original: [Table]
Revised: s T , a T } by running policy π θ // sample trajectory b t ← take action a t , and compute embedding h ( e ( x k , b t )) for current window b t

Original: [Table] R [Table]
Revised: R tk ← ∥ b t − 1 − c k ∥

Original:  // compute reward in equationend 
Revised: − ∥ b t − c k ∥ // compute reward in equationend θ ← θ + α ∇ θ (cid:80) Nk =1 (cid:104) log π θ (cid:16)(cid:80)

Original: [Table]
Revised: Tt ′ =

Revised: t γ t ′ − t R tk (cid:17) − λ · π θ log π θ (cid:105) //

Original: policy update end
Revised: policy update end

Original: 3. T EST - TIME A DAPTATION
Revised: 3. T EST - TIME ADAPTATION

Original: During test time„ the agent has the option of further updating the policy network using the received reward from Eq. 4.
Revised: During test time, the agent has the option of further updating the policy network using the received reward from Eq. 4.

Original: To match test conditions, the training batch is split into two groups, and c is computed on a small subset that does not overlap with the training images to localize;During test adaptation, c becomes the prototype of the test exemplary set E test .
Revised: To match test conditions, the training batch is split into two groups, and c is computed on a small subset that does not overlap with the training images to localize; During test-time adaptation, c becomes the prototype of the test exemplary set E test .

Original: Different from E train , only cropped images are needed in E test as shown in Figure 1.
Revised: Different from E train , only cropped images are needed in E test as shown in Figure 1.

Original: Figure 3 shows an illustrative example of query object localization with task adaptation.

Original: Given two common objects coexisting in the image environment, the agent is trained to pick up one from the two during training, while quickly adapt to pick another one during testing.

Original: Different from unsupervised co-localization approaches (Wei et al., 2017) relying on low-level images clues to distinguish foreground and background, our learning objectives can be speciﬁed explicitly by switching the train/test exemplary sets.

Original: With test-time adaptation, our agent can handle even partially conﬂicting learning objectives.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we evaluate the generalization ability of the ordinal embedding as well as the performance of the trained localization agent on both source and target domains.
Revised: In this section, we evaluate the generalization ability of the ordinal embedding as well as the performance of the trained localization agent on both source and target domains.

Original: The embedding-based reward not only improves the RL training, but also enables test-time adaptation of the learned policy.
Revised: The embedding-based reward not only improves the RL training, but also enables test-time adaptation of the learned policy.

Original: Experimental results demonstrate the effectiveness of our approach, with empirical comparisons to generic RL-based object localization, co-localization and few-shot  object detection baselines.
Revised: Experimental results demonstrate the effectiveness of our approach, with empirical comparisons to fine-tuning, co-localization, few-shot and Faster RCNN object detection baselines.

Original: Evaluation protocol.
Revised: Implementation details.

Original: To evaluate the learned ordinal embedding, we use OrdAcc deﬁned as the percentage of images within which the preference between a pair of perturbed boxes is correctly predicted.
Revised: To evaluate the learned ordinal embedding, we use OrdAcc defined as the percentage of images within which the preference between a pair of perturbed boxes is correctly predicted.

Original: To evaluate object localization performance, we use the Correct Localization ( CorLoc ) (Deselaers et al., 2012) metric, which is deﬁned as the percentage of images correctly localized according to the criterion IoU ( b p , g ) ≥ 0 . 5 , where b p is the predicted box and g is the ground-truth box.
Revised: To evaluate object localization performance, we use the Correct Localization ( CorLoc ) (Deselaers et al., 2012) metric, which is defined as the percentage of images correctly localized according to the criterion IoU ( b p , g ) ≥ 0 . 5 , where b p is the predicted box and g is the ground-truth box.

Original: Mean and standard deviation ( displayed as subscript) are reported from 10 independent runs.
Revised: Mean and standard deviation (displayed as subscript) are reported from 10 independent runs.

Original: Implementation.

Original: We evaluate our approach on distorted versions of the MNIST handwriting, the CUB-200-2011 birds (Wah et al., 2011), and the COCO (Lin et al., 2014) dataset.
Revised: We evaluate our approach on distorted versions of the MNIST handwriting, the CUB-200-2011 birds (Wah et al., 2011), and the COCO (Lin et al., 2014) dataset.

Original: For MNIST, we use three convolutional layers with ReLU activation after each layer asimage encoder, while the same but mirrored structure as decoder to learn an autoencoder, and then attach RoI alignment layer followed by two fully connected ( fc ) layers as projection head for ordinal reward learning.
Revised: For the MNIST dataset, we use three convolutional layers with ReLU activation after each layer as the image encoder.

Original: For the CUB and the COCO datasets, we adopt layers before conv5_3 of VGG-16 pretrained on ImageNet as encoder.
Revised: For the CUB and the COCO datasets, we adopt layers before conv5_3 of VGG-16 pre-trained on ImageNet as the encoder, unless otherwise specified.

Original: The projection head uses the same structure as before but with more units for each fc layer.

Original: All of our models were trained with the Adam optimizer (Kingma & Ba, 2015).

Original: We set margin m = 60 in all the experiments heuristically.

Original: All the models take less than one hour to ﬁnish training, implemented on PyTorch on a single NVIDIA A100 GPU.

Original: 4. A BLATION S TUDIES

Original: We ﬁrst study the effectiveness of ordinal embedding and learned reward as well as a few other design choices.

Original: Setting and main results.

Original: We analyze the effectiveness of using ordinal embedding in terms of representation and reward ﬁrst on synthetic datasets of corrupted MNIST digits under cluttered, random patch, Gaussian, or impulse noises (see Figure 8 in Appendix for examples).

Original: We want to ﬁnd the answers to two main questions listed below:

Original: Is the embedding distance based reward signal as effective as the IoU counterpart?

Original: Accordingly, we consider baselines in which embeddings are trained with only autoencoder or jointly with the ordinal projection head.

Original: Besides, we also compare our embedding distance based reward against the IoU based reward used in Caicedo & Lazebnik (2015).

Original: Without loss of generality, the agent is trained on images of digit 4, and tested on images from all rest classes.

Original: In particular, we are interested in evaluating the sample efﬁciency of different approaches.

Original: The results under different training set sizes are shown in Figure 4.

Original: To distract the localization agent, cluttered noises with random 6 × 6 crops of digits are added to the background.

Original: All comparisons are based on the policy gradient (PG) method.

Original: With ordinal embedding present in both the representation and reward (“AE+Ord+Embed"), our model performance is consistently better than other settings, especially when the training set size is small.

Original: The beneﬁt of a triplet loss is demonstrated by the comparison between “AE+Ord+IoU" and “AE+IoU".

Original: Intuitively, the similarity to a queried object is also a more natural goal than objectives deﬁned by bounding boxes IoU, which is used in both generic object detection or localization and previous RL-based approaches.

Original: Signed reward or not.

Original: Another intriguing question is that other than beneﬁting state representation learning, whether the embedding-based reward itself can bring any additional beneﬁts than the IoU- based reward.

Original: From Figure 4, there is a large gap between “AE+Ord+Embed" and “AE+Ord+IoU", especially when the training set size is small.

Original: This is counter-intuitive as the ordinal reward is trained with supervision from IoU in embedding space, supposably it shall not bring any additional information.

Original: To further analyze this phenomenon, we take the sign operation off in Eq.

Original: and retrain the agent.

Original: In an experiment with 50 training images, without sign operation, the localization accuracy increases from 90.92% to 94.36% on test digit 4 , and from 82.43% to 88.64% on other digit test set.

Original: Perhaps, this improvement can be attributed to more informative feedback provided by the continuous-valued, unsigned reward.

Original: On policy vs. off policy.

Original: Many deep RL approaches are in favor of using deep Q-Network (DQN) to train an agent.

Original: Different from Caicedo & Lazebnik (2015) and Jie et al.

Original: , we apply Policy

Original: Gradient (PG) to optimize it.

Original: Besides, we adopt a top-down search strategy through a RNN, while they used a vector of history actions to encode memory.

Original: We evaluate these design choices with four baselines, with "AE+IoU" setting, and trained on the same 600 sampled cluttered digit 4 images.

Original: As

Original: Table 1 shows, the agent achieves the best performance with "PG+RNN". We ﬁnd that empirically, with history action vectors the accuracy becomes worse when the agent is trained by DQN. with/without history action vector) vs. PG (with/without RNN) [Table]
Revised: More details are presented in Appendix A.2.

Original:  P OLICY A DAPTATION DURING T EST -T IME
Revised: 4. P OLICY ADAPTATION DURING TEST TIME

Original: Wefurther demonstrate the performance improvement with test-time policy adaptation.
Revised: We demonstrate the performance improvement with test-time policy adaptation.

Original: Through all the experiments, we assume source domain contains abundant data annotations, and target domain annotations are only available in an exemplary set of size 5 .
Revised: Through all the experiments, we assume source domain contains abundant data annotations, and target domain annotations are only available in an exemplary set of size 5 .

Original: We compared our policy adaptation scheme with a standard ﬁne-tuning scheme on the pre-trained policy network.
Revised: We compare our policy adaptation scheme with a standard fine-tuning scheme on the pre-trained policy network.

Original: Results on the corrupted MNIST dataset.
Revised: Results on the corrupted MNIST dataset.

Original: We use 50 “digit 4  under random patch background noises" to train the ordinal embedding and the localization agent.During test time, either the class of the foreground object or the background changes.
Revised: For the new class adaptation experiment, we use“digit 4 images under random patch background noises" to train the ordinal embedding and the localization agent.

Original: The results on policy adaptation to new digits  are shown in Table 2.
Revised: The results on policy adaptation to new digits (other than 4 ) are shown in Table 1.

Original: Row 1 illustrates the transferability of the ordinal embedding reward, trained prototype embedding of a subgroup without the training instance, and evaluated using instance embedding from the same test image (“Ord Ins").
Revised: Row 1 illustrates the transferability of the ordinal embedding reward, trained prototype embedding of a subgroup without the training instance, and evaluated using instance embedding from the same test image (“ OrdAcc ").

Original: Rows 2 to 4 list the resulting localization accuracy after direct generalization (“Before"), ﬁne-tuning on the exemplary set (“Finetune"), and adaptation using all test images (“Adapt"), respectively.
Revised: Rows 2 to 4 list the resulting localization accuracy after direct generalization (“before"), fine-tuning on the exemplary set (“fine-tune"), and adaptation using all test images (“adapt"), respectively.

Original: Our policy adaptation approach produces a substantial improvement over direct generalization, while ﬁne-tuning approach experiences overﬁtting on the limited exemplary set.
Revised: Our policy adaptation approach produces a substantial improvement over direct generalization, while fine-tuning approach experiences overfitting on the limited exemplary set.

Original: For the background adaptation experiment, we train on 50 digit-3 images  and test on digit-2 images.
Revised: For the background adaptation experiment, we train on 50 digit-3 images under random patch noise, and test on digit-2 images under all four noises.

Original: The localization accuracy on both sourcebackground and new backgrounds environment are shown in Table 3, signiﬁcant improvements are achieved using our policy adaptation scheme. 
Revised: The localization accuracy on both source and new backgrounds environment are shown in Table 2, significant improvements are achieved using our policy adaptation scheme. and CorLoc (%) on new digits environment

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results on the CUB dataset.
Revised: Results on the CUB dataset.

Original: We also evaluate the policy adaptation performance on the CUB dataset.
Revised: We also evaluate the policy adaptation performance on the CUB dataset.

Original: The localization agent is trained on 15 species from the “Warbler" class, and adapted to different classes of “Warbler" ( 5 new species), “Wren", “Sparrow", “Oriole", “Kingﬁsher", “Vireo", and “Gull".
Revised: The localization agent is trained on 15 species from the “Warbler" class, and adapted to different classes of “Warbler" ( 5 new species), “Wren", “Sparrow", “Oriole", “Kingfisher", “Vireo" [Table] and “Gull".

Original: Each test class contains a single bird class.
Revised: Each test class contains a single bird class.

Original: We also implement deep descriptor transforming (DDT) (Wei et al., 2017), a deep learning based co-localization approach, and add it to the comparison.
Revised: We also implement deep descriptor transforming (DDT) (Wei et al., 2017), a deep learning based co-localization approach, and add it to the comparison.

Original: [Table]
Revised: [Table]

Original: C OMPARISON TO F EW - SHOT O BJECT D ETECTORS
Revised: Results on the COCO dataset.

Original: Given a target domain that one would like to deploy models to, a natural question is whether one should collect labeled data from an abundant number of source domains or from one speciﬁc class with potential lower data collection cost.
Revised: Given a target domain that one would like to deploy models to, a natural question is whether one should collect labeled data from an abundant number of source domains or from one specific class with potential lower data collection cost.

Original: On the COCO dataset, we train on one of the ﬁve classes: cat, cow, dog, horse, zebra , then adapt to another four classes.
Revised: On the COCO dataset, we train on one of the five classes: cat, cow, dog, horse, zebra , then adapt to another four classes.

Original: The results are shown in Figure 5.
Revised: The results are shown in Figure 3.

Original: The models perform best using ordinal embedding reward with  adaptation (see “ours-adapt").
Revised: The models perform best using ordinal embedding reward with [Figure] adaptation (see “ours-adapt").

Original: It shows that, although an agent is trained in a highly specialized way on only one-source tasks, it can still be ﬂexibly generalized to different target tasks.
Revised: It shows that, although an agent is trained in a highly specialized way on only one-source tasks, it can still be flexibly generalized to different target tasks.

Original: One interesting observation is that, it is easy to transfer from other four classes to zebra, but not vice versa.
Revised: One interesting observation is that, it is easy to transfer from other four classes to zebra, but not vice versa.

Original: A possible explanation might be that, the embedding net is biased on textures, and texture information is less adaptable than shape information.
Revised: A possible explanation might be that, the embedding net is biased on textures, and texture information is less adaptable than shape information.

Revised: 4. C OMPARISON TO FEW - SHOT OBJECT DETECTORS

Original: Few-shot object detection (Kang et al., 2019; Yan et al., 2019; Wang et al., 2019; 2020)
Revised: Few-shot object detection (Kang et al., 2019; Yan et al., 2019; Wang et al., 2019; 2020)

Original: methods are similar to our work in adapting models to a new test case with the help of limited annotations, but they have a different requirement on the number of training classes.
Revised: methods are similar to our work in adapting models to a new test case with the help of limited annotations, but they have a different requirement on the number of training classes.

Original: Being different in the generalization mechanism, they usually require multiple classes in both stages of training and ﬁne-tuning.
Revised: Being different in the generalization mechanism, they usually require multiple classes in both stages of training and fine-tuning.

Original: On the
Revised: On the

Original: COCO dataset, we compared the performance of our methods with TFA (Wang et al., 2020), and
Revised: COCO dataset, we compared the performance of our methods with TFA (Wang et al., 2020), and

Original: Meta-RCNN (Yan et al., 2019), on the same one-way 5-shot test setting.
Revised: Meta-RCNN (Yan et al., 2019), on the same one-way 5-shot test setting.

Original: TFA adopts two-stage ﬁnetuning of object detector, while Meta-RCNN incorporates additional meta-learner to acquire class-level meta knowledge for generalization to novel classes.
Revised: TFA adopts two-stage finetuning of object detector, while Meta-RCNN incorporates additional meta-learner to acquire class-level meta knowledge for generalization to novel classes.

Original: These two few-shot baselines are trained on 60 base classes and ﬁne-tuned on all 80 classes, while our model is trained from one single class of the 5 classes randomly selected from the base classes set: elephant, sheep, giraffe, laptop and suitcase .
Revised: These two few-shot baselines are trained on 60 base classes and fine-tuned on all 80 classes, while our model is trained from one single class of the 5 classes randomly selected from the base classes set: elephant, sheep, giraffe, laptop and suitcase .

Original: Table 5 shows that with ordinal embedding, our agent achieves better performance even without adaptation, and the performance can be further improved after adaptation.
Revised: Table 4 shows that with ordinal embedding, our agent achieves better performance even without adaptation, and the performance can be further improved after adaptation [Table] [Table]

Revised: C OMPARE WITH SUPERVISED BASELINE - F ASTER RCNN

Revised: We compare our framework with a strong supervised object localization baseline

Original: [Figure]
Revised: [Table]

Revised: Faster RCNN (Ren et al., 2016).

Revised: Both methods are trained in one class (foreground vs. background as the classification labels in Faster RCNN) and adapted to a different class.

Revised: We fine-tune the pre-trained VGG-16 model and test on each of the five classes: cow, cat, dog, horse, zebra .

Revised: The results on source domain are

Original: [Table]
Revised: [Table]

Original: C ONCLUSION AND F UTURE W ORK
Revised: shown in Table 5. It shows that Faster RCNN can also be made into a class-specific model and it still yields superior performance on source domain. On the target domain, we fine-tune Faster RCNN using only query set for each target class. The results are shown in Table 6.

Revised: It can be seen that our method works better on new classes with test-time adaptation over the traditional fine-tuning of

Revised: Faster RCNN.

Revised: Note that fine-tuning requires image-box pairs in the target domain while our policy adaptation approach does not.

Revised: [Table]

Revised: [Table]

Revised: 4. A BLATION STUDIES

Revised: We analyze the effectiveness of ordinal embedding and RL component separately on COCO dataset (Appendix B.1 provides more in-depth analysis on the corrupted MNIST dataset, including policy gradient vs. deep Q-Network, continuous vs. binary reward).

Revised: First, we remove RL

Revised: [Table]

Revised: and substitute it with a simple linear search approach.

Revised: Specifically, we adopt selective search (Uijlings et al., 2013) for candidate boxes generation.

Revised: The candidate boxes are ranked according to their embedding distances to the prototype

Revised: [Table]

Revised: and the one with the smallest embedding distance is returned as the final output.

Revised: We consider two backbone networks as the embedding, including ImageNet pre-trained VGG-16 and

Revised: Faster RCNN VGG-16 trained locally on COCO dataset.

Revised: We also compare both with and without the ordinal component, making it a 2 × 2 ablation study.

Revised: It can be seen from the blue and green bars from Figure 4 and Appendix Figure 10 that with ordinal structure, the ranking method performs much better.

Revised: We find that pre-training with the proposed ordinal loss significantly improves the rank consistency of these backbone networks (Appendix B.1).

Revised: In contrast, being able to learn across different images, the RL localization agent shows more tolerance to the defects of the reward function.

Revised: We analyze the benefits of RL component.

Revised: From both Figureand Figure 10, when using ordinal embedding, the model with RL is generally better than ranking, especially after adaptation.

Revised: Ordinal component is necessary for several possible reasons.

Revised: First, the ordinal component makes the generic pre-trained network more specific to the downstream object localization task.

Revised: Second, it helps remove the dataset bias if the pre-trained network is non-native.

Revised: Third, ordinal pre-training encourages the reward signal to reflect step-by-step improvements

Revised: Encouraged by the strong baseline performance of Faster RCNN on both source and target domain (detailed in Table 5 and Table 6), we investigate whether it is adequate to use Faster RCNN embedding in the RL reward function, with or without ordinal pre-training.

Revised: Since Faster RCNN emebedding is also trained for object localization on the COCO dataset, it can separate out the effects of the ordinal pre-training component better.

Revised: The result are shown in Table 7.

Revised: It can be seen that without ordinal pre-training, the performance degrades significantly and the CorLoc is much lower even in source domain.

Revised: More results of different backbones are presented in Appendix Table 15 and Table 16, from which the test-time adaptation still brings a large margin of improvement.

Revised: [Table]

Revised:  C ONCLUSION AND F UTURE W ORK

Original: We propose an ordinal representation learning based reward, for training a localization agent to search queried objects of interest.
Revised: We propose an ordinal representation learning based reward, for training a localization agent to search queried objects of interest.

Original: In particular, we use a small exemplary set as a guidance signal for delivering learning objectives without ambiguity.
Revised: In particular, we use a small exemplary set as a guidance signal for delivering learning objectives without ambiguity.

Original: Meanwhile, we use test image environments to inform the agent about the domain shifts, without requiring image-box pairs during test time.
Revised: Meanwhile, we use test image environments to inform the agent about the domain shifts, without requiring image-box pairs during test time.

Original: Instead of jointly training a localization and classiﬁcation model, we learn box annotations from image class labels, in a similar spirit to weakly-supervised learning.
Revised: Instead of jointly training a localization and classification model, we learn box annotations from image class labels, in a similar spirit to weakly-supervised learning.

Original: Empirically, we focus extensively on the transfer learning setting from one single data abundant training task to data-scarce (one-way few-shot) test tasks.
Revised: Empirically, we focus extensively on the transfer learning setting from one single data abundant training task to data-scarce (oneway few-shot) test tasks (Taylor & Stone, 2009; Zhu et al., 2020).

Original: To make a trained RL agent generalizable to new unseen test cases, one can either expose the agent to as many cases as possible during training, or make the agent more speciﬁc but adaptable upon changed scenarios.
Revised: To make a trained RL agent generalizable to new unseen test cases, one can either expose the agent to as many cases as possible during training, or make the agent more specific but adaptable upon changed scenarios.

Original: Our approach belongs to the latter.
Revised: Our approach belongs to the latter.

Original: The transferability of our reward signal from training to test crucially relies on the generalization ability of the learned ordinal representation.
Revised: The transferability of our reward signal from training to test crucially relies on the generalization ability of the learned ordinal representation.

Original: If the ordinal preference does not hold in the test domain, the proposed test-time policy adaptation scheme will not work.
Revised: If the ordinal preference does not hold in the test domain, the proposed test-time policy adaptation scheme will not work.

Original: By adapting the representation with self-supervised objectives (Hansen et al., 2021), this issue might be remedied.
Revised: By adapting the representation with self-supervised objectives (Hansen et al., 2021), this issue might be remedied.

Original: Although our work focuses on the generalization ability of a single object localizer, extensions to multiple objects can possibly be done by applying it multiple times and marking the found regions, in a similar manner as (Caicedo & Lazebnik, 2015).
Revised: Although our work focuses on the generalization ability of a single object localizer, extensions to multiple objects can possibly be done by applying it multiple times and marking the found regions, in a similar manner as (Caicedo & Lazebnik, 2015).

Original: Curriculum learning with a designed sequence of targets in the exemplary set may guide the agent toward solving challenging localization tasks.
Revised: Curriculum learning with a designed sequence of targets in the exemplary set may guide the agent toward solving challenging localization tasks.

Original: We leave those extensions for future study.
Revised: We leave those extensions for future study.

Original: R EPRODUCIBILITY S TATEMENT
Revised: A CKNOWLEDGEMENT

Original: Code is available at: https://anonymous.4open.science/r/Localization-by-Ordinal-Embedd ing-2F
Revised: This research has been partially funded by the following grants ARO MURI 805491, NSF IIS1793883, NSF CNS-1747778, NSF IIS 1763523, DOD-ARO ACC-W911NF, NSF OIA-2040638 to

Original: More implementation details and experiment results can be found in Appendix.
Revised: D. Metaxas. We would like to thank anonymous reviewers for their comments and suggestions

Original: A A PPENDIX : I MPLEMENTATION D ETAILS A. A LGORITHM P IPELINE
Revised: A A PPENDIX : I MPLEMENTATION D ETAILS A. A LGORITHM P IPELINE

Original: General supervised training methods are usually class-agnostic and require exposure to a large number of training classes, box-image pairs, and foreground and background variations in order to generalize well.
Revised: General supervised training methods are usually class-agnostic and require exposure to a large number of training classes, box-image pairs, and foreground and background variations in order to generalize well.

Original: In contrast, we allow specialized agent to be trained, with the ability of adaptation to changes during the test time.
Revised: In contrast, we allow specialized agent to be trained, with the ability of adaptation to changes during the test time.

Original: Our approach is based on the feature similarity with query images, which departs from previous bounding-box regression and RL approaches based on objectiveness.
Revised: Our approach is based on the feature similarity with query images, which departs from previous bounding-box regression and RL approaches based on objectiveness.

Original: Compared to general supervised training or ﬁne-tuning methods, our approach is able to ﬂexibly make use of various types of data in these phases.
Revised: Compared to general supervised training or fine-tuning methods, our approach is able to flexibly make use of various types of data in these phases.

Original: This is summarized in Table 6.
Revised: This is summarized in Table 8.

Original: [Table]
Revised: [Table]

Original: Our agent cantake both  human feedback in terms of the exemplary set and perform test-time policy adaptation using unlabeled test data.
Revised: Our agent can both take human feedback in terms of the exemplary set and perform test-time policy adaptation using unlabeled test data.

Original: It includes three stages: ordinal representation pre-training, RL agent training, and test-time adaptation.
Revised: It includes three stages: ordinal representation pre-training, RL agent training, and test-time adaptation.

Original: Details are as follows:
Revised: Details are as follows:

Original: Stage 1: Pretrain ordinal embedding for state representation and reward In this stage, we assume pairs of groundtruth bounding box and training image are available.
Revised: Stage 1: Pre-train ordinal embedding for state representation and reward In this stage, we assume pairs of ground-truth bounding box and training image are available.

Original: We train the ordinal embedding by attaching a projection head after RoI encoder, as Figure 6 shows.
Revised: We train the ordinal embedding by attaching a projection head after RoI encoder, as Figure 5 shows.

Original: RoI encoder is composed of image encoder and an RoI alignment layer (He et al., 2017).
Revised: RoI encoder is composed of image encoder and a RoIAlign layer (He et al., 2017).

Original: It extracts corresponding bounding box feature directly from image feature output by image encoder.
Revised: It extracts corresponding bounding box feature directly from image feature output by image encoder.

Original: After training, all the modules are ﬁxed.
Revised: After training, all the modules are fixed.

Original: The output of RoI Encoder then becomes the state for agent, and output of projection head is then used for reward computation.
Revised: The output of RoI Encoder then becomes the state for agent, and output of projection head is then used for reward computation.

Original: On Corrupted MNIST (cMNIST) dataset, We attach a decoder in order to train the encoder, but discard it during inference.
Revised: On Corrupted MNIST (cMNIST) dataset, we attach a decoder in order to train the encoder, but discard it during inference.

Original: The loss function is deﬁned as
Revised: The loss function is defined as

Original: [Equation]
Revised: [Equation]

Original: where we set λ 1 = 0 . 1 , and L reconstruct is mean square error loss to measure the reconstruction ability of the autoencoder.
Revised: where we set λ 1 = 0 . 1 , and L reconstruct is mean square error loss to measure the reconstruction ability of the autoencoder.

Original: On CUB and COCO dataset, we adopt the pretrained ImageNet encoder, thus no additional decoder is needed.
Revised: On CUB and COCO dataset, we adopt the pre-trained ImageNet encoder, thus no additional decoder is needed.

Original: The triplet loss L triplet is optimized, such that the embedding of positive bounding boxes to be closer to the embedding of anchor than negative boxes.
Revised: The triplet loss L triplet is optimized, such that the embedding of positive bounding boxes are closer to the embedding of anchor than negative boxes.

Original: Different from traditional deﬁnition of triplet loss, our positive and negative boxes are not ﬁxed.
Revised: Different from traditional definition of triplet loss, our positive and negative boxes are not fixed.

Original: When comparing two perturbed boxes, the positive box is always the one with larger IoU.
Revised: When comparing two perturbed boxes, the positive box is always the one with larger IoU.

Original: Thus, a box with IoU=0.
Revised: Thus, a box with IoU=0.

Original: could be positive box if the negative box has even smaller IoU.
Revised: could be positive box if the negative box has even smaller IoU.

Original: The anchor box is not restricted to ground truth box of the image instance.
Revised: The anchor box is not restricted to ground truth box of the image instance.

Original: For example, we use the mean vector of ordinal embeddings in this paper, computed from groundtruth boxes of images from the exemplary set.
Revised: For example, we use the mean vector of ordinal embeddings in this paper, computed from ground-truth boxes of images from the exemplary set.

Original: [Table]
Revised: [Table]

Original: Stage 2: Localization agent training In this stage, we assume the groundtruth bounding box is available for each training image.
Revised: Stage 2: Localization agent training In this stage, we assume the ground-truth bounding box is available for each training image.

Original: All the models are trained and tested on the same datasets as stage 1.
Revised: All the models are trained and tested on the same datasets as stage 1.

Original: We ﬁx all the layers in RoI encoder and projection head.
Revised: We fix all the layers in RoI encoder and projection head.

Original: The adopted RNN model is shown in Figure 7.
Revised: The adopted RNN model is shown in Figure 6.

Original: In each step, the agent takes the RoI encoder output e roi and the hidden state of RNN h t − 1 , concatenated as the state representation.
Revised: In each step, the agent takes the RoI encoder output e roi and the hidden state of RNN h t − 1 , concatenated as the state representation.

Original: Then it predicts the action for next step by sampling according to the logits.
Revised: Then it predicts the action for next step by sampling according to the logits.

Original: The loss is deﬁned as
Revised: The loss is defined as

Original: [Equation]
Revised: [Equation]

Original: (6) where L entropy prevents the agent from being stuck in the local minimum, π is the policy, and R is the reward deﬁned in Eq.
Revised: [Equation] where L entropy prevents the agent from being stuck in the local minimum, π is the policy, and R is the reward defined in Eq.

Original: Stage 3: Test-time adaptation To adapt to new class, we make use of a limited number of groundtruth box cropped images in the test exemplary set.
Revised: Stage 3: Test-time adaptation To adapt to new class, we make use of a limited number of groundtruth-box cropped images in the test exemplary set.

Original: We leverage abundant unlabelled images and this exemplary set to update the agent.
Revised: We leverage abundant unlabelled images and this exemplary set to update the agent.

Original: We ﬁx the RoI encoder and projection head, and only update the agent.
Revised: We fix the RoI encoder and projection head, and only update the agent.

Original: The process is almost the same as stage 2, except that c in reward (Eq. 4) is the prototype of the test exemplary set.
Revised: The process is almost the same as stage 2, except that c in reward (Eq. 4) is the prototype of the test exemplary set.

Original: In summary, the conﬁguration of different model components and how they are updated is listed in
Revised: In summary, the configuration of different model components and how they are updated is listed in

Revised: Table 9.

Revised: The losses for each dataset and training stage are in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. E XPERIMENTS D ETAILS
Revised: A. E XPERIMENTS D ETAILS

Revised: For MNIST, we use three convolutional layers with ReLU activation after each layer as image encoder, while the same but mirrored structure as decoder to learn an autoencoder, and then attach ROIAlign layer followed by two fully connected ( fc ) layers as projection head for ordinal reward learning.

Revised: For the CUB and the COCO datasets, we adopt layers before conv5_3 of VGG-16 pre-trained on

Revised: ImageNet as encoder unless otherwise specified.

Revised: The projection head uses the same structure as before but with more units for each fc layer.

Original:  We list experiment details for the datasets used in Section 4 as follows.
Revised: All of our models were trained with the Adam optimizer (Kingma & Ba, 2015). We set margin m = 60 in all the experiments heuristically. All the models take less than one hour to finish training, implemented on PyTorch on a single NVIDIA A100 GPU. We list experiment details for the datasets used in Section 4 as follows.

Original: Corrupted MNIST Dataset.
Revised: Corrupted MNIST Dataset.

Original: Examples of the four types of corrupted MNIST (cMNIST) images are shown in Figure 8.
Revised: Examples of the four types of corrupted MNIST (cMNIST) images are shown in Figure 7.

Original: In stage 1 and 2, we randomly sample 50 images of one class in original
Revised: In stage 1 and 2, we randomly sample 50 images of one class in original

Original: MNIST training set, and add noise as our training set.
Revised: MNIST training set, and add noise as our training set.

Original: During testing, the models are evaluated on all the corrupted MNIST test set.
Revised: During testing, the models are evaluated on all the corrupted MNIST test set.

Original: In stage 3, to adapt to a new digit, we annotate a limited number of images as exemplary set for adaptation and use all the (corrupted) MNIST training set from that digit.
Revised: In stage 3, to adapt to a new digit, we annotate a limited number of images as exemplary set for adaptation and use all the (corrupted) MNIST training set from that digit.

Original: We then test the agent on all (corrupted)
Revised: We then test the agent on all (corrupted)

Original: MNIST test set of the same digit.
Revised: MNIST test set of the same digit.

Original: There is no overlap between the training and test set used in stage 3.
Revised: There is no overlap between the training and test set used in stage 3.

Original: CUB Dataset.
Revised: CUB Dataset.

Original: In stage 1 and 2, we train on 15 warbler classes with class id between 158 and 172.
Revised: In stage 1 and 2, we train on 15 warbler classes with class id between 158 and 172.

Original: There are 896 images in total.
Revised: There are 896 images in total.

Original: Then test the models on 5 new warbler classes with class id between 178 and 182, resulting in 294 images in total.
Revised: Then test the models on 5 new warbler classes with class id between 178 and 182, resulting in 294 images in total.

Original: In stage 3, the number and class ids of images for each class are presented in Table 9.
Revised: In stage 3, the number and class ids of images for each class are presented in Table 11.

Original: We also randomly select limited number of images as exemplary set and use all unlabled data for adaptation.
Revised: We also randomly select limited number of images as exemplary set and use all unlabled data for adaptation.

Original: The CorLoc is calculated using all the images of this class.
Revised: The CorLoc is calculated using all the images of this class.

Original: [Table]
Revised: [Table]

Original: COCO Dataset.
Revised: COCO Dataset.

Original: For the results of Figure 5, we train on one of the ﬁve classes: cat, cow, dog, horse, zebra , then adapt to another four classes.
Revised: For the results of Figure 3, we train on one of the five classes: cat, cow, dog, horse, zebra , then adapt to another four classes.

Original: The detailed number of each class for training and testing in stage 1 and 2 is shown in Table 10.
Revised: The detailed number of each class for training and testing in stage 1 and 2 is shown in Table 12.

Original: [Table]
Revised: [Table]

Original: In stage 3, the agent is tested on new class in target domain, within which we annotate limited number of images for adaptation.
Revised: In stage 3, the agent is tested on new classes in target domain, within which we annotate limited number of images for adaptation.

Original: In comparison with few-shot object detection experiment, the models in stage 1 and 2 are trained using one single class of the 5 classes: elephant, sheep, giraffe, laptop and suitcase.
Revised: In comparison with few-shot object detection experiment, the models in stage 1 and 2 are trained using one single class of the 5 classes: elephant, sheep, giraffe, laptop and suitcase.

Original: Then being adapted to the four classes in Table 5.
Revised: Then being adapted to the four classes in Table 4.

Original: Note that the ﬁve classes are in the base classes, and the four classes are in the novel classes used in other methods.
Revised: Note that the five classes are in the base classes, and the four classes are in the novel classes used in other methods.

Original: Thus, it’s harder to transfer from one single class to another due to scarcity of training data and training class.
Revised: Thus, it’s harder to transfer from one single class to another due to scarcity of training data and training class.

Original: Tablereports the average CorLoc from the ﬁve classes to each target class.
Revised: Table 4 reports the average CorLoc from the five classes to each target class.

Original: We also provide the results of each source class in Table 11.
Revised: We also provide the results of each source class in Table 13.

Original: From this table, we can see that transferring from related classes with target class usually performs better.
Revised: From this table, we can see that transferring from related classes with target class usually performs better.

Original: For example, the CorLoc from laptop and suitcase are lower than other three animal classes, especially before adaptation.
Revised: For example, the CorLoc from laptop and suitcase are lower than other three animal classes, especially before adaptation.

Original: After adaptation, the gap becomes smaller.
Revised: After adaptation, the gap becomes smaller.

Original: [Table]
Revised: [Table]

Original: Selective localization.
Revised: Selective localization.

Original: We investigate the agent’s ability in localizing the object speciﬁed by the query set, when the set of images have two common objects.
Revised: We investigate the agent’s ability in localizing the object specified by the query set, when the set of images have two common objects.

Original: We use random patched MNIST, where each image has digit 3 and digit 4.
Revised: We use random patched MNIST, where each image has digit 3 and digit 4.

Original: First, the RoI encoder and projection head is trained with an additional contrastive loss to enlarge the distance between the two digits in embedding space, 
Revised: First, the RoI encoder and projection head are trained with an additional contrastive loss to enlarge the distance between the two digits in embedding space, loss

Original: [Equation]
Revised: [Equation]

Original: where loss trip = loss trip 3 + loss trip 4 , learning two local ordinal structure around each class center in embedding space.
Revised: where loss trip = loss trip 3 + loss trip 4 , learning two local ordinal structure around each class center in embedding space.

Original: We set the margin for both triplet losses as 10, and the margin for contrastive loss as 320 heuristically.
Revised: We set the margin for both triplet losses as 10, and the margin for contrastive loss as 320 heuristically.

Original: We found that the larger the gap between the two margins, the better the performance.
Revised: We found that the larger the gap between the two margins, the better the performance.

Original: Detailed results can be found in Table 17 and its related discussion.
Revised: Detailed results can be found in Table 17 and its related discussion.

Original: After learning the
Revised: After learning the

Original: RoI encoder and projection head, we train the agent with the reward deﬁned with Eq. 4 in Sect.
Revised: RoI encoder and projection head, we train the agent with the reward defined with Eq. 4 in Sect.

Original: 3.3, where c is the prototype embedding of the targeted digit exemplary set.
Revised: 3.3, where c is the prototype embedding of the targeted digit exemplary set (exemplary size is 5).

Original: Figure 3 in Sect.
Revised: Figure 8 shows an illustrative example of query object localization with task adaptation. Given two common objects coexisting in the image environment, the agent is trained to pick up one from the two during training, while quickly adapt to pick another one during testing.

Original: 4 shows an example where the  exemplary size is 5.
Revised: Different from unsupervised co-localization approaches (Wei et al., 2017) relying on low-level image clues to distinguish foreground and background, our learning objectives can be specified explicitly by switching the train/test exemplary sets. With test-time adaptation, our agent can handle even partially conflicting learning objectives.

Original: B A DDITIONAL E XPERIMENTS B. A BLATION OF O RDINAL E MBEDDING AND RL
Revised: B A DDITIONAL E XPERIMENTS B. A BLATION OF O RDINAL E MBEDDING AND RL

Revised: Setting and main results on corrupted MNIST dataset.

Revised: We analyze the effectiveness of using ordinal embedding in terms of representation and reward first on synthetic datasets of corrupted

Revised: MNIST digits under cluttered, random patch, Gaussian, or impulse noises (see Figure 7 in Appendix for examples).

Original: We analyze theeffectiveness of ordinal embedding and RL component separately.
Revised: We want to find the answers to two main questions listed below: Is the embedding distance based reward signal as effective as the IoU counterpart? Does the ordinal triplet loss benefit the state representation learning? Accordingly, we consider baselines in which embeddings are trained with only autoencoder or jointly with the ordinal projection head.

Original: First, we remove
Revised: Besides, we also compare our embedding distance based reward against the IoU based reward used in Caicedo & Lazebnik (2015). Without loss of generality, the agent is trained on images of digit 4, and tested on images from all rest classes. In particular, we are interested in evaluating the sample efficiency of different approaches. The results under different training set sizes are shown in Figure 9.

Revised: To distract the localization agent, cluttered noises with random 6 × 6 crops of digits are added to the background.

Revised: All comparisons are based on the policy gradient (PG) method.

Revised: With ordinal embedding present in both the representation and reward (“AE+Ord+Embed"), our model performance is consistently better than other settings, especially when the training set size is small.

Revised: The benefit of a triplet loss is demonstrated by the comparison between “AE+Ord+IoU" and “AE+IoU".

Revised: Intuitively, the similarity to a queried object is also a more natural goal than objectives defined by bounding boxes IoU, which is used in both generic object detection or localization and previous RL-based approaches.

Revised: Signed reward or not.

Revised: Another intriguing question is that other than benefiting state representation learning, whether the embedding-based reward itself can bring any additional benefits than the IoUbased reward.

Revised: From Figure 9, there is a large gap between “AE+Ord+Embed" and “AE+Ord+IoU", especially when the training set size is small.

Original: RL, and substitute it  with a simple linear search approach.
Revised: This is counter-intuitive as the ordinal reward is trained with supervision from IoU in embedding space, supposably it shall not bring any additional information. To further analyze this phenomenon, we take the sign operation off in Eq. and retrain the agent. In an experiment with 50 training images, without sign operation, the localization accuracy increases from 90.92% to 94.36% on test digit 4 , and from 82.43% to 88.64% on other digit test set.

Original: Speciﬁcally, we adopt selective search (Uijlings et al., 2013) for candidate boxes generation.
Revised: Perhaps, this improvement can be attributed to more informative feedback provided by the continuous-valued, unsigned reward. On policy vs. off policy. Many deep RL approaches are in favor of using deep Q-Network (DQN) to train an agent. Different from Caicedo & Lazebnik (2015) and Jie et al.

Original: The candidate boxes are ranked according to their embedding distances to the prototype, and the one with  the smallest embedding distance is returned as the ﬁnal output.
Revised: , we apply Policy Gradient (PG) to optimize it. Besides, we adopt a top-down search strategy through a RNN, while they used a vector of history actions to encode memory. We evaluate these design choices with four baselines, with "AE+IoU" setting, and trained on the same 600 sampled cluttered digit 4 images.

Original: We considered two backbone networks as the embedding, including
Revised: As

Original: ImageNet pretrained VGG-16 and faster-RCNN VGG-16 trained locally on COCO.
Revised: Table 14 shows, the agent achieves the best performance with "PG+RNN".

Original: We also compare both with and without the ordinal component, making it a 2 × 2 ablation study.
Revised: We find that empirically, with history action vectors the accuracy becomes worse when the agent is trained by DQN [Table] [Table]

Original: It can be seen from the blue and green bars from Figure 9 and Figure 10 that with ordinal structure, the ranking method performs much more better. We found that pre-training with the proposed ordinal loss signiﬁcantly improves the rank consistency of these backbone networks.
Revised: More results on COCO dataset. Figure 10 provides results using ImageNet pre-trained VGG16 network as backbone with the same training strategy as Figure 4.

Original: To further demonstrate the effectiveness of ordinal embedding, we compute the Spearman’s Rho rank correlation between embedding distance to the prototype and IoU to the groundtruth.
Revised: To further demonstrate the effectiveness of ordinal embedding, we compute the Spearman’s Rho rank correlation between embedding distance to the prototype and IoU to the ground-truth.

Original: The results are shown in Figure 11.
Revised: The results are shown in Figure 11.

Original: Here we also add CLIP (Radford et al., 2021) pretrained ViT as backbone for comparison.
Revised: Here we also add CLIP (Radford et al., 2021) pre-trained ViT as backbone for comparison.

Original: The rank correlation is smaller than − 0 .
Revised: The rank correlation is smaller than − 0 .

Original: 7 on all backbones with ordinal embedding, exhibiting ordinal embedding preserves the IoU order, thus is better for the ranking purpose.
Revised: 7 on all backbones with ordinal embedding, exhibiting ordinal embedding preserves the IoU order, thus is better for the ranking purpose.

Original: Although pretty effective, embedding distance is still not a perfect indicator of the ranking of 
Revised: Although pretty effective, embedding distance is still not a perfect indicator of the ranking of IoU.

Original: IoU. Thus, directly formulate the object localization problem as a search problem leads to suboptimal localization accuracy.
Revised: Thus, directly formulate the object localization problem as a search problem leads to suboptimal localization accuracy.

Original: In contrast, being able to learn across different images, the RL localization agent shows more tolerance for the defects of the reward function.

Original: We analyze the beneﬁts of RL component.

Original: For both Figure and Figure 10, when using ordinal embedding, the model with RL is generally better than ranking, especially after adaptation.

Original: Ordinal component is necessary for several possible reasons.

Original: First, the ordinal component makes the generic pre-trained network more speciﬁc to the downstream object localization task.

Original: Second, it helps remove the dataset bias if the pre-trained network is non-native.

Original: Third, ordinal pretraining encourages the reward signal to reﬂect step-by-step improvements.

Original: Encouraged by the strong baseline performance of Faster RCNN

Original: (detailed in Table 13 and Table 14), we investigate whether it is adequate to use Faster RCNN embedding in the RL reward function, without ordinal pre-training.

Original: Since Faster RCNN emebdding is also trained for object localization on the COCO dataset, it can better separate out the effects of the ordinal pretraining component.

Original: Result is shown in Table 12.

Original: Comparing to the second row of Table 15, it can be seen that without ordinal pretraining the performance degrades signiﬁcantly, where the

Original: CorLoc is much lower even in source domain.

Original: [Table]
Revised: [Table]

Original: We compare our framework with a strong supervised object localiztion baseline, Faster RCNN (Ren et al., 2016).

Original: Both methods are trained in one class (foreground vs. background as the classiﬁcation labels in Faster RCNN) and adapted to a different class.

Original: We use VGG-16 model pretrained on

Original: ImageNet, and ﬁnetune and test on each of the ﬁve classes: cow, cat, dog, horse, zebra .

Original: The results on source domain are shown in Table 13.

Original: It shows that Faster RCNN can also be made into a class-speciﬁc model and it still yields superior performance on source domain.

Original: On the target domain, we ﬁnetune Faster RCNN using only support set for each target class.

Original: The results are shown in Table 14.

Original: It can be seen that our method works better on new classes with test-time adaptation over the traditional ﬁne-tuning of Faster RCNN.

Original: Note that ﬁne-tuning requires image-box pairs in the target domain while our policy adaptation approach does not.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: B. C OMPARE DIFFERENT OFF - THE - SHELF NETWORKS AS THE BACKBONE
Revised: C OMPARE DIFFERENT OFF - THE - SHELF NETWORKS AS THE BACKBONE

Original: It is interesting to study the choices of off-the-shelf pre-trained networks as the backbone, such as
Revised: It is interesting to study the choices of off-the-shelf pre-trained networks as the backbone, such as

Original: CLIP or supervised embedding provided by Faster RCNN or a classiﬁcation network.
Revised: CLIP or supervised embedding provided by Faster RCNN or a classification network.

Original: Since these networks have been exposed to large-scale dataset, it is interesting to see whether policy adaptation is still needed.
Revised: Since these networks have been exposed to large-scale dataset, it is interesting to see whether policy adaptation is still needed.

Original: We compare different backbones on both source domain and target domain using our method.
Revised: We compare different backbones on both source domain and target domain using our method.

Original: Table 15 reports the CorLoc of training and testing on source domain.
Revised: Table 15 reports the CorLoc of training and testing on source domain.

Original: The large-scale pretraind ViT backbone consistently performs the best, comparing to the other two VGG-16 models.
Revised: The large-scale pre-traind ViT backbone consistently performs the best, comparing to the other two VGG-16 models.

Original: Table 16 compares the backbones on target domain with new classes.
Revised: Table 16 compares the backbones on target domain with new classes.

Original: The test-time adaptation still brings a large margin of improvement.
Revised: The test-time adaptation still brings a large margin of improvement.

Original: Interestingly, we also found that the Faster-RCNN embedding offers the best performance on the target domain before adaptation, while the ViT network trained on CLIP dataset provides the best performance after adaptation, indicating different generalization mechanisms.
Revised: Interestingly, we also found that the Faster-RCNN embedding offers the best performance on the target domain before adaptation, while the ViT network trained on CLIP dataset provides the best performance after adaptation, indicating different generalization mechanisms.

Original: They both outperform the ImageNet backbone intially considered.
Revised: They both outperform the ImageNet backbone initially considered.

Original: [Table]
Revised: [Table]

Original:  T HE EFFECTS OF M ARGIN
Revised: B. T HE EFFECTS OF M ARGIN

Original: The margin in triplet loss is selected heuristically.
Revised: The margin in triplet loss is selected heuristically.

Original: It is not sensitive except in the selective localization experiment (Figure 3), where there are two different digits in each image.
Revised: It is not sensitive except in the selective localization experiment (Figure 8), where there are two different digits in each image.

Original: For this experiment, we trained two ordinal structures around each digit using triplet loss with margin m [Table] and add additional contrastive loss with margin m 2 to separate the centers of the two different digits as far as possible.
Revised: For this experiment, we trained two ordinal structures around each digit using triplet loss with margin m 1 , and add additional contrastive loss with margin m 2 to separate the centers of the two different digits as far as possible.

Original: And we found out that the model works best when m 2 (cid:29) m 1 .
Revised: And we found out that the model works best when m 2 ≫ m 1 .

Original: In our experiment, we set m 1 = 10 , m 2 = 320 .
Revised: In our experiment, we set m 1 = 10 , m 2 = 320 .

Original: The results of using different set of m 1 and m 2 are presented in Table 17.[Table] B.
Revised: The results of using different set of m 1 and m 2 are presented in Table 17.

Original: [Table]
Revised: [Table]

Original:  S IZE OF TRAINING SET
Revised: B. S IZE OF TRAINING SET

Original: In this experiment [Table] we train on class giraffe in stage 1 and 2, then adapt to cat, cow, dog and horse.
Revised: In this experiment, we train on class giraffe in stage 1 and 2, then adapt to cat, cow, dog and horse.

Original: We set the training set size as [200, 500, 700, 1146], exemplary set size as 5.
Revised: We set the training set size as [200, 500, 700, 1146], exemplary set size as 5.

Original: We compare our after adaptation results with TFA w/ fc (Wang et al., 2020) one-way 5-shot setting, where their model is trained on all 60 base classes, while ours is only trained on one of the base classes.
Revised: We compare our after adaptation results with TFA w/ fc (Wang et al., 2020) one-way 5-shot setting, where their model is trained on all 60 base classes, while ours is only trained on one of the base classes.

Original: Figure 12 shows the results, in which the dotted lines are the results of TFA w/ fc.
Revised: Figure 12 shows the results, in which the dotted lines are the results of TFA w/ fc.

Original: Ours performs muchmore better than their method.
Revised: Ours performs much better than their method.

Original: Except  cat class, ours is better than theirs with only 200 images for training.
Revised: Except on cat class, ours is better than theirs with only 200 images for training.

Original: B. S IZE OF E XEMPLARY S ET
Revised: B. S IZE OF E XEMPLARY S ET

Original: We also compare the effect of different size of exemplary set during training and adaptation on
Revised: We also compare the effect of different size of exemplary set during training and adaptation on

Original: CUB-warbler dataset.
Revised: CUB-warbler dataset.

Original: During training stage, we use shufﬂe proto training strategy, and set exemplary set size as 2, 5, 15, 25.
Revised: During training stage, we use shuffle proto training strategy, and set exemplary set size as 2, 5, 15, 25.

Original: The results without adaptation on test set are in Table 18.
Revised: The results without adaptation on test set are in Table 18.

Original: Both OrdAcc and CorLoc increase with exemplary set size.
Revised: Both OrdAcc and CorLoc increase with exemplary set size.

Original: For adaptation stage, the range of exemplary set size is from 2 to 200.
Revised: For adaptation stage, the range of exemplary set size is from 2 to 200.

Original: And the results are in Table 19.
Revised: And the results are in Table 19.

Original: The test performance does not increase much with the exemplary set size.
Revised: The test performance does not increase much with the exemplary set size.

Original: One possible explanation is that the data points in embedding space are compact, thus prototype doesn’t change much when increasing exemplary set size.
Revised: One possible explanation is that the data points in embedding space are compact, thus prototype doesn’t change much when increasing exemplary set size.

Original: We will analyze the inﬂuence of multiple protoptypes per class in future experiments [Table] [Table] [Table]
Revised: We will analyze the influence of multiple prototypes per class in future experiments.

Original: B. P ROTOTYPE S ELECTION
Revised: B. P ROTOTYPE S ELECTION

Original: We further evaluate the choice of anchor in the triplet loss for both the pretraining of state represtation and the ordinal reward for the training of agent.
Revised: We further evaluate the choice of anchor in the triplet loss for both the pre-training of state representation and the ordinal reward for the training of agent.

Original: We study ( i ) whether ordinal embedding can be  trained in reference to an anchor from a different image instances, ( ii ) whether it is advantageous to use the proptotype embedding of an exemplary set, rather than instance embeddings
Revised: We study ( i ) whether ordinal embedding can be [Table] [Table] trained in reference to an anchor from a different image instances, ( ii ) whether it is advantageous to use the prototype embedding of an exemplary set, rather than instance embeddings

Original: [Table]
Revised: [Table]

Original: and ( iii ) whether mimicking the test condition in training yields any improvement.
Revised: and ( iii ) whether mimicking the test condition in training yields any improvement.

Revised: [Table]

Original: We use the CUB-Warbler dataset with more foreground background variations than the corrupted
Revised: We use the CUB-Warbler dataset with more foreground background variations than the corrupted

Original: MNIST dataset.
Revised: MNIST dataset.

Original: The train and test set contains 15 and 5 disjoint ﬁne-grained classes respectively, resulting 896 images for training (viewed as a single class) and 294 for testing.
Revised: The training and test set contains 15 and 5 disjoint fine-grained classes respectively, resulting 896 images for training (viewed as a single class) and 294 for testing.

Original: Table 20 shows the OrdAcc and CorLoc in four settings.
Revised: Table 20 shows the OrdAcc and CorLoc in four settings.

Original: “Self", use the embedding from images cropped by the ground-truth box from the same instance; “Shufﬂe self", use the ground-truth box cropped image emebedding from a different instance; Similarly, “Proto", use the prototype of a subgroup containing the training instance within the same batch; “Shufﬂe proto (SP)", use the prototype of a subgroup[Table] from a different batch without the training instance.
Revised: “Self" uses the embedding from images cropped by the ground-truth box from the same instance; “Shuffle self" uses the ground-truth box cropped image emebedding from a different instance; Similarly, “Proto" uses the prototype of a subgroup containing the training instance within the same batch; “Shuffle proto (SP)" uses the prototype of a subgroup from a different batch without the training instance.

Original: Results suggest that this training strategy brings compactness to the training set, constructing an ordinal structure around the cluster.
Revised: Results suggest that this training strategy brings compactness to the training set, constructing an ordinal structure around the cluster.

Original: For “Shufﬂe proto", while the OrdAcc is lower than others, its CorLoc is the best with large margin.
Revised: For “Shuffle proto", while the OrdAcc is lower than others, its CorLoc is the best with large margin.

Original: Matching the condition between training and testing indeed improves generalization to new classes on this dataset.
Revised: Matching the condition between training and testing indeed improves generalization to new classes on this dataset.

Original: B. A NNOTATION REFINEMENT
Revised: B. A NNOTATION REFINEMENT

Original: Annotation collection plays an important role in building machine learning systems.
Revised: Annotation collection plays an important role in building machine learning systems.

Original: It is one task that could beneﬁt greatly from automation, especially in cost-sensitive applications.
Revised: It is one task that could benefit greatly from automation, especially in cost-sensitive applications.

Original: The human labeling efforts can be reduced, not only in terms of the number of annotate samples per class, number of annotate classes (one-class at the minimum), but also the level of accuracy required.
Revised: The human labeling efforts can be reduced, not only in terms of the number of annotate samples per class, number of annotate classes (one-class at the minimum), but also the level of accuracy required.

Original: We explore the direction of iterative reﬁnement of annotation quality with minimal human guidance.
Revised: We explore the direction of iterative refinement of annotation quality with minimal human guidance.

Original: Given an agent trained extensively with inexact human annotations, can we adapt it to reﬁne the annotation with a few well-annotated examples?
Revised: Given an agent trained extensively with inexact human annotations, can we adapt it to refine the annotation with a few well-annotated examples?

Original: We purposefully enlarge the ground truth bounding boxes in the corrupted MNIST and CUB datasets, and adapt the trained agent using the original ground truth box in an exemplary set of size 5 .
Revised: We purposefully enlarge the ground truth bounding boxes in the corrupted MNIST and CUB datasets, and adapt the trained agent using the original ground truth box in an exemplary set of size 5 .

Original: For corrupted MNIST dataset, the agent, RoI encoder and projection head are ﬁrst trained withdigit 3 images with loose (38 × 38 ) bounding box annotation.
Revised: For corrupted MNIST dataset, the agent, RoI encoder and projection head are first trained withdigit 3 images with loose (38 × 38 ) bounding box annotation.

Original: Then the agent is adapted to digit 1 images, where the exemplary size=5, and the ground-truth boxes are 28 × 28.
Revised: Then the agent is adapted to digit 1 images, where the exemplary size=5, and the ground-truth boxes are 28 × 28.

Original: For CUB dataset, during training, the ground-truth box in each resized 224 × 224 image is expanded 10 pixels each side.
Revised: For CUB dataset, during training, the ground-truth box in each resized 224 × 224 image is expanded 10 pixels each side.

Original: During adaptation we use the original ground-truth boxes.
Revised: During adaptation we use the original ground-truth boxes.

Original: In adaptation from entire body to head experiment on CUB dataset, we train on original entire body ground-truth bounding box.
Revised: In adaptation from entire body to head experiment on CUB dataset, we train on original entire body ground-truth bounding box.

Original: During adaptation, 15 randomly sampled exemplar images are annotated with head bounding box.
Revised: During adaptation, 15 randomly sampled exemplar images are annotated with head bounding box.

Original: The agent is trained to adapt to localize head by minimizing the distance between embedding of predicted box and prototype of head embeddings.
Revised: The agent is trained to adapt to localize head by minimizing the distance between embedding of predicted box and prototype of head embeddings.

Original: The results are shown in Figure 13 (left panel).
Revised: The results are shown in Figure 13 (left panel).

Original: Preliminary results with agent trained on the entire bird body and adapt to bird head are shown in Figure 13 (right panel), with a few successful cases on the ﬁrst row and failure cases on the second row.B.
Revised: Preliminary results with agent trained on the entire bird body and adapt to bird head are shown in Figure 13 (right panel), with a few successful cases on the first row and failure cases on the second row.

Original: [Figure]
Revised: [Figure]

Original: F EW -S HOT L OCALIZATION ON O MNIGLOT

Original: The proposed ordinal reward signal also makes our approach amenable to few-shot training, when only a small subset of training images per class are annotated.
Revised: The proposed ordinal reward signal also makes our approach amenable to few-shot training, when only a small subset of training images per class are annotated.

Original: Different from the transfer learning setting, in few-shot setting limited annotations across multiple classes are available during training.
Revised: Different from the transfer learning setting, in few-shot setting limited annotations across multiple classes are available during training.

Original: The ordinal reward can be viewed as meta information.
Revised: The ordinal reward can be viewed as meta information.

Original: We evaluate our method under few-shot setting on corrupted Omniglot dataset (Lake et al., 2015) and CUB-warbler dataset.
Revised: We evaluate our method under few-shot setting on corrupted Omniglot dataset (Lake et al., 2015) and CUB-warbler dataset.

Original: For Omniglot,
Revised: For Omniglot,

Original: We put each 28 × 28 character in 84 × 84 random patch background.
Revised: We put each 28 × 28 character in 84 × 84 random patch background.

Original: The train and test set contains different classes respectively, thus 500 images for each set.
Revised: The train and test set containsdifferent classes respectively, thus 500 images for each set.

Original: We randomly sample 100 iterations for training and testing.
Revised: We randomly sample 100 iterations for training and testing.

Original: For CUB-warbler datset, as we did in Sect.
Revised: For CUB-warbler datset, as we did in Sect.

Original: 4.2 we train on the same 15 species from the “Warbler" class, and adapted to 5 new species of “Warbler", thus 896 and 294 images respectively.
Revised: 4.2 we train on the same 15 species from the “Warbler" class, and adapted to 5 new species of “Warbler", thus 896 and 294 images respectively.

Original: We randomly sample 100 and 50 iterations for training and testing.
Revised: We randomly sample 100 and 50 iterations for training and testing.

Original: We use 5-shot 5-way, set exemplary set size as 5, and use proto training strategy for both dataset.
Revised: We use 5-shot 5-way, set exemplary set size as 5, and use proto training strategy for both dataset.

Original: The results are shown in
Revised: The results are shown in

Original: Table 21.
Revised: Table 21.

Original: As an implicit meta learning method, our approach achieves 99 .
Revised: As an implicit meta learning method, our approach achieves 99 .

Original: 94% and 90 .
Revised: 94% and 90 .

Original: 52% CorLoc on the two datasets.
Revised: 52% CorLoc on the two datasets.

Original: We can also leverage explicit meta learning method, such as MAML (Finn et al., 2017) to further improve the results.
Revised: We can also leverage explicit meta learning method, such as MAML (Finn et al., 2017) to further improve the results.

Original: We will leave this part as future work.
Revised: We will leave this part as future work.

Original: Although initial results are promising, more efforts are needed to validate whether the proposed RL approach can achieve state-of-the-art performance, but it is beyond the scope of this work.
Revised: Although initial results are promising, more efforts are needed to validate whether the proposed RL approach can achieve state-of-the-art performance, but it is beyond the scope of this work [Table]

Original: [Table]
Revised: [Table]

Original: C I NTERMEDIATE R ESULTS
Revised: C I NTERMEDIATE R ESULTS

Original: The agent tends to predict expected actions to transform the box so that it can ﬁnally get to the target location.
Revised: The agent tends to predict expected actions to transform the box so that it can finally get to the target location.

Original: It can also be observed from Figure 14 that the embedding distance between predicted boxes and ground-truth boxes is in agreement with the corresponding IoU.
Revised: It can also be observed from Figure 14 that the embedding distance between predicted boxes and ground-truth boxes is in agreement with the corresponding IoU.

Original: This further demonstrates the effectiveness of the localization agent.
Revised: This further demonstrates the effectiveness of the localization agent.