Original: L EARNING T RANSFERABLE R EWARD FOR Q UERY O BJECT L OCALIZATION WITH P OLICY A DAPTATION
Revised: L EARNING T RANSFERABLE R EWARD FOR Q UERY O BJECT L OCALIZATION WITH P OLICY A DAPTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a reinforcement learning based approach tothe problem of query object localization , where an agent is trained to localize objects of interest speciﬁed by a small exemplary set.
Revised: We propose a reinforcement learning based approach to query object localization , for which an agent is trained to localize objects of interest speciﬁed by a small exemplary set.

Original: We learn a transferable reward signal formulated using the exemplary set by ordinal metric learning.
Revised: We learn a transferable reward signal formulated using the exemplary set by ordinal metric learning.

Original: It enables test-time policy adaptation to new environments where the reward signals are not readily available, and outperforms ﬁne-tuning approaches that are limited to annotated images.
Revised: Our proposed method enables test-time policy adaptation to new environments where the reward signals are not readily available, and outperforms ﬁne-tuning approaches that are limited to annotated images.

Original: In addition, the transferable reward can allow repurposingof the trained agent from one speciﬁc class to another class.
Revised: In addition, the transferable reward allows repurposing the trained agent from one speciﬁc class to another class.

Original: Experiments on  corrupted MNIST,  CU-Birds, and  COCO datasets demonstrate the effectiveness of our approach.
Revised: Experiments on a corrupted MNIST, the CU-Birds, and the COCO datasets demonstrate the effectiveness of our approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is increasingly interest in designing  models that can be adapted to unseen environments or repurposed for new tasks, with only minimal human guidance (Vinyals et al., 2016; Snell et al., 2017;
Revised: There is increasing interest in designing machine learning models that can be adapted to unseen environments or repurposed for new tasks, with only minimal human guidance (Vinyals et al., 2016; Snell et al., 2017;

Original: Finn et al., 2017; Sun et al., 2020; Hansen et al., 2021).
Revised: Finn et al., 2017; Sun et al., 2020; Hansen et al., 2021).

Original: A small set of examples can not only serve the purpose of implicitly generalizing the trained model to the new environment during training time, but also provide the chance of updating the  learning objectiveswith the learner during test-time.
Revised: To this end, a small set of examples can not only serve the purpose of implicitly generalizing a trained model to a new environment during training time, but also enable the learner to update its learning objectives during test time.

Original: In this paper, we focus on the reinforcement learning (RL) formulation to the problem of query object localization , where an agent is trained to localize the target object speciﬁed by a small set of exemplary images.
Revised: In this paper, we focus on a reinforcement learning (RL) formulation to the problem of query object localization , in which an agent is trained to localize the target object speciﬁed by a small set of exemplary images.

Original: The vision-based agent can be viewed as a proactive information gatherer (Guo, 2003) that actively interacts with the image environment, following a class-speciﬁc localization policy,  thus will be more suitable for aerial imagery (Xia et al., 2018), robotic manipulation (Kalashnikov et al., 2018), or embodied AI (Savva et al., 2019) tasks.
Revised: Instead of using ﬁxed bounding box proposals, our vision-based agent can be viewed as a proactive information gatherer (Guo, 2003), which actively interacts with an image environment; Furthermore, it follows a class-speciﬁc localization policy, and thus is more suitable for aerial imagery (Xia et al., 2018), robotic manipulation (Kalashnikov et al., 2018), or embodied AI (Savva et al., 2019) tasks.

Original: During test-time, the queried object  to localize may be novel, or the background environment may undergo substantial change, hindering the applicability of class-agnostic agents with  ﬁxed policy.
Revised: During the test time of query object localization, the queried object class to localize may be novel, or the background environment may undergo substantial changes, hindering the applicability of class-agnostic agents with a ﬁxed policy.

Original: When reward signal is  available,  ﬁne-tuning methods (Julian et al., 2020) can effectively adapt agents tothe new environment and yield improved performance. Different with standard RL settings, in our application the reward signal is not available during test-time, as the bounding box annotations are to be found by the localization agent on test images.
Revised: In addition, in standard RL settings for object localization, the reward signal is often available, so ﬁne-tuning methods (Julian et al., 2020) can effectively adapt agents to new environments and yield improved performance; On the contrary, in query object localization, the reward signal is not available at test time, because the bounding box annotations are to be found by the localization agent on test images.

Original: To address this problem, we propose an ordinal metric learning based framework for learning an implicitly transferable reward signal deﬁned with a small exemplary set.
Revised: To address these problems, we propose an ordinal metric learning based framework for learning an implicitly transferable reward signal deﬁned with a small exemplary set.

Original: An ordinal embedding network is pre-trained with data augmentation under a loss function designed to be relevant to the RL task.
Revised: An ordinal embedding network is pre-trained with data augmentation under a loss function designed to be relevant to the RL task.

Original: The reward signal allows explicit updates of the controller in the policy network with continual training during test-time.
Revised: The reward signal allows explicit updates of the controller in the policy network with continual training during test time.

Original: Compared to ﬁne-tuning approaches, the agents can get exposed tothe new environment more extensively with unlimited usage of test images.
Revised: Compared to ﬁne-tuning approaches, the agents can get exposed to new environments more extensively with unlimited usage of test images.

Original: Informed by the exemplary set precisely, the agent is versatile to the change of  localization target.
Revised: Informed by the exemplary set precisely, the agent is adapted to the changes of the localization target.

Revised: Our contributions in this paper are summarized as follows: (1) We propose a novel ordinal metric learning based RL framework to learn a transferable reward signal deﬁned with a small exemplary set for query object localization; (2) Our RL framework enables test-time policy adaptation to new environments where there is no readily available reward signal; (3) Our learned transferable reward allows repurposing a trained agent from localizing one speciﬁc class to another; (4) Extensive experiments on several datasets demonstrate the effectiveness of our proposed approach.

Original:  R ELATED W ORK 2. O BJECT L OCALIZATION
Revised:  R ELATED W ORK 2. O BJECT L OCALIZATION

Original: Compared to bounding-box regression approaches (Redmon et al., 2016; Ren et al., 2016), deep
Revised: Compared to bounding-box regression approaches (Redmon et al., 2016; Ren et al., 2016), deep

Original: RL based object localization approaches (Caicedo & Lazebnik, 2015; Jie et al., 2016) have the advantage of being region-proposal free, with customized search paths to each image environment.
Revised: RL based object localization approaches (Caicedo & Lazebnik, 2015; Jie et al., 2016) have the advantage of being region-proposal free, with customized search paths for each image environment.

Original: Lazebnik
Revised: The speciﬁcity of an agent purely depends on the classes of bounding-boxes used in the reward. They can be made class-speciﬁc (Caicedo & Lazebnik, 2015), but the agent for each class would need to be trained separately.

Original: Despite the rise of crowdsourcing platforms, obtaining ample amount of bounding-box annotations remains costly and error-prone.
Revised: Despite the rise of crowdsourcing platforms, obtaining ample amount of bounding-box annotations remains costly and error-prone.

Original: Furthermore, the quality of annotation often varies, and precise annotations for certain object class may require special expertise from annotators.
Revised: Furthermore, the quality of annotations often varies, and precise annotations for certain object class may require special expertise from annotators.

Original: The emergence of weakly supervised object localization (WSOL) (Song et al., 2014; Oquab et al., 2015; Zhou et al., 2016) methods alleviate the situation,  whichutilize image class labels in deriving bounding box annotations.
Revised: The emergence of weakly supervised object localization (WSOL) (Song et al., 2014; Oquab et al., 2015; Zhou et al., 2016) methods alleviates the situation, in which image class labels are used to derive bounding box annotations.

Original: It is known that WSOL methods have drawbacks of overly relying onthe inter-class discriminative features and not being able to generalize to classes unseen during the training phase.
Revised: It is known that WSOL methods have drawbacks of overly relying on inter-class discriminative features and failing to generalize to classes unseen during the training phase.

Original: We argue that intra-class similarity is a more natural objective for the problem of localizing objects belonging to the target class. 
Revised: We argue that intra-class similarity is a more natural objective for the problem of localizing objects belonging to a target class. "

Original: The name "query object localization" emphasizes more about the class-speciﬁc nature of the localization task, with the relevance deﬁned by the similarity tothe query images.
Revised: Query object localization" emphasizes more about the class-speciﬁc nature of a localization task, with the relevance deﬁned by the similarity to query images.

Original: A similar problem is image co-localization
Revised: A similar problem is image co-localization

Original: (Tang et al., 2014; Wei et al., 2017), where the task is to identify the common objects within a set of images.
Revised: (Tang et al., 2014; Wei et al., 2017), in which the task is to identify the common objects within a set of images.

Original: Co-localization approaches exploit the common characteristics across images to localize objects.
Revised: Co-localization approaches exploit the common characteristics across images to localize objects.

Original: Being unsupervised, co-localization approaches could suffer from ambiguity if there exist multiple common objects or parts, e.g., bird head and body (Jaderberg et al., 2015), which may providethe unwanted common objects as output.
Revised: Being unsupervised, co-localization approaches could suffer from ambiguity if there exist multiple common objects or parts, e.g., bird head and body (Jaderberg et al., 2015), which may provide unwanted common objects as output.

Original: 2. P OLICY A DAPTATION
Revised: 2. P OLICY A DAPTATION

Original: There seems to be a contradiction between the goals of training an agent with high task-speciﬁcity and better generalization performance to new situations at the same time.
Revised: There seems to be a contradiction between the goals of training an agent with high task-speciﬁcity and better generalization performance to new situations at the same time.

Original: The key to reconcile these two goals lies in the usage of a small set of examples.
Revised: The key to reconcile these two goals lies in the usage of a small set of examples.

Original: There has been a paradigm shift from training static models deﬁned with parameters to ones deﬁned together with a support set (Vinyals et al., 2016;
Revised: There has been a paradigm shift from training static models deﬁned with parameters to ones deﬁned together with a support set (Vinyals et al., 2016;

Original: Snell et al., 2017), proved to be very effective in few-shot classiﬁcation.
Revised: Snell et al., 2017), proved to be very effective in few-shot classiﬁcation.

Original: Besides the effort of  meta learning implicitly adjustable models, ﬁne-tuning on a pretrained model (Oquab et al., 2014; Finn et al., 2017; Julian et al., 2020) has also been widely used in transferring knowledge from data-abundant to data-scarce tasks.
Revised: Besides the efforts of employing meta learning to adjust models, ﬁne-tuning on a pretrained model (Oquab et al., 2014; Finn et al., 2017; Julian et al., 2020) has also been widely used in transferring knowledge from data-abundant to data-scarce tasks.

Original: When reward signal is not available, Hansen et al.
Revised: When reward signal is not available, Hansen et al.

Original: (2021) proposed a policy adaptation approach, in which the intermediate representation is ﬁne-tuned via optimizing a self-supervised auxiliary loss while the controller is kept ﬁxed.
Revised: (2021) proposed a policy adaptation approach, in which the intermediate representation is ﬁne-tuned via optimizing a self-supervised auxiliary loss while the controller is kept ﬁxed.

Original: Our work shares the same motivation of test-time training (Sun et al., 2020), but we focus instead on the settings where the controller needs to be adapted or even repurposed for new tasks.
Revised: Our work shares the same motivation of test-time training (Sun et al., 2020), but we focus instead on the settings where the controller needs to be adapted or even repurposed for new tasks.

Original:  M ETHOD 3. P ROBLEM F ORMULATION
Revised:  L EARNING T RANSFERABLE R EWARD FOR L OCALIZING Q UERY O BJECTS 3. RL FORMULATION FOR QUERY OBJECT LOCALIZATION

Original: In query object localization, we are given a set of images I , and a small set of exemplary images E . Our goal is to ﬁnd the location  of the bounding box containing the queried object in each test image, without candidate proposal boxes.
Revised: Given a set of images I with known class labels, the task of query object localization is to ﬁnd the location b of the bounding box most relevant to the query object, speciﬁed in a small set of exemplary images E .

Original: Following the Markov Decision Process (MDP) framework for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016), we consider raw pixels in each image
Revised: Following the Markov Decision Process (MDP) framework for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016), this task can be formulated as a RL problem.

Original: I i as the Environment .
Revised: In the training phase, a localization agent learns to take actions of moving the bounding box to maximize a reward function reﬂecting localization accuracy. In the test phase, the agent is informed by a small test exemplary set about a new task objective.

Original: we deﬁne Action set with discrete actions facilitating a top-down search, including ﬁve scaling, eight translation transformations as in (Jie et al., 2016), plus one stay action.
Revised: Figure 1 shows an overview of the query object localization problem and the RL formulation, with components listed as follows.

Original: Starting from the whole image pixels as input, the agent is trained to select the actions to transform current box at each step, by maximizing the total discounted Reward .
Revised: • Environment: Raw pixels of a single image I i , without candidate proposal boxes Action

Original: We use policy gradient with recurrent neural network (RNN)
Revised: Discrete actions facilitating a top-down search scaling scale towards top left right

Original: (Mnih et al., 2014) rather than DeepQNetwork with a vector of history actions and states.
Revised: left center eight translation transformations: move left right/up

Revised: - down, enlarge width/height, reduce width/heigh

Original: The agent takes pooled feature from the current box as State , while it also maintains an internal state within RNN, which encodes information from history observations. 
Revised: plus one stay action • State: Pooled features from the current box and an internal state within RNN that encodes information from history observations. • Reward: Improvements the agent made in localizing the query images (in different tasks).

Original: 3. L EARNING O RDINAL R EWARD S IGNAL
Revised: 3. P RETRAINING O RDINAL R EWARD S IGNAL

Revised: For training an agent to localize different objects according to queries, the reward function needs to be transferable.

Original: Existing deep RL approaches for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016) use its ground-truth object bounding box g i as the reward signal,
Revised: Existing deep RL approaches for object localization (Caicedo & Lazebnik, 2015; Jie et al., 2016) use its ground-truth object bounding box g i as the reward signal,

Original: [Equation]
Revised: [Equation]

Original: where IoU ( b t , g i ) denotes the Intersection-over-Union (IoU) between the current window b t and the corresponding ground-truth box g i 1 , and IoU ( b, g ) =
Revised: where IoU ( b t , g i ) denotes the Intersection-over-Union (IoU) between the current window b t and the corresponding ground-truth box g i 1 , and IoU ( b, g ) =

Original: area ( b ∩ g ) / area ( b ∪ g ) .
Revised: area ( b ∩ g ) / area ( b ∪ g ) .

Original: Similar to the bounding box regression approaches which learns a mapping φ : I (cid:55)→ g , the image and box must be paired.
Revised: Similar to the bounding box regression approaches which learns a mapping φ : I (cid:55)→ g , the image and box must be paired.

Original: However, annotated image-box pairs ( I, g ) may be scarce in both the training and testing phases.
Revised: However, annotated image-box pairs ( I, g ) may be scarce in both the training and testing phases.

Original: The reward signal  in Eq.
Revised: The reward signal deﬁned in Eq.

Original: 1 isdeﬁned instance-wise, which is not transferable across images.
Revised: 1 is instance-wise, which is not transferable across images.

Original: To address this problem, a natural idea is to deﬁne the reward signal based on learned representations between the cropped images by current window b t, and the ground-truth window g .
Revised: To address this problem, a natural idea is to deﬁne the reward signal based on learned representations of the cropped images by current window b t and the ground-truth window g .

Original: Given their M - dimensional representations b t and g produced by an embedding function f : R D (cid:55)→ R M from D - dimensional image feature vectors, a distance function d : R M × R M (cid:55)→ [0 , + ∞ ) return the embedding distance d ( b t , g ) .
Revised: Given their Mdimensional representations b t and g produced by an embedding function f : R D (cid:55)→ R M from D - dimensional image feature vectors, a distance function d : R M × R M (cid:55)→ [0 , + ∞ ) returns the embedding distance d ( b t , g ) .

Original: However, d ( b t , g ) may not decrease monotonically as the agent approaches to the ground-truth box g .
Revised: However, an embedding distance based on off-the-shelf pretrained networks is not adequate, since it may not decrease monotonically as the agent approaches to the ground-truth box g .

Original: As a result, the embedding distance based reward signal may be less effective than Eq.1.
Revised: As a result, the embedding distance based reward signal may be less effective than Eq.

Revised: Therefore, an embedding function customized for the downstream RL-based localization task is needed.

Original: Ordinal metric learning.
Revised: Ordinal metric learning.

Original: We propose to use an ordinal embedding based reward signal.
Revised: We propose to use an ordinal embedding based reward signal.

Original: For any two perturbed boxes b j , b k from ground truth g , embeddings b j , b k , g are learned, such that the relative preferences between any pair of boxes are preserved in the Euclidean space,
Revised: For any two perturbed boxes b j , b k from ground truth g , embeddings b j , b k , g are learned, such that the relative preferences between any pair of boxes are preserved in the Euclidean space,

Original: [Equation]
Revised: [Equation]

Original: where ρ j and ρ k denote the preference (derived from either IoU to ground-truth box or ordinal feedbacks from user), and C is an ordinal constraint set constructed by sampling box pairs around g .
Revised: where ρ j and ρ k denote the preference (derived from either IoU to ground-truth box or ordinal feedbacks from user), and C is an ordinal constraint set constructed by sampling box pairs around g .

Original: This problem is originally posed as non-metric multidimensional scaling (Agarwal et al., 2007;
Revised: This problem is originally posed as non-metric multidimensional scaling (Agarwal et al., 2007;

Original: Jamieson & Nowak, 2011).
Revised: Jamieson & Nowak, 2011).

Original: Although we apply a very simple pairwise-based approach, there exist other extensions such as the listwise-based (Cao et al., 2007), quadruplet-based (Terada & Luxburg, 2014) and landmark-based (Anderton & Aslam, 2019; Ghosh et al., 2019). Data augmentation.
Revised: Although we apply a very simple pairwise-based approach, there exist other extensions such as the listwise-based (Cao et al., 2007), quadruplet-based (Terada & Luxburg, 2014) and landmark-based (Anderton & Aslam, 2019; Ghosh et al., 2019) approaches.

Original: We assume that during the training time, the exemplary set E train contains both image I and box g .

Original: We adopt a tailored data augmentation scheme - box perturbation . We have found that using IoU-based partition scheme is more effective than random sampling (more details in 4.1). This can be viewed as a procedure to enhance the robustness of neural network (Szegedy et al., 2014) against box perturbations, and protect the special purpose of its usage on distinguishing reward increases or decreases. It also makes the downstream task of policy network training more efﬁcient.
Revised: Loss function.

Original: 3. M ODEL ARCHITECTURE

Original: In this paper, we deﬁne  ρ as the IoU of box b to the ground-truth box g , i.e., ρ = IoU ( b, g ) .
Revised: In this paper, we deﬁne preference ρ as the IoU of box b to the ground-truth box g , i.e., ρ = IoU ( b, g ) .

Original: We choose to optimize the triplet loss (Hermans et al., 2017) for learning the desired embedding,
Revised: We choose to optimize a triplet loss (Hermans et al., 2017) for learning the desired embeddings,

Original: [Equation]
Revised: [Equation]

Original: anchor, ordinal pairs
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where a is the “anchor" embedding. p , n are the “positive" and “negative" embeddings  with larger and smaller IoUs with ground truth box g , respectively.
Revised: where a is the “anchor" embedding that will be discussed in detail later, p , n are the “positive" and “negative" embeddings for boxes with larger and smaller IoUs with ground truth box g , respectively, m is a margin hyperparameter.

Original: We learn an embedding space consistent to the local ordinal constraintsspeciﬁed on the image pairs obtained via data augmentation.
Revised: We learn an embedding space consistent with local ordinal constraints on positive and negative data obtained via data augmentation such as box perturbation.

Revised: Box Perturbation Assuming the training exemplary set E train contains both image I and box g , we adopt a tailored data augmentation scheme - box perturbation , in which a pair of boxes are randomly sampled from I with different IoUs to g .

Revised: We compare the efﬁciency of sampling schemes on generating augmented bounding box pairs, (a) Random sampling , where the pair of boxes are generated completely randomly, and (b) Group-based sampling , where dense boxes with variant scales are ﬁrst generated, and then divided into M = 10 groups according to the IoU with groundtruth box.

Revised: Each group has an IoU interval of 1 /M .

Original: The  choice of anchor.
Revised: The sampling is ﬁrst done on the group level, i.e., 2 out of M , then sample one box from each group. Thus, the sampled boxes are likely to cover more diverse cases comparing to random sampling. We have found that using IoU-based partition scheme is more effective than random sampling. The choice of anchor.

Original: The anchor embedding g in Eq.
Revised: The anchor embedding a in Eq.

Original: 3 is not restrictive to the cropped image from the same image.
Revised: 3 is not restricted to the cropped image from the same image.

Original: For example, it could be replaced by the prototype embedding (Snell et al., 2017) of the exemplary set E , c
Revised: For example, it could be replaced by the prototype embedding (Snell et al., 2017) of the exemplary set E , c

Original: = 1 / |E| (cid:80) i ∈E g i , where g i is the embedding of the cropped image
Revised: = 1 / |E| (cid:80) i ∈E g i , where g i is the embedding of the cropped image

Original: I i by ground-truth box g i .
Revised: I i by ground-truth box g i .

Original: If images from multiple class are available, the prototype can be further made to be class-dependent, or clustering-based.
Revised: If images from multiple classes are available, the prototype can be further made to be class-dependent, or clustering-based.

Revised: A prototype as the anchor enables test-time policy adaptation without image-box pairs in the exemplary set E test .

Revised: In comparison, supervised methods such as faster-RCNN would require image-box pairs for ﬁne-tuning on new tasks.

Original: We ﬁnd that using prototype-based embedding as the anchor leads to better generalization performance than using g in some experiments.
Revised: We ﬁnd that using prototype-based embedding as the anchor leads to better generalization performance than using g in some experiments.

Revised: It is also interesting to explore other treatment on the exemplary set, such as permutation-invariant architectures (Zaheer et al., 2017; Ilse et al., 2018; Lee et al., 2019).

Revised: 3. L OCALIZATION A GENT T RAINING

Revised: As discussed in Section 3.2, we use the ordinal embedding rather than the bounding box coordinates to quantify the improvement that an agent makes, and the continuous-valued reward for the agent moving from state s (cid:48) to s takes the following form,

Original: Moreover, even if the exemplary set during the test time only contains the  cropped object, test-time policy adaptation is still feasible without image-box pairs.
Revised: [Equation] [Equation] ) where c is the protoptype embedding, b t is the embedding of the cropped region at step t . We expect positive reward if the embedding distance between cropped image b t and query image a is decreased.

Original: State representation.
Revised: State representation.

Original: Note that a good representation for deﬁning reward may not necessarily be a good state representation at the same time - it may not contain enough information guiding the agent taking the right actions.
Revised: Note that a good representation for deﬁning reward may not necessarily be a good state representation at the same time - it may not contain enough information guiding the agent taking the right actions.

Original: Chen et al.
Revised: Chen et al.

Original: (2020) suggests that adding a projection head between the representation and the contrastive loss substantially improves the quality of the learned representation.
Revised: (2020) suggests that adding a projection head between the representation and the contrastive loss substantially improves the quality of the learned representation.

Original: We ﬁnd  the use of projection head is crucial in balancing the two objectives in our task.
Revised: We ﬁnd that the use of projection head is crucial in balancing the two objectives in our task.

Original: The network architecture is shown in Fig. 2, in which an MLP projection headh is attached after an (Region of Interest) RoI encoder e .
Revised: The network architecture is shown in Figure 2(a), in which an MLP projection head is attached after an (Region of Interest) RoI encoder.

Original: According to the given image and RoI, the RoI encoder extracts RoI feature s that will be used as the state representation for localization.

Original: The projection head learns ordinal embedding b for computing reward.

Original: The ROI alignment module handles boxes of different size.
Revised: The ROIAlign module handles boxes of different sizes.

Original: Under a joint loss function L embed = L rec + λ ·L trip , the state representation s can indirectly beneﬁt from the ordinal supervision on b , while it still must render satisfactory image reconstruction results. Beside the autoencoder scheme, the RoI encoder can use pre-trained network as well. 3. T RAINING L OCALIZATION A GENT AND A DAPTATION
Revised: Policy network.

Original: As discussed in Section 3.2, we use the ordinal embedding rather than the bounding box coordinates to compute the improvement that agent makes, and the reward for an agent moving from state s (cid:48) to s takes the following form, [Equation]
Revised: Starting from a whole image as input, the agent is trained to select actions to transform the current box at each step, maximizing the total discounted reward. We use policy gradient based on a recurrent neural network (RNN) (Mnih et al., 2014) with a vector of history actions and states.

Original: where c is the protoptype embedding, b t is the embedding of the cropped region at step t . We expect positive reward if distance between embedded cropped image b t and embedded query image a is decreased.
Revised: More detailed architecture can be found in Figure 2(b) and Appendix Figure 7.

Original: For all experiments, we apply policy gradient (REINFORCE)
Revised: For all experiments, we apply policy gradient (REINFORCE)

Original: (Williams, 1992) with entropy loss  to encourage exploration.
Revised: (Williams, 1992) with entropy loss regularization to encourage exploration.

Original: The full algorithm is outlined in Algorithm 1.
Revised: The main algorithm is outlined in Algorithm 1.

Revised: Require: initial policy π θ , batch size N , pretrained RoI encoder e and projection head h , learning rate α Output : policy network π θ for sampled minibatch { x k } Nk =1 do for k=1,...

Revised: N do

Revised: Construct an exemplary set E batch g j ← h ( e ( x k , g j )) // compute the ordinal embedding in Figure 2(a)

Revised: c

Revised: k ← 1 |E batch | (cid:80)

Original: Test-time adaptation.
Revised: j ∈E batch g j // compute the prototype vector τ k [Table] [Table] [Table]

Revised: [Table]

Revised: by

Revised: [Table]

Revised: // sample trajectory b

Revised: [Table]

Revised: [Table]

Revised: R

Revised: [Table]

Revised: // compute reward in equationend

Original:  During test-time, the agent has the option of further updating the policy network using the received reward from Eq. 4.
Revised: [Table] policy update end 3. T EST - TIME A DAPTATION During test time„ the agent has the option of further updating the policy network using the received reward from Eq. 4.

Original: To match test conditions, the training batch is split into two group and c is computed on a small subset that does not overlap with the training images to localize, while during test adaptation, c becomes the prototype of the test exemplary set E test .
Revised: To match test conditions, the training batch is split into two groups, and c is computed on a small subset that does not overlap with the training images to localize;During test adaptation, c becomes the prototype of the test exemplary set E test .

Original: Different from E train , only cropped images are needed in E test .
Revised: Different from E train , only cropped images are needed in E test as shown in Figure 1.

Original: Figure 3 shows an illustrative example of query object localization with task adaptation.
Revised: Figure 3 shows an illustrative example of query object localization with task adaptation.

Original: Given two common objects coexisting in the image environment, the agent is trained to pick up one from the two during training, while quickly adapt to pick another one during testing.
Revised: Given two common objects coexisting in the image environment, the agent is trained to pick up one from the two during training, while quickly adapt to pick another one during testing.

Original: Different from unsupervised co-localization approaches (Wei et al., 2017) relying on low-level images clues to distinguish foreground and background,  learning objectives can be speciﬁed explicitly by switching the train/test exemplary sets.
Revised: Different from unsupervised co-localization approaches (Wei et al., 2017) relying on low-level images clues to distinguish foreground and background, our learning objectives can be speciﬁed explicitly by switching the train/test exemplary sets.

Original: With test-time adaptation, the agent can handle even partially conﬂict learning objectives.
Revised: With test-time adaptation, our agent can handle even partially conﬂicting learning objectives.

Original: Require: initial policy π θ , batch size N , pretrained e and h , learning rate α Output : policy controller π θ for sampled minibatch { x k } Nk =1 do for k=1,...

Original: N do

Original: Construct an exemplary set E batch g j ← h ( e ( x k , g j )) // compute the ordinal embedding c k ← 1 |E batch | (cid:80) j ∈E batch g j // compute the prototype vector τ k ← { s 1 , a 1 , ...s T , a T } by running policy π θ // sample trajectory b t ← take action a t , and compute embedding h ( e ( x k , b t )) for current window b t

Original: R tk ← (cid:107) b t − 1 − c k (cid:107) − (cid:107) b t − c k (cid:107) //

Original: compute reward end θ ← θ + α ∇ θ (cid:80) Nk

Original: =1 log π θ (cid:16)(cid:80)

Original: Tt (cid:48)

Original: =

Original: t γ t (cid:48) − t R tk (cid:17) // policy update end

Original: [Figure]

Original: D ISCUSSIONS

Original: Our approach is based on the feature similarity with query images, which departs from previous bounding-box regression and RL approaches based on objectiveness.

Original: General supervised training methods are usually class-agnostic and require exposure to a large number of training classes, boximage pairs, and foreground and background variations in order to generalize well.

Original: In contrast, we allow specialized agents to be trained, with the ability of adapt to changes during the test-time.

Original: Our approach includes three phases: ordinal representation learning, RL agent training, and test-time adaptation.

Original: Compared to general supervised training or ﬁne-tuning methods, our approach is able to ﬂexibly make use of various types of data in these phases (summarized in Table 1).

Original: Our agent can both take human feedback in terms of the exemplary set and perform test-time policy adaptation using unlabeled test data.

Original: [Table]

Original: The transferability of our reward signal from training to testing crucially relies on the generalization ability of the learned ordinal representation.

Original: If the ordinal preference does not hold in the test domain, the proposed test-time policy adaptation scheme will not work.

Original: By adapting the representation with self-supervised objectives (Hansen et al., 2021), this issue might be remedied.

Original: We left it as future work.

Original: Although our approach does not directly handle the special cases of multiple queried objects or no queried object within the image environment, it can be easily modiﬁed to accomplish these tasks.

Original:  E XPERIMENT
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we evaluate the generalization ability of the ordinal embedding as well as the performance of the trained localization agent on both source and target domain.
Revised: In this section, we evaluate the generalization ability of the ordinal embedding as well as the performance of the trained localization agent on both source and target domains.

Original: The embedding-based reward not only improves the RL training, but also enables test-time adaptation of the learned policy.
Revised: The embedding-based reward not only improves the RL training, but also enables test-time adaptation of the learned policy.

Original: Experimental results demonstrate the effectiveness of our approach, with empirical comparisons to generic RL object localization, co-localization, and few-shot object detection baselines.
Revised: Experimental results demonstrate the effectiveness of our approach, with empirical comparisons to generic RL-based object localization, co-localization and few-shot object detection baselines.

Original: Evaluation protocol.
Revised: Evaluation protocol.

Original: To evaluate  learned ordinal embedding, we use OrdAcc deﬁned as the percentage of images within which the preference between a pair of perturbed boxes is correctly predicted.
Revised: To evaluate the learned ordinal embedding, we use OrdAcc deﬁned as the percentage of images within which the preference between a pair of perturbed boxes is correctly predicted.

Original: We use the Correct Localization ( CorLoc ) (Deselaers et al., 2012)) metric, which is deﬁned as the percentage of images correctly localized according to the criterion IoU ( b p , g ) ≥ 0 . 5 , where b p is the predicted box and g is the ground-truth box.
Revised: To evaluate object localization performance, we use the Correct Localization ( CorLoc ) (Deselaers et al., 2012) metric, which is deﬁned as the percentage of images correctly localized according to the criterion IoU ( b p , g ) ≥ 0 . 5 , where b p is the predicted box and g is the ground-truth box.

Original: Mean and standard deviation± ( displayed as subscript) are reported from 10 independent runs (unless mentioned elsewise).
Revised: Mean and standard deviation ( displayed as subscript) are reported from 10 independent runs.

Original: Implementation.
Revised: Implementation.

Original: We evaluate our approach on distorted versions of the MNIST handwriting dataset, the CUB-200-2011 birdsdataset (Wah et al., 2011), and  COCOdataset (Lin et al., 2014). For
Revised: We evaluate our approach on distorted versions of the MNIST handwriting, the CUB-200-2011 birds (Wah et al., 2011), and the COCO (Lin et al., 2014) dataset.

Original:  MNIST, we use three convolutional layers with ReLU activation after each layer as image encoder, while the same but mirrored structure as decoder to learn an autoencoder, and then attach RoI align layer following two fully connected ( fc ) layers as projection head for ordinal reward learning.
Revised: For MNIST, we use three convolutional layers with ReLU activation after each layer as image encoder, while the same but mirrored structure as decoder to learn an autoencoder, and then attach RoI alignment layer followed by two fully connected ( fc ) layers as projection head for ordinal reward learning.

Original: For the
Revised: For the

Original: CUB and the COCO datasets, we adopt layers before conv5_3 of VGG16 pretrained on ImageNet for encoder.
Revised: CUB and the COCO datasets, we adopt layers before conv5_3 of VGG-16 pretrained on ImageNet as encoder.

Original: The projection head uses the same structure as before but with more units for each fc layer.
Revised: The projection head uses the same structure as before but with more units for each fc layer.

Original: All of our models were trained with  Adamas optimizer (Kingma & Ba, 2015).
Revised: All of our models were trained with the Adam optimizer (Kingma & Ba, 2015).

Revised: We set margin m = 60 in all the experiments heuristically.

Original: All the models take less than one hour to ﬁnish training, implemented on PyTorch on a single NVIDIA A100 GPU.
Revised: All the models take less than one hour to ﬁnish training, implemented on PyTorch on a single NVIDIA A100 GPU.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDIES

Original: We ﬁrst study the effectiveness of ordinal embedding and  reward as well as a few other design choices.
Revised: We ﬁrst study the effectiveness of ordinal embedding and learned reward as well as a few other design choices.

Original: We measure the generalization on test data without policy adaptation.

Original: Setting and main results.
Revised: Setting and main results.

Original: We analyze the effectiveness of using ordinal embedding in terms of representation and reward ﬁrst on synthetic datasets of corrupted MNIST digits under cluttered, random patch, Gaussian, or impulse noises (see Figure 8 in Appendix for examples).
Revised: We analyze the effectiveness of using ordinal embedding in terms of representation and reward ﬁrst on synthetic datasets of corrupted MNIST digits under cluttered, random patch, Gaussian, or impulse noises (see Figure 8 in Appendix for examples).

Original: We want to ﬁnd the answers to two main questions listed below:
Revised: We want to ﬁnd the answers to two main questions listed below:

Original: Is the embedding distance based reward signal as effective as the IoU counterpart?
Revised: Is the embedding distance based reward signal as effective as the IoU counterpart?

Original: Does the ordinal triplet loss beneﬁt the state representation learning?

Original: Accordingly, we consider baselines in which embeddings are trained with only autoencoder or jointly with the ordinal projection head.
Revised: Accordingly, we consider baselines in which embeddings are trained with only autoencoder or jointly with the ordinal projection head.

Original: Besides, we also comparethe our embedding distance based reward against the IoU based reward used in Caicedo & Lazebnik (2015).
Revised: Besides, we also compare our embedding distance based reward against the IoU based reward used in Caicedo & Lazebnik (2015).

Original: Without loss of generality, the agent is trained on images of digit 4, and tested on images from all rest classes in testing.
Revised: Without loss of generality, the agent is trained on images of digit 4, and tested on images from all rest classes.

Original: In particular, we are interested in evaluating the sample efﬁciency of different approaches.
Revised: In particular, we are interested in evaluating the sample efﬁciency of different approaches.

Original: The results under different train set size are shown in Fig.
Revised: The results under different training set sizes are shown in Figure 4.

Original: To distract the localization agent, cluttered noises with random 6 × 6 crops of digits are added to the background.
Revised: To distract the localization agent, cluttered noises with random 6 × 6 crops of digits are added to the background.

Original: All comparisons are based on the policy gradient (PG) method.
Revised: All comparisons are based on the policy gradient (PG) method.

Original: With ordinal embedding presents in both the representation and reward (“AE+Ord+Embed"), the model performance is consistently better than other settings, especially when train set size is small.
Revised: With ordinal embedding present in both the representation and reward (“AE+Ord+Embed"), our model performance is consistently better than other settings, especially when the training set size is small.

Original: The beneﬁt of  triplet lossback propagation is evidenced by the comparison between “AE+Ord+IoU" and “AE+IoU".
Revised: The beneﬁt of a triplet loss is demonstrated by the comparison between “AE+Ord+IoU" and “AE+IoU".

Original: Intuitively, the similarity to the queried object is also a more natural goal than objectiveness deﬁned by bounding boxes IoU, which is used in both generic object detection or localization and previous RL-based approaches.
Revised: Intuitively, the similarity to a queried object is also a more natural goal than objectives deﬁned by bounding boxes IoU, which is used in both generic object detection or localization and previous RL-based approaches.

Original: Signed reward or not.
Revised: Signed reward or not.

Original: Another intriguing question is that other than beneﬁting state representation learning, whether the embedding-based reward itself can bring any additional beneﬁts than the IoU-based reward.
Revised: Another intriguing question is that other than beneﬁting state representation learning, whether the embedding-based reward itself can bring any additional beneﬁts than the IoU- based reward.

Original: From Figure 4, it appears to be a large gap between “AE+Ord+Embed" and ‘AE+Ord+IoU", especially when the train set size is small.
Revised: From Figure 4, there is a large gap between “AE+Ord+Embed" and “AE+Ord+IoU", especially when the training set size is small.

Original: This is counter-intuitive as the ordinal reward is trained with supervision fromthe IoU inthe embedding space, supposably it shall not bring any additional information.
Revised: This is counter-intuitive as the ordinal reward is trained with supervision from IoU in embedding space, supposably it shall not bring any additional information.

Original: To further analyze this phenomenon, we take the sign operation off in 
Revised: To further analyze this phenomenon, we take the sign operation off in Eq.

Original: Eq. 1 and retrain the models.
Revised: and retrain the agent.

Original: In an experiment with 50 training images, with sign operation, the localization accuracy increase from 90.92% to 94.36% on test digit 4 , and from 82.43% to 88.64% on other digits test set.
Revised: In an experiment with 50 training images, without sign operation, the localization accuracy increases from 90.92% to 94.36% on test digit 4 , and from 82.43% to 88.64% on other digit test set.

Original: This results show that unsigned reward works better in our case.
Revised: Perhaps, this improvement can be attributed to more informative feedback provided by the continuous-valued, unsigned reward.

Original: On policy vs. off policy.
Revised: On policy vs. off policy.

Original: Many deep RL approaches are in favor to use deep Q-Network (DQN) to train the agent.
Revised: Many deep RL approaches are in favor of using deep Q-Network (DQN) to train an agent.

Original: Different from Caicedo & Lazebnik (2015) and Jie et al.
Revised: Different from Caicedo & Lazebnik (2015) and Jie et al.

Original: [Table] we apply PG to optimize it.
Revised: , we apply Policy Gradient (PG) to optimize it.

Original: Besides, we adopt a top-down search strategy through  RNN, while they used a vector of history actions to encode memory in these works.
Revised: Besides, we adopt a top-down search strategy through a RNN, while they used a vector of history actions to encode memory.

Original: We evaluate these design choices with four baselines, all using "AE+Ord+Embd" for state and reward.
Revised: We evaluate these design choices with four baselines, with "AE+IoU" setting, and trained on the same 600 sampled cluttered digit 4 images.

Original: As Table 2 shows, the agent achieves the best performance with "PG+RNN".
Revised: As Table 1 shows, the agent achieves the best performance with "PG+RNN".

Original: Perhaps surprisingly, we ﬁnd that empirically, with history action vectors the accuracy becomes worse when the agent is trained by DQN
Revised: We ﬁnd that empirically, with history action vectors the accuracy becomes worse when the agent is trained by DQN.

Original: [Table] [Table] Sampling scheme. We compare the efﬁciency of sampling schemes on generating augmented bounding box pairs, (a) Random sampling , where the pair of boxes are generated completely randomly, and (b) Group-based sampling , where dense boxes with variant scales are ﬁrst generated, and then divided into 10 groups according to the IoU with ground-truth box.
Revised: with/without history action vector) vs. PG (with/without RNN)

Original: Each group has an IoU interval of 0 . 1 . The sampling is ﬁrst done on the group level, i.e., 2 out of 10 , then sample one box from each group. Thus, the sampled boxes are likely to cover more diverse cases comparing to random sampling. Using the same number of pairs, the resulting OrdAcc for the two schemes is 90.12% (random sampling), 92.97% (group sampling), respectively.
Revised: [Table]

Original: 4. P OLICY A DAPTATION DURING T EST -T IME
Revised: P OLICY A DAPTATION DURING T EST -T IME

Original: We further demonstrate the performance improvement with test-time policy adaptation.
Revised: We further demonstrate the performance improvement with test-time policy adaptation.

Original: Through all the experiments, we assumea single source domainthat contains abundant data annotations, andthe target domain annotation are only available in an exemplary set of size 5 .
Revised: Through all the experiments, we assume source domain contains abundant data annotations, and target domain annotations are only available in an exemplary set of size 5 .

Original: We compared our policy adaptation scheme with a standard ﬁne-tuning scheme on the pre-trained policy network.
Revised: We compared our policy adaptation scheme with a standard ﬁne-tuning scheme on the pre-trained policy network.

Original: Results on the corrupted MNIST dataset.
Revised: Results on the corrupted MNIST dataset.

Original: We use  “digit 4 under random patch background noises"as the base class to train the ordinal embedding and the localization agent.
Revised: We use 50 “digit 4 under random patch background noises" to train the ordinal embedding and the localization agent.

Original: During test-time, either the class of the foreground object or the background changes.
Revised: During test time, either the class of the foreground object or the background changes.

Original: The results on policy adaptation to new digits are shown in Table 3.
Revised: The results on policy adaptation to new digits are shown in Table 2.

Original: Row 1 illustrates the transferability of the ordinal embedding reward, trained prototype embedding of a subgroup without the training instance, and evaluated using instances embedding from the same test image (“Ord Ins").
Revised: Row 1 illustrates the transferability of the ordinal embedding reward, trained prototype embedding of a subgroup without the training instance, and evaluated using instance embedding from the same test image (“Ord Ins").

Original: Rows 2 to 4 list the resulting localization accuracy after direct generalization (“Before"), ﬁne-tuning on the exemplary set (“Finetune"), and adaptation using all test images (“Adapt"), respectively.
Revised: Rows 2 to 4 list the resulting localization accuracy after direct generalization (“Before"), ﬁne-tuning on the exemplary set (“Finetune"), and adaptation using all test images (“Adapt"), respectively.

Original: Our policy adaptation approach produces a substantial improvement over directly generalization, while ﬁne-tuning approach experiences overﬁtting on the limited exemplary set.
Revised: Our policy adaptation approach produces a substantial improvement over direct generalization, while ﬁne-tuning approach experiences overﬁtting on the limited exemplary set.

Revised: For the background adaptation experiment, we train on 50 digit-3 images and test on digit-2 images.

Original: The localization accuracy on boththe source background and new backgrounds environment are shown in Table 4, signiﬁcant improvements are achieved using our policy adaptation scheme
Revised: The localization accuracy on both source background and new backgrounds environment are shown in Table 3, signiﬁcant improvements are achieved using our policy adaptation scheme.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Results on the CUB dataset.
Revised: Results on the CUB dataset.

Original: We also evaluated the policy adaptation performance on the CUB dataset.
Revised: We also evaluate the policy adaptation performance on the CUB dataset.

Original: The localization agent is trained on 15 species from the “Warbler" class, and adapted to different classes of “Warbler" ( 5 new species), “Wren", “Sparrow", “Oriole", “Kingﬁsher", “Vireo", and “Gull".
Revised: The localization agent is trained on 15 species from the “Warbler" class, and adapted to different classes of “Warbler" ( 5 new species), “Wren", “Sparrow", “Oriole", “Kingﬁsher", “Vireo", and “Gull".

Original: Each test class contains a single bird species type, so we also implemented deep descriptor transforming (DDT) (Wei et al., 2017), a deep learning based co-localization approach, and added to the comparison.
Revised: Each test class contains a single bird class. We also implement deep descriptor transforming (DDT) (Wei et al., 2017), a deep learning based co-localization approach, and add it to the comparison.

Original: [Table]
Revised: [Table]

Original: 4. T RANSFER FROM SPECIFIC TO SPECIFIC
Revised: C OMPARISON TO F EW - SHOT O BJECT D ETECTORS

Original: Given a target domain  one would like to deploy models to, a natural question arises on whether one should collect labeled data from an abundant number of source domains, or  one speciﬁc class with potential lower data collection cost.
Revised: Given a target domain that one would like to deploy models to, a natural question is whether one should collect labeled data from an abundant number of source domains or from one speciﬁc class with potential lower data collection cost.

Original: On the COCO dataset, we selected a subset of images from ﬁve classes:cow, cat,  dog, horse, zebra , and evaluated the transfer learning performance to the same target domain, from every other source domains.
Revised: On the COCO dataset, we train on one of the ﬁve classes: cat, cow, dog, horse, zebra , then adapt to another four classes.

Original: The results are shown in Fig.
Revised: The results are shown in Figure 5.

Original: The models perform best using  embedding reward with adaptation (see “ours-adapt").
Revised: The models perform best using ordinal embedding reward with adaptation (see “ours-adapt").

Revised: It shows that, although an agent is trained in a highly specialized way on only one-source tasks, it can still be ﬂexibly generalized to different target tasks.

Original: One interesting observation is that it is easy to transfer from other four classes to zebra, but not vice versa.
Revised: One interesting observation is that, it is easy to transfer from other four classes to zebra, but not vice versa.

Original: A possible explanation might be  the embedding net is biased on texture, while texture information is less adaptable than shape.
Revised: A possible explanation might be that, the embedding net is biased on textures, and texture information is less adaptable than shape information.

Original: Few-shot object detection (Kang et al., 2019; Yan et al., 2019; Wang et al., 2019; 2020)
Revised: Few-shot object detection (Kang et al., 2019; Yan et al., 2019; Wang et al., 2019; 2020)

Original: methods are similar to our work in adapting models to a new test case with the help of limited annotations, but  have a different requirement on the number of training classes.
Revised: methods are similar to our work in adapting models to a new test case with the help of limited annotations, but they have a different requirement on the number of training classes.

Original: These methods usually require multiple classes in both stages of training and ﬁne-tuning.
Revised: Being different in the generalization mechanism, they usually require multiple classes in both stages of training and ﬁne-tuning.

Original: On the COCO dataset, we compared the performance of our methods with TFA (Wang et al., 2020) andMeta-RCNN Yan et al.
Revised: On the COCO dataset, we compared the performance of our methods with TFA (Wang et al., 2020), and

Original: (2019), on the same one-way 5-shot test setting.
Revised: Meta-RCNN (Yan et al., 2019), on the same one-way 5-shot test setting.

Revised: TFA adopts two-stage ﬁnetuning of object detector, while Meta-RCNN incorporates additional meta-learner to acquire class-level meta knowledge for generalization to novel classes.

Original: Few-shot baselines are trained on 60 base classes and ﬁne-tuned on all 80 classes, while our model is trained from one single class of the 5 classes randomly selected from the base classes set: elephant, sheep, giraffe, laptop and suitcase .
Revised: These two few-shot baselines are trained on 60 base classes and ﬁne-tuned on all 80 classes, while our model is trained from one single class of the 5 classes randomly selected from the base classes set: elephant, sheep, giraffe, laptop and suitcase .

Original: Table 6 shows that with ordinal embedding, the agent achieves better performance even without adaptation, and  performance can be further improved after adaptation.[Figure] ours [Figure]
Revised: Table 5 shows that with ordinal embedding, our agent achieves better performance even without adaptation, and the performance can be further improved after adaptation.

Original: adapt iou

Original: [Figure]
Revised: [Figure]

Original: before iou

Original: [Figure]
Revised: [Table]

Original: adapt [Figure] [Table]  C ONCLUSION 
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We propose an ordinal representation learning based reward, for training a localization agent to search queried object of interest.
Revised: We propose an ordinal representation learning based reward, for training a localization agent to search queried objects of interest.

Original: In particular, we use a small exemplary set as a guidance signal for delivering learning objectives, which can avoid potential learning ambiguity.
Revised: In particular, we use a small exemplary set as a guidance signal for delivering learning objectives without ambiguity.

Original: Meanwhile, we use test image environments to inform the agent about the domain shifts, without requiring image-box pairs during the test-time.
Revised: Meanwhile, we use test image environments to inform the agent about the domain shifts, without requiring image-box pairs during test time.

Original: Annotation collection plays an important role in building machine learning systems.

Original: It is one task that could beneﬁt greatly from automation, especially in cost-sensitive applications.

Original: We aim to reduce the human labeling efforts, in terms of the number of annotate samples per class, number of annotate classes (one-class at the minimum), and the level of accuracy required, all under the same query-adaptation framework.

Original: Instead of jointly training the localization and classiﬁcation model, we learn box annotations from image class labels, in a similar spirit with weakly-supervised learning.
Revised: Instead of jointly training a localization and classiﬁcation model, we learn box annotations from image class labels, in a similar spirit to weakly-supervised learning.

Original: Empirically, we focus extensively on the transfer learning setting from one single data abundant training task to data-scarce (one-way few-shot) test tasks.
Revised: Empirically, we focus extensively on the transfer learning setting from one single data abundant training task to data-scarce (one-way few-shot) test tasks.

Original: Our approach is also applicable to the many-way few-shot setting where limited annotations across multiple classes are available during training, and we leave it for future study.

Original: In making the trained RL agent generalizable to new unseen test cases, one can either expose the agent to as many cases as possible during training, or make the agent more speciﬁc but adaptable upon changed scenarios.
Revised: To make a trained RL agent generalizable to new unseen test cases, one can either expose the agent to as many cases as possible during training, or make the agent more speciﬁc but adaptable upon changed scenarios.

Original: Our approach belongs to the latter, which shows that even if only one training class is accessible, the agent with transferable reward can still adapt to various cases during the test time.
Revised: Our approach belongs to the latter. The transferability of our reward signal from training to test crucially relies on the generalization ability of the learned ordinal representation. If the ordinal preference does not hold in the test domain, the proposed test-time policy adaptation scheme will not work. By adapting the representation with self-supervised objectives (Hansen et al., 2021), this issue might be remedied.

Revised: Although our work focuses on the generalization ability of a single object localizer, extensions to multiple objects can possibly be done by applying it multiple times and marking the found regions, in a similar manner as (Caicedo & Lazebnik, 2015).

Original: As an extension work, curriculum learning with a designed sequence of targets in the exemplary set may guide the agent toward solving challenging localization tasks.
Revised: Curriculum learning with a designed sequence of targets in the exemplary set may guide the agent toward solving challenging localization tasks.

Original: We leave the robustness to bias and vulnerability to adversarial attacks for future study.
Revised: We leave those extensions for future study.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Code is available at: https://anonymous.4open.science/r/Localization-by-Ordinal-Embedd ing-2C
Revised: Code is available at: https://anonymous.4open.science/r/Localization-by-Ordinal-Embedd ing-2F

Original: More implementation details and intermediate results can be found in Appendix A.2.
Revised: More implementation details and experiment results can be found in Appendix.

Original: A A PPENDIX  A. A DDITIONAL E XPERIMENTS A.1. V ISUALIZATION OF D EEP D ESCRIPTOR T RANSFORMING
Revised: A A PPENDIX : I MPLEMENTATION D ETAILS A. A LGORITHM P IPELINE

Revised: General supervised training methods are usually class-agnostic and require exposure to a large number of training classes, box-image pairs, and foreground and background variations in order to generalize well. In contrast, we allow specialized agent to be trained, with the ability of adaptation to changes during the test time. Our approach is based on the feature similarity with query images, which departs from previous bounding-box regression and RL approaches based on objectiveness.

Original: Deep Descriptor Transforming (DDT) (Wei et al., 2017) is a co-localization method.
Revised: Compared to general supervised training or ﬁne-tuning methods, our approach is able to ﬂexibly make use of various types of data in these phases. This is summarized in Table 6. [Table] Our agent can take both human feedback in terms of the exemplary set and perform test-time policy adaptation using unlabeled test data.

Original: It applies
Revised: It includes three stages: ordinal representation pre-training, RL agent training, and test-time adaptation. Details are as follows: Stage 1: Pretrain ordinal embedding for state representation and reward In this stage, we assume pairs of groundtruth bounding box and training image are available. We train the ordinal embedding by attaching a projection head after RoI encoder, as Figure 6 shows.

Revised: RoI encoder is composed of image encoder and an RoI alignment layer (He et al., 2017).

Revised: It extracts corresponding bounding box feature directly from image feature output by image encoder.

Revised: After training, all the modules are ﬁxed.

Original: PCA on deep descriptors to generate an indicator matrix, the values of which reﬂect the correlations of these deep descriptors.
Revised: The output of RoI Encoder then becomes the state for agent, and output of projection head is then used for reward computation. On Corrupted MNIST (cMNIST) dataset, We attach a decoder in order to train the encoder, but discard it during inference. The loss function is deﬁned as [Equation]

Revised: where we set λ 1 = 0 . 1 , and L reconstruct is mean square error loss to measure the reconstruction ability of the autoencoder.

Revised: On CUB and COCO dataset, we adopt the pretrained ImageNet encoder, thus no additional decoder is needed.

Revised: The triplet loss L triplet is optimized, such that the embedding of positive bounding boxes to be closer to the embedding of anchor than negative boxes.

Revised: Different from traditional deﬁnition of triplet loss, our positive and negative boxes are not ﬁxed.

Original: The bounding box of the common object is then predicted by setting a threshold to ﬁlter out negatively correlated locations.
Revised: When comparing two perturbed boxes, the positive box is always the one with larger IoU. Thus, a box with IoU=0. could be positive box if the negative box has even smaller IoU. The anchor box is not restricted to ground truth box of the image instance.

Original: However, when there are multiple common objects in  the dataset, without specifying the desired target object, these co-localization methods will fail.
Revised: For example, we use the mean vector of ordinal embeddings in this paper, computed from groundtruth boxes of images from the exemplary set. [Table] Stage 2: Localization agent training In this stage, we assume the groundtruth bounding box is available for each training image. All the models are trained and tested on the same datasets as stage 1.

Revised: We ﬁx all the layers in RoI encoder and projection head.

Revised: The adopted RNN model is shown in

Revised: Figure 7.

Revised: In each step, the agent takes the RoI encoder output e roi and the hidden state of RNN h t − 1 , concatenated as the state representation.

Revised: Then it predicts the action for next step by sampling according to the logits.

Revised: The loss is deﬁned as

Revised: [Equation]

Revised: (6) where L entropy prevents the agent from being stuck in the local minimum, π is the policy, and R is the reward deﬁned in Eq.

Revised: Stage 3: Test-time adaptation To adapt to new class, we make use of a limited number of groundtruth box cropped images in the test exemplary set.

Revised: We leverage abundant unlabelled images and this exemplary set to update the agent.

Revised: We ﬁx the RoI encoder and projection head, and only update the agent.

Revised: The process is almost the same as stage 2, except that c in reward (Eq. 4) is the prototype of the test exemplary set.

Revised: In summary, the conﬁguration of different model components and how they are updated is listed in

Revised: [Table]

Revised: [Table]

Revised: A. E XPERIMENTS D ETAILS

Revised: We list experiment details for the datasets used in Section 4 as follows.

Revised: Corrupted MNIST Dataset.

Revised: Examples of the four types of corrupted MNIST (cMNIST) images are shown in Figure 8.

Revised: In stage 1 and 2, we randomly sample 50 images of one class in original

Revised: MNIST training set, and add noise as our training set.

Revised: During testing, the models are evaluated on all the corrupted MNIST test set.

Revised: In stage 3, to adapt to a new digit, we annotate a limited number of images as exemplary set for adaptation and use all the (corrupted) MNIST training set from that digit.

Revised: We then test the agent on all (corrupted)

Revised: MNIST test set of the same digit.

Revised: There is no overlap between the training and test set used in stage 3.

Revised: CUB Dataset.

Revised: In stage 1 and 2, we train on 15 warbler classes with class id between 158 and 172.

Revised: There are 896 images in total.

Revised: Then test the models on 5 new warbler classes with class id between 178 and 182, resulting in 294 images in total.

Revised: In stage 3, the number and class ids of images for each class are presented in Table 9.

Revised: We also randomly select limited number of images as exemplary set and use all unlabled data for adaptation.

Revised: The CorLoc is calculated using all the images of this class.

Revised: [Table]

Revised: COCO Dataset.

Revised: For the results of Figure 5, we train on one of the ﬁve classes: cat, cow, dog, horse, zebra , then adapt to another four classes.

Revised: The detailed number of each class for training and testing in stage 1 and 2 is shown in Table 10.

Revised: [Table]

Revised: In stage 3, the agent is tested on new class in target domain, within which we annotate limited number of images for adaptation.

Revised: In comparison with few-shot object detection experiment, the models in stage 1 and 2 are trained using one single class of the 5 classes: elephant, sheep, giraffe, laptop and suitcase.

Revised: Then being adapted to the four classes in Table 5.

Revised: Note that the ﬁve classes are in the base classes, and the four classes are in the novel classes used in other methods.

Revised: Thus, it’s harder to transfer from one single class to another due to scarcity of training data and training class.

Revised: Tablereports the average CorLoc from the ﬁve classes to each target class.

Revised: We also provide the results of each source class in Table 11.

Revised: From this table, we can see that transferring from related classes with target class usually performs better.

Revised: For example, the CorLoc from laptop and suitcase are lower than other three animal classes, especially before adaptation.

Revised: After adaptation, the gap becomes smaller.

Original: Figure 6 shows examples when DDT is applied on corrupted MNIST with random patch background.
Revised: [Table] Selective localization. We investigate the agent’s ability in localizing the object speciﬁed by the query set, when the set of images have two common objects. We use random patched MNIST, where each image has digit 3 and digit 4.

Revised: First, the RoI encoder and projection head is trained with an additional contrastive loss to enlarge the distance between the two digits in embedding space,

Revised: [Equation]

Revised: where loss trip = loss trip 3 + loss trip 4 , learning two local ordinal structure around each class center in embedding space.

Revised: We set the margin for both triplet losses as 10, and the margin for contrastive loss as 320 heuristically.

Revised: We found that the larger the gap between the two margins, the better the performance.

Original: As most of the patches are recognized as positively correlated (highlighted in yellow) to a common object, the method fails to localize target object digit 4.
Revised: Detailed results can be found in Table 17 and its related discussion. After learning the RoI encoder and projection head, we train the agent with the reward deﬁned with Eq. 4 in Sect. 3.3, where c is the prototype embedding of the targeted digit exemplary set.

Revised: Figure 3 in Sect.

Revised: 4 shows an example where the exemplary size is 5.

Revised: B A DDITIONAL E XPERIMENTS B. A BLATION OF O RDINAL E MBEDDING AND RL

Revised: We analyze the effectiveness of ordinal embedding and RL component separately.

Revised: First, we remove

Revised: RL, and substitute it with a simple linear search approach.

Revised: Speciﬁcally, we adopt selective search (Uijlings et al., 2013) for candidate boxes generation.

Revised: The candidate boxes are ranked according to their embedding distances to the prototype, and the one with the smallest embedding distance is returned as the ﬁnal output.

Revised: We considered two backbone networks as the embedding, including

Revised: ImageNet pretrained VGG-16 and faster-RCNN VGG-16 trained locally on COCO.

Revised: We also compare both with and without the ordinal component, making it a 2 × 2 ablation study.

Revised: It can be seen from the blue and green bars from Figure 9 and Figure 10 that with ordinal structure, the ranking method performs much more better.

Revised: We found that pre-training with the proposed ordinal loss signiﬁcantly improves the rank consistency of these backbone networks.

Revised: To further demonstrate the effectiveness of ordinal embedding, we compute the Spearman’s Rho rank correlation between embedding distance to the prototype and IoU to the groundtruth.

Revised: The results are shown in Figure 11.

Revised: Here we also add CLIP (Radford et al., 2021) pretrained ViT as backbone for comparison.

Revised: The rank correlation is smaller than − 0 .

Revised: 7 on all backbones with ordinal embedding, exhibiting ordinal embedding preserves the IoU order, thus is better for the ranking purpose.

Revised: Although pretty effective, embedding distance is still not a perfect indicator of the ranking of

Revised: IoU. Thus, directly formulate the object localization problem as a search problem leads to suboptimal localization accuracy.

Revised: In contrast, being able to learn across different images, the RL localization agent shows more tolerance for the defects of the reward function.

Revised: We analyze the beneﬁts of RL component.

Revised: For both Figure and Figure 10, when using ordinal embedding, the model with RL is generally better than ranking, especially after adaptation.

Revised: Ordinal component is necessary for several possible reasons.

Revised: First, the ordinal component makes the generic pre-trained network more speciﬁc to the downstream object localization task.

Revised: Second, it helps remove the dataset bias if the pre-trained network is non-native.

Revised: Third, ordinal pretraining encourages the reward signal to reﬂect step-by-step improvements.

Revised: Encouraged by the strong baseline performance of Faster RCNN

Revised: (detailed in Table 13 and Table 14), we investigate whether it is adequate to use Faster RCNN embedding in the RL reward function, without ordinal pre-training.

Revised: Since Faster RCNN emebdding is also trained for object localization on the COCO dataset, it can better separate out the effects of the ordinal pretraining component.

Revised: Result is shown in Table 12.

Revised: Comparing to the second row of Table 15, it can be seen that without ordinal pretraining the performance degrades signiﬁcantly, where the

Revised: CorLoc is much lower even in source domain.

Revised: [Table]

Revised: We compare our framework with a strong supervised object localiztion baseline, Faster RCNN (Ren et al., 2016).

Revised: Both methods are trained in one class (foreground vs. background as the classiﬁcation labels in Faster RCNN) and adapted to a different class.

Revised: We use VGG-16 model pretrained on

Revised: ImageNet, and ﬁnetune and test on each of the ﬁve classes: cow, cat, dog, horse, zebra .

Revised: The results on source domain are shown in Table 13.

Revised: It shows that Faster RCNN can also be made into a class-speciﬁc model and it still yields superior performance on source domain.

Revised: On the target domain, we ﬁnetune Faster RCNN using only support set for each target class.

Revised: The results are shown in Table 14.

Revised: It can be seen that our method works better on new classes with test-time adaptation over the traditional ﬁne-tuning of Faster RCNN.

Revised: Note that ﬁne-tuning requires image-box pairs in the target domain while our policy adaptation approach does not.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: B. C OMPARE DIFFERENT OFF - THE - SHELF NETWORKS AS THE BACKBONE

Revised: It is interesting to study the choices of off-the-shelf pre-trained networks as the backbone, such as

Revised: CLIP or supervised embedding provided by Faster RCNN or a classiﬁcation network.

Revised: Since these networks have been exposed to large-scale dataset, it is interesting to see whether policy adaptation is still needed.

Revised: We compare different backbones on both source domain and target domain using our method.

Revised: Table 15 reports the CorLoc of training and testing on source domain.

Revised: The large-scale pretraind ViT backbone consistently performs the best, comparing to the other two VGG-16 models.

Revised: Table 16 compares the backbones on target domain with new classes.

Revised: The test-time adaptation still brings a large margin of improvement.

Revised: Interestingly, we also found that the Faster-RCNN embedding offers the best performance on the target domain before adaptation, while the ViT network trained on CLIP dataset provides the best performance after adaptation, indicating different generalization mechanisms.

Revised: They both outperform the ImageNet backbone intially considered.

Revised: [Table]

Revised: T HE EFFECTS OF M ARGIN

Revised: The margin in triplet loss is selected heuristically.

Revised: It is not sensitive except in the selective localization experiment (Figure 3), where there are two different digits in each image.

Revised: For this experiment, we trained two ordinal structures around each digit using triplet loss with margin m

Revised: [Table]

Revised: and add additional contrastive loss with margin m 2 to separate the centers of the two different digits as far as possible.

Revised: And we found out that the model works best when m 2 (cid:29) m 1 .

Revised: In our experiment, we set m 1 = 10 , m 2 = 320 .

Revised: The results of using different set of m 1 and m 2 are presented in Table 17.

Revised: [Table]

Revised: B.

Revised: [Table]

Revised: S IZE OF TRAINING SET

Revised: In this experiment

Revised: [Table]

Revised: we train on class giraffe in stage 1 and 2, then adapt to cat, cow, dog and horse.

Revised: We set the training set size as [200, 500, 700, 1146], exemplary set size as 5.

Revised: We compare our after adaptation results with TFA w/ fc (Wang et al., 2020) one-way 5-shot setting, where their model is trained on all 60 base classes, while ours is only trained on one of the base classes.

Revised: Figure 12 shows the results, in which the dotted lines are the results of TFA w/ fc.

Revised: Ours performs much more better than their method.

Revised: Except cat class, ours is better than theirs with only 200 images for training.

Revised: B. S IZE OF E XEMPLARY S ET

Revised: We also compare the effect of different size of exemplary set during training and adaptation on

Revised: CUB-warbler dataset.

Revised: During training stage, we use shufﬂe proto training strategy, and set exemplary set size as 2, 5, 15, 25.

Revised: The results without adaptation on test set are in Table 18.

Revised: Both OrdAcc and CorLoc increase with exemplary set size.

Revised: For adaptation stage, the range of exemplary set size is from 2 to 200.

Revised: And the results are in Table 19.

Revised: The test performance does not increase much with the exemplary set size.

Revised: One possible explanation is that the data points in embedding space are compact, thus prototype doesn’t change much when increasing exemplary set size.

Revised: We will analyze the inﬂuence of multiple protoptypes per class in future experiments

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: B. P ROTOTYPE S ELECTION

Revised: We further evaluate the choice of anchor in the triplet loss for both the pretraining of state represtation and the ordinal reward for the training of agent.

Revised: We study ( i ) whether ordinal embedding can be trained in reference to an anchor from a different image instances, ( ii ) whether it is advantageous to use the proptotype embedding of an exemplary set, rather than instance embeddings

Revised: [Table]

Revised: and ( iii ) whether mimicking the test condition in training yields any improvement.

Revised: We use the CUB-Warbler dataset with more foreground background variations than the corrupted

Revised: MNIST dataset.

Revised: The train and test set contains 15 and 5 disjoint ﬁne-grained classes respectively, resulting 896 images for training (viewed as a single class) and 294 for testing.

Revised: Table 20 shows the OrdAcc and CorLoc in four settings.

Revised: “Self", use the embedding from images cropped by the ground-truth box from the same instance; “Shufﬂe self", use the ground-truth box cropped image emebedding from a different instance; Similarly, “Proto", use the prototype of a subgroup containing the training instance within the same batch; “Shufﬂe proto (SP)", use the prototype of a subgroup

Revised: [Table]

Revised: from a different batch without the training instance.

Revised: Results suggest that this training strategy brings compactness to the training set, constructing an ordinal structure around the cluster.

Revised: For “Shufﬂe proto", while the OrdAcc is lower than others, its CorLoc is the best with large margin.

Revised: Matching the condition between training and testing indeed improves generalization to new classes on this dataset.

Original: A.1. A NNOTATION REFINEMENT
Revised: B. A NNOTATION REFINEMENT

Revised: Annotation collection plays an important role in building machine learning systems.

Revised: It is one task that could beneﬁt greatly from automation, especially in cost-sensitive applications.

Revised: The human labeling efforts can be reduced, not only in terms of the number of annotate samples per class, number of annotate classes (one-class at the minimum), but also the level of accuracy required.

Original: We explore the direction of iterative reﬁnement of annotation quality with minimal human guidance.
Revised: We explore the direction of iterative reﬁnement of annotation quality with minimal human guidance.

Original: Given an agent trained extensively with inexact human annotations, can we adapt it to reﬁne the annotation with a few well-annotated examples?
Revised: Given an agent trained extensively with inexact human annotations, can we adapt it to reﬁne the annotation with a few well-annotated examples?

Original: We purposefully enlarge the ground truth bounding boxes in the corrupted MNIST and CUB datasets, and adapt the trained agent using the original ground truth box in an exemplary set of size 5 .
Revised: We purposefully enlarge the ground truth bounding boxes in the corrupted MNIST and CUB datasets, and adapt the trained agent using the original ground truth box in an exemplary set of size 5 .

Original: For corrupted MNIST dataset, the agent, RoI encoder and projection head are ﬁrst trained withdigit 3 images with loose (38 × 38 ) bounding box annotation.
Revised: For corrupted MNIST dataset, the agent, RoI encoder and projection head are ﬁrst trained withdigit 3 images with loose (38 × 38 ) bounding box annotation.

Original: Then the agent is adapted to digit 1 images, where the exemplary size=5, and the ground-truth boxes are 28 × 28.
Revised: Then the agent is adapted to digit 1 images, where the exemplary size=5, and the ground-truth boxes are 28 × 28.

Original: For CUB dataset, during training, the ground-truth box in each resized 224 × 224 image is expanded 10 pixels each side.
Revised: For CUB dataset, during training, the ground-truth box in each resized 224 × 224 image is expanded 10 pixels each side.

Original: During adaptation we use the original ground-truth boxes.
Revised: During adaptation we use the original ground-truth boxes.

Original: In adaptation from entire body to head experiment on CUB dataset, we train on original entire body ground-truth bounding box.
Revised: In adaptation from entire body to head experiment on CUB dataset, we train on original entire body ground-truth bounding box.

Original: During adaptation, 15 randomly sampled exemplar images are annotated with head bounding box.
Revised: During adaptation, 15 randomly sampled exemplar images are annotated with head bounding box.

Original: The agent is trained to adapt to localize head by minimizing the distance between embedding of predicted box and prototype of head embeddings.
Revised: The agent is trained to adapt to localize head by minimizing the distance between embedding of predicted box and prototype of head embeddings.

Original: The results are shown in Figure 7 (left panel).
Revised: The results are shown in Figure 13 (left panel).

Original: Preliminary results with agent trained on the entire bird body and adapt to bird head are shown in Figure 7 (right panel), with a few successful cases on the ﬁrst row and failure cases on the second row. 
Revised: Preliminary results with agent trained on the entire bird body and adapt to bird head are shown in Figure 13 (right panel), with a few successful cases on the ﬁrst row and failure cases on the second row. B. [Figure]

Original: A.1. F EW -S HOT L OCALIZATION ON O MNIGLOT
Revised: F EW -S HOT L OCALIZATION ON O MNIGLOT

Original: The proposed ordinal reward signal also makes our approach amenable to few-shot training, when only a small subset of training images per class are annotated. 
Revised: The proposed ordinal reward signal also makes our approach amenable to few-shot training, when only a small subset of training images per class are annotated. Different from the transfer learning setting, in few-shot setting limited annotations across multiple classes are available during training.

Original: The ordinal reward can be viewed as meta information.
Revised: The ordinal reward can be viewed as meta information.

Original: We evaluate our method under few-shot setting on corrupted Omniglot dataset (Lake et al., 2015) and CUB-warbler dataset.
Revised: We evaluate our method under few-shot setting on corrupted Omniglot dataset (Lake et al., 2015) and CUB-warbler dataset.

Original: For Omniglot, We put each 28 × 28 character in 84 ×random patch background.
Revised: For Omniglot, We put each 28 × 28 character in 84 × 84 random patch background.

Original: The train and test set contains25 different classes respectively, thusimages for each set.
Revised: The train and test set contains different classes respectively, thus 500 images for each set.

Original: We randomly sample 100 iterations for training and testing.
Revised: We randomly sample 100 iterations for training and testing.

Original: For CUB-warbler datset, as we did in Sect.
Revised: For CUB-warbler datset, as we did in Sect.

Original: 4.2 we train on the same 15 species from the “Warbler" class, and adapted to 5 new species of “Warbler", thus 896 and 294 images respectively.
Revised: 4.2 we train on the same 15 species from the “Warbler" class, and adapted to 5 new species of “Warbler", thus 896 and 294 images respectively.

Original: We randomly sampleand 50 iterations for training and testing.
Revised: We randomly sample 100 and 50 iterations for training and testing.

Original: We use 5-shot 5-way, set exemplary set size as 5, and use proto training strategy for both dataset.
Revised: We use 5-shot 5-way, set exemplary set size as 5, and use proto training strategy for both dataset.

Original: The results are shown in Table 7.
Revised: The results are shown in Table 21.

Original: As an implicit meta learning method, our approach achieves 99 .
Revised: As an implicit meta learning method, our approach achieves 99 .

Original: 94% and 90 .
Revised: 94% and 90 .

Original: 52% CorLoc on the two datasets.
Revised: 52% CorLoc on the two datasets.

Original: We can also leverage explicit meta learning method, such as MAML (Finn et al., 2017) to further improve the results.
Revised: We can also leverage explicit meta learning method, such as MAML (Finn et al., 2017) to further improve the results.

Original: We will leave this part as future work.
Revised: We will leave this part as future work.

Original: Although initial results are promising, more efforts are needed to validate whether the proposed RL approach can achieve state-of-the-art performance, but it is beyond the scope of this work.[Table] A.1. S IZE OF E XEMPLARY S ET
Revised: Although initial results are promising, more efforts are needed to validate whether the proposed RL approach can achieve state-of-the-art performance, but it is beyond the scope of this work.

Original: We also compare the effect of different size of exemplary set during training and adaptation on

Original: CUB-warbler dataset.

Original: During training stage, we use shufﬂe proto training strategy, and set exemplary set size as 2, 5, 15, 25.

Original: The results without adaptation on test set are in Table 8.

Original: Both OrdAcc and

Original: CorLoc increase with exemplary set size.

Original: For adaptation stage, the range of exemplary set size is from 2 to 200.

Original: And the results are in Table 9.

Original: The test performance does not increase much with the exemplary set size.

Original: One possible explanation is that the data points in embedding space are compact, thus prototype doesn’t change much when increasing exemplary set size.

Original: We will analyze the inﬂuence of multiple protoptypes per class in future experiments.

Original: Effect of exemplary set size during training stage.

Original: [Table]

Original: [Table]

Original: A. R EPRODUCIBILITY

Original: Corrupted MNIST.

Original: Examples of the four types of corrupted MNIST images are shown in Fig.

Original: Training strategies.

Original: We further evaluate the choice of anchor in the triplet loss for both the pretraining of state represtation and the ordinal reward for the training of agent.

Original: We study ( i ) whether ordinal embedding can be trained in reference to an anchor from a different image instances, ( ii ) whether it is advantageous to use the proptotype embedding of an exemplary set, rather than instance embeddings, and ( iii ) whether mimicking the test condition in training yields any improvement.

Original: [Table]
Revised: [Table]

Original: We use the CUB-Warbler dataset with more foreground background variations than the corrupted

Original: MNIST dataset.

Original: The train and test set contains 15 and 5 disjoint ﬁne-grained classes respectively, resulting 896 images for training (viewed as a single class) and 294 for testing.

Original: Table 10 shows the OrdAcc and CorLoc in four settings.

Original: “Self", use the embedding from images cropped by the ground-truth box from the same instance; “Shufﬂe self", use the ground-truth box cropped image emebedding from a different instance; Similarly, “Proto", use the prototype of a subgroup containing the training instance within the same batch; “Shufﬂe proto (SP)", use the prototype of a subgroup from a different batch without the training instance.

Original: Results suggest that this training strategy brings compactness to the train set, constructing an ordinal structure around the cluster.

Original: For “Shufﬂe proto", while the OrdAcc is lower than others, its CorLoc is the best with large margin.

Original: Matching the condition between training and testing indeed improves generalization to new classes on this dataset.

Original: A.2. I MPLEMENT D ETAILS OF A DAPTATION TO N OVEL T ASKS

Original: Selective localization.

Original: We investigate the agent’s ability in localizing the object speciﬁed by the query set, when the set of images have two common objects.

Original: We use random patched MNIST, where each image has digit 3 and digit 4.

Original: First, the RoI encoder and projection head is trained with an additional contrastive loss to enlarge the distance between the two digits in embedding space,

Original: [Equation]

Original: where loss trip = loss trip 3 + loss trip 4 , learning two local ordinal structure around each class center in embedding space.

Original: We set the margin for both triplet losses as 10, and the margin for contrastive loss as 320 heuristically.

Original: We found that the larger the gap between the two margins, the better the performance.

Original: After learning the RoI encoder and projection head, we train the agent with the reward deﬁned with Eq. 4 in Sect.

Original: 3.3, where a is the prototype embedding of the targeted digit exemplary set.

Original: Figure 5 in Sect.

Original: 4.3 shows an example where the exemplary size is 5.

Original: A.2. A RCHITECTURE OF THE A GENT

Original: The architecture of the agent is shown in Fig.

Original: A.2. I NTERMEDIATE R ESULTS
Revised: C I NTERMEDIATE R ESULTS

Original: The agent tends to predict expected actions to transform the box so that it can ﬁnally get to the target location.
Revised: The agent tends to predict expected actions to transform the box so that it can ﬁnally get to the target location.

Original: It can also be observed from Figure. 10 that the embedding distance between predicted boxes and ground-truth boxes is in agreement with the corresponding IoU.
Revised: It can also be observed from Figure 14 that the embedding distance between predicted boxes and ground-truth boxes is in agreement with the corresponding IoU.

Original: This further demonstrates the effectiveness of the localization agent.
Revised: This further demonstrates the effectiveness of the localization agent.