Original: H OT -R EFRESH M ODEL U PGRADES WITH R EGRESSION - ALLEVIATING C OMPATIBLE T RAINING IN I MAGE R ETRIEVAL
Revised: H OT -R EFRESH M ODEL U PGRADES WITH R EGRESSION -A LLEVIATING C OMPATIBLE T RAINING IN I MAGE R ETRIEVAL

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The task of hot-refresh model upgrades of image retrieval systems plays an essential role in the industry but has never been investigated in academia before.
Revised: The task of hot-refresh model upgrades of image retrieval systems plays an essential role in the industry but has never been investigated in academia before.

Original: Conventional cold-refresh model upgrades can only deploy new models after the gallery is overall backﬁlled, taking weeks or even months for massive data.
Revised: Conventional cold-refresh model upgrades can only deploy new models after the gallery is overall backfilled, taking weeks or even months for massive data.

Original: In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backﬁlling the gallery on-the-ﬂy.
Revised: In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backfilling the gallery on-the-fly.

Original: Compatible training has made it possible, however, the problem of model regression * with negative ﬂips poses a great challenge to the stable improvement of user experience.
Revised: Compatible training has made it possible, however, the problem of model regression * with negative flips poses a great challenge to the stable improvement of user experience.

Original: We argue that it is mainly due to the fact that new-to-old positive query-gallery pairs may show less similarity than new-to-new negative pairs.
Revised: We argue that it is mainly due to the fact that new-to-old positive query-gallery pairs may show less similarity than new-to-new negative pairs.

Original: To solve the problem, we introduce a Regression-Alleviating Compatible Training (RACT) method to properly constrain the feature compatibility while reducing negative ﬂips.
Revised: To solve the problem, we introduce a Regression-Alleviating Compatible Training (RACT) method to properly constrain the feature compatibility while reducing negative flips.

Original: The core is to encourage the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs.
Revised: The core is to encourage the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs.

Original: An efﬁcient uncertainty-based backﬁlling strategy is further introduced to fasten accuracy improvements.
Revised: An efficient uncertainty-based backfilling strategy is further introduced to fasten accuracy improvements.

Original: Extensive experiments on large-scale retrieval benchmarks ( e.g. , Google Landmark) demonstrate that our RACT effectively alleviates the model regression for one more step towards seamless model upgrades.
Revised: Extensive experiments on large-scale retrieval benchmarks ( e.g. , Google Landmark) demonstrate that our RACT effectively alleviates the model regression for one more step towards seamless model upgrades § .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the rapid development of deep learning, image retrieval algorithms have shown great success in large-scale industrial applications, e.g. , face recognition (Sun et al., 2014a;b), object re-identiﬁcation
Revised: With the rapid development of deep learning, image retrieval algorithms have shown great success in large-scale industrial applications, e.g. , face recognition (Sun et al., 2014a;b), object re-identification

Original: (Li et al., 2014; Zheng et al., 2015), image localization (Arandjelovic et al., 2016).
Revised: (Li et al., 2014; Zheng et al., 2015), image localization (Arandjelovic et al., 2016).

Original: Model upgrades are essential to improve user experience in those applications.
Revised: Model upgrades are essential to improve user experience in those applications.

Original: By properly leveraging massive training data and improved network architectures, the retrieval performances could be well boosted.
Revised: By properly leveraging massive training data and improved network architectures, the retrieval performances could be well boosted.

Original: Generally, once the model updates, the process of “backﬁlling” or “re-indexing” is called for ofﬂine re-encoding the gallery image features with the new model.
Revised: Generally, once the model updates, the process of “backfilling” or “re-indexing” is called for offline re-encoding the gallery image features with the new model.

Original: The retrieval system can only beneﬁt from the new model after the gallery is overall backﬁlled, which may cost weeks or even months for billions of industry images in practice.
Revised: The retrieval system can only benefit from the new model after the gallery is overall backfilled, which may cost weeks or even months for billions of industry images in practice.

Original: The above process is termed cold-refresh model upgrades.
Revised: The above process is termed cold-refresh model upgrades.

Original: To harvest the reward of the new model immediately, Shen et al.
Revised: To harvest the reward of the new model immediately, Shen et al.

Original: (2020) introduces the compatible representation learning to train the new model with backward compatibility constraints, so that queries encoded by the new model can be directly indexed by the old gallery features.
Revised: (2020) introduces the compatible representation learning to train the new model with backward compatibility constraints, so that queries encoded by the new model can be directly indexed by the old gallery features.

Original: Meanwhile, as the new features and old features are interchangeable with each other, the gallery images can be backﬁlled on-the-ﬂy, and the retrieval performances would be gradually improved to approach the optimal accuracy of the new model, dubbed as hot-refresh model upgrades (see Figure 1 (a)).
Revised: Meanwhile, as the new features and old features are interchangeable with each other, the gallery images can be backfilled on-the-fly, and the retrieval performances would be gradually improved to approach the optimal accuracy of the new model, dubbed as hot-refresh model upgrades (see Figure 1 (a)).

Original: Although existing compatible training methods (Meng et al., 2021; Shen et al., 2020) make it possible to upgrade the model in a hot-refresh manner, they still face the challenge of model regression (Yan et al., 2020; Li & Hoiem, 2018), which is actually caused by negative ﬂips , i.e. , queries correctly indexed by the old model are incorrectly recognized by the new model (see Figure 1 (b)).

Original: We claimed that, during the procedure of hot-refresh model upgrades, the negative ﬂips occur when

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: B
Revised: B

Original: O
Revised: O

Original: C m e a n A v e r a g e
Revised: C m e a n A v e r a g e

Original: P r e c
Revised: P r e c

Original: i s i o n ( m A P )
Revised: i s i o n ( m A P )

Original: Upgrade progress √ × √
Revised: Upgrade progress √ × √

Original: Nega
Revised: Nega

Original: [Figure]
Revised: [Figure]

Original: Flip
Revised: Although existing compatible training methods (Meng et al., 2021; Shen et al., 2020) make it possible to upgrade the model in a hot-refresh manner, they still face the challenge of model regression (Yan et al., 2020; Xie et al., 2021), which is actually caused by negative flips , i.e. , queries correctly indexed by the old model are incorrectly recognized by the new model (see Figure 1 (b)).

Original: [Figure] the new-to-new similarities between negative query-gallery pairs are larger than the new-to-old similarities between compatible positive query-gallery pairs.
Revised: We claimed that, during the procedure of hot-refresh model upgrades, the negative flips occur when the new-to-new similarities between negative query-gallery pairs are larger than the new-to-old similarities between compatible positive query-gallery pairs.

Original: To solve the problem, we introduce a R egression-
Revised: To solve the problem, we introduce a R egression-

Original: A lleviating C ompatible T raining ( RACT ) method to effectively reduce the negative ﬂips for compatible representation learning and further improve the hot-refresh model upgrades with faster and smoother accuracy convergence in image retrieval systems.
Revised: A lleviating C ompatible T raining ( RACT ) method to effectively reduce the negative flips for compatible representation learning and further improve the hot-refresh model upgrades with faster and smoother accuracy convergence in image retrieval systems.

Original: Speciﬁcally, the conventional backward-compatible regularization is generally designed to pull the new-to-old positive pairs closer and push the new-to-old negative pairs apart from each other.
Revised: Specifically, the conventional backward-compatible regularization is generally designed to pull the new-to-old positive pairs closer and push the new-to-old negative pairs apart from each other.

Original: To reduce negative ﬂips during the online gallery backﬁlling, we improve the backward-compatible regularization by enforcing the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs.
Revised: To reduce negative flips during the online gallery backfilling, we improve the backward-compatible regularization by enforcing the new-to-old positive pairs to be more similar than both the new-to-old negative pairs and the new-to-new negative pairs.

Original: To further improve the user experience during the model upgrades, we introduce an uncertainty-based hot-refresh backﬁlling strategy, which reindexes the gallery images following “the-poor-ﬁrst” policy.
Revised: To further improve the user experience during the model upgrades, we introduce an uncertainty-based hot-refresh backfilling strategy, which reindexes the gallery images following “the-poor-first” policy.

Original: To measure the refresh priority of gallery images in a lightweight manner, we feed the old gallery features into the new classiﬁer (one fully-connected layer in general), yielding the probability distribution on new classes.
Revised: To measure the refresh priority of gallery images in a lightweight manner, we feed the old gallery features into the new classifier (one fully-connected layer in general), yielding the probability distribution on new classes.

Original: Even though ground-truth classes may not exist for the old gallery features, we can assume that more discriminative old features are expected to have sharper probability vectors given their compatible latent space.
Revised: Even though ground-truth classes may not exist for the old gallery features, we can assume that more discriminative old features are expected to have sharper probability vectors given their compatible latent space.

Original: Uncertainty estimation ( e.g. , Entropy-based) is utilized here to evaluate the sharpness of the probability vectors, and gallery images with larger uncertainty should have higher priority for backﬁlling.
Revised: Uncertainty estimation ( e.g. , Entropy-based) is utilized here to evaluate the sharpness of the probability vectors, and gallery images with larger uncertainty should have higher priority for backfilling.

Original: Experimental results on large-scale image retrieval benchmarks ( i.e. , Google Landmark v2 (Weyand et al., 2020), RParis, and ROxford (Radenovi ´ c et al., 2018)) demonstrate the effectiveness of our
Revised: Experimental results on large-scale image retrieval benchmarks ( i.e. , Google Landmark v2 (Weyand et al., 2020), RParis, and ROxford (Radenovi´c et al., 2018)) demonstrate the effectiveness of our

Original: RACT method in alleviating the problem of model regression for hot-refresh model upgrades.
Revised: RACT method in alleviating the problem of model regression for hot-refresh model upgrades.

Original: Our contributions can be summarized three-fold.
Revised: Our contributions can be summarized three-fold.

Original: (1) We study the model regression problem in hot-refresh model upgrades of image retrieval systems for the ﬁrst time.
Revised: (1) We study the model regression problem in hot-refresh model upgrades of image retrieval systems for the first time.

Original: (2) A regression-alleviating backward-compatible regularization is proposed to alleviate the issue of negative ﬂips during compatible training.
Revised: (2) A regression-alleviating backward-compatible regularization is proposed to alleviate the issue of negative flips during compatible training.

Original: An uncertainty-based backﬁlling strategy is introduced to further improve the user experience with “the-poor-ﬁrst” hot refresh.
Revised: An uncertainty-based backfilling strategy is introduced to further improve the user experience with “the-poor-first” hot refresh.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Compatible Representation Learning.
Revised: Compatible Representation Learning.

Original: The task of compatible representation learning aims at encoding new features that are interoperable with the old features.
Revised: The task of compatible representation learning aims at encoding new features that are interoperable with the old features.

Original: It breaks the inherent cold-refresh way of model iteration in the industry - the beneﬁt of new models can be immediately reaped in a backﬁll-free manner.
Revised: It breaks the inherent cold-refresh way of model iteration in the industry - the benefit of new models can be immediately reaped in a backfill-free manner.

Original: Shen et al.
Revised: Shen et al.

Original: ﬁrst formulated the problem of backward-compatible learning and proposed to utilize the old classiﬁer for supervising the compatible feature learning.
Revised: (2020) first formulated the problem of backward-compatible learning and proposed to utilize the old classifier for supervising the compatible feature learning.

Original: Budnik & Avrithis (2020) made an effort to reduce the gap between the large model and a small one without the parametric classiﬁer.
Revised: Budnik & Avrithis (2020) made an effort to reduce the gap between the large model and a small one with- out the parametric classifier.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) introduced a projection head to transform the old features to be compatible with the new ones.
Revised: (2020) introduced a projection head to transform the old features to be compatible with the new ones.

Original: Li et al.
Revised: Li et al.

Original: (2020b) proposed a framework, namely “Feature Lenses”, to encourage image representations transformation-invariant.
Revised: (2020b) proposed a framework, namely “Feature Lenses”, to encourage image representations transformation-invariant.

Original: To balance the trade-off between performance and efﬁciency, Duggal et al.
Revised: To balance the trade-off between performance and efficiency, Duggal et al.

Original: (2021) designed a compatibility-aware neural architecture search scheme to improve the compatibility of models with different sizes.
Revised: (2021) designed a compatibility-aware neural architecture search scheme to improve the compatibility of models with different sizes.

Original: However, since existing compatible algorithms for image retrieval have not investigated the application of hot-refresh model upgrades, the problem of model regression has been overlooked.
Revised: However, since existing compatible algorithms for image retrieval have not investigated the application of hot-refresh model upgrades, the problem of model regression has been overlooked.

Original: Model Regression.
Revised: Model Regression.

Original: Model upgrades play a vital role in improving the performance of industrial applications, where the issue of model regression inevitably harms the user experience.
Revised: Model upgrades play a vital role in improving the performance of industrial applications, where the issue of model regression inevitably harms the user experience.

Original: Yan et al.
Revised: Yan et al.

Original: (2020) studied the problem of model regression in image classiﬁcation, where the negative ﬂips indicate the images correctly classiﬁed by the old model but misclassiﬁed by the new model.
Revised: (2020) studied the problem of model regression in image classification, where the negative flips indicate the images correctly classified by the old model but misclassified by the new model.

Original: A method, namely focal distillation, was introduced to encourage model congruence on positive samples.
Revised: A method, namely focal distillation, was introduced to encourage model congruence on positive samples.

Original: Srivastava et al.
Revised: Srivastava et al.

Original: (2020) also investigated the model regression in classiﬁcation and proposed a new metric to evaluate the regression effects.
Revised: (2020) also investigated the model regression in classification and proposed a new metric to evaluate the regression effects.

Original: Xie et al.
Revised: Xie et al.

Original: (2021) probed the model regression problem in natural language processing.
Revised: (2021) probed the model regression problem in natural language processing.

Original: The works mentioned above all focused on the classiﬁcation tasks.
Revised: The works mentioned above all focused on the classification tasks.

Original: Directly employing their methods on the retrieval tasks leads to sub-optimal performances.
Revised: Directly employing their methods on the retrieval tasks leads to sub-optimal performances.

Original: Moreover, they did not need to consider hot-refresh gallery backﬁlling in image classiﬁcation.
Revised: Moreover, they did not need to consider hot-refresh gallery backfilling in image classification.

Original: Active Learning.
Revised: Active Learning.

Original: Our uncertainty-based backﬁlling strategy is related to active learning.
Revised: Our uncertainty-based backfilling strategy is related to active learning.

Original: As the real world is teeming with unlabeled data, active learning makes efforts to achieve better performance with fewer annotated samples by choosing valuable samples actively (Settles, 2009).
Revised: As the real world is teeming with unlabeled data, active learning makes efforts to achieve better performance with fewer annotated samples by choosing valuable samples actively (Settles, 2009).

Original: Active learning has been widely adopted in deep learning tasks, e.g. , object detection (Holub et al., 2008), text classiﬁcation (Zhu et al., 2008), and image classiﬁcation (Cao et al., 2020).
Revised: Active learning has been widely adopted in deep learning tasks, e.g. , object detection (Holub et al., 2008), text classification (Zhu et al., 2008), and image classification (Cao et al., 2020).

Original: Though obtaining the priority of samples is a common issue between active learning and gallery backﬁlling, the latter focuses more on enjoying beneﬁts in model upgrades with minimal computational overhead.
Revised: Though obtaining the priority of samples is a common issue between active learning and gallery backfilling, the latter focuses more on enjoying benefits in model upgrades with minimal computational overhead.

Original: Image Retrieval.
Revised: Image Retrieval.

Original: The task of image retrieval requires searching images of the same content or object from a large-scale database based on their feature similarities, given images of interest.
Revised: The task of image retrieval requires searching images of the same content or object from a large-scale database based on their feature similarities, given images of interest.

Original: It is a fundamental task for many visual applications, e.g. , face recognition (Guillaumin et al., 2009; Cui et al., 2013), person re-identiﬁcation (Bak & Carr, 2017;
Revised: It is a fundamental task for many visual applications, e.g. , face recognition (Guillaumin et al., 2009; Cui et al., 2013), person re-identification (Bak & Carr, 2017;

Original: Liu et al., 2017), image localization (Arandjelovic et al., 2016;
Revised: Liu et al., 2017), image localization (Arandjelovic et al., 2016;

Original: Ge et al., 2020b).
Revised: Ge et al., 2020b).

Original: Deep learning-based methods improve the accuracy of image retrieval by casting it as metric learning tasks (Gordo et al., 2016; Chen et al., 2018; Brown et al., 2020), architecture design/search tasks (Zhou et al., 2019), or self-/semi-supervised learning tasks (Ge et al., 2020c;a).
Revised: Deep learning-based methods improve the accuracy of image retrieval by casting it as metric learning tasks (Gordo et al., 2016; Chen et al., 2018; Brown et al., 2020), architecture design/search tasks (Zhou et al., 2019), or self-/semi-supervised learning tasks (Ge et al., 2020c;a).

Original: Although many efforts have been made to improve retrieval performance, few people pay attention to the bottleneck of model upgrades in industrial applications.
Revised: Although many efforts have been made to improve retrieval performance, few people pay attention to the bottleneck of model upgrades in industrial applications.

Original:  H OT -R EFRESH M ODEL U PGRADES OF I MAGE R ETRIEVAL S YSTEMS
Revised:  H OT -R EFRESH M ODEL U PGRADES OF I MAGE R ETRIEVAL S YSTEMS

Original: Given images of interest (referred to as query Q ), image retrieval targets correctly recognizing images of the same content or objects from a large-scale database (referred to as gallery G ).
Revised: Given images of interest (referred to as query Q ), image retrieval targets correctly recognizing images of the same content or objects from a large-scale database (referred to as gallery G ).

Original: An image encoder φ ( · ) is employed to map the images into feature vectors.
Revised: An image encoder ϕ ( · ) is employed to map the images into feature vectors.

Original: Then, the dot product of normalized vectors is used for measuring the similarity between query and gallery images for ranking.
Revised: Then, the dot product of normalized vectors is used for measuring the similarity between query and gallery images for ranking.

Original: 3. H OT -R EFRESH M ODEL U PGRADES WITH C OMPATIBLE R EPRESENTATIONS
Revised: 3. H OT -R EFRESH M ODEL U PGRADES WITH C OMPATIBLE R EPRESENTATIONS

Original: Let φ old to be the old image encoder that was deployed in the retrieval system and φ new to be a stronger new encoder which is expected to replace the old one.
Revised: Let ϕ old to be the old image encoder that was deployed in the retrieval system and ϕ new to be a stronger new encoder which is expected to replace the old one.

Original: We denote the evaluation metric for image retrieval, e.g. , mean Average Precision (mAP), as M ( · , · ) .
Revised: We denote the evaluation metric for image retrieval, e.g. , mean Average Precision (mAP), as M ( · , · ) .

Original: The accuracy of the old system is denoted as M ( Q old , G old ) , where Q old , G old are the sets of query and gallery features embedded by φ old , respectively.
Revised: The accuracy of the old system is denoted as M ( Q old , G old ) , where Q old , G old are the sets of query and gallery features embedded by ϕ old , respectively.

Original: The new encoder shows an improved accuracy of M ( Q new , G new ) .
Revised: The new encoder shows an improved accuracy of M ( Q new , G new ) .

Original: In conventional cold-refresh model updating pipeline, ofﬂine backﬁlling the gallery images from G old to G new may take months for the massive data in industry applications, reducing the efﬁciency of the retrieval system upgrades to a large extent.
Revised: In conventional cold-refresh model updating pipeline, offline backfilling the gallery images from G old to G new may take months for the massive data in industry applications, reducing the efficiency of the retrieval system upgrades to a large extent.

Original: Thanks to the compatible representation learning (Shen et al., 2020), query features encoded by the new encoder ( i.e. , Q new ) can be directly compared to the old gallery features ( i.e. , G old ).
Revised: Thanks to the compatible representation learning (Shen et al., 2020), query features encoded by the new encoder ( i.e. , Q new ) can be directly compared to the old gallery features ( i.e. , G old ).

Original: The retrieval system can beneﬁt from the new encoder immediately in a backﬁll-free manner, signiﬁcantly accelerating the model iteration at the expense of the optimal performance,
Revised: The retrieval system can benefit from the new encoder immediately in a backfill-free manner, significantly accelerating the model iteration at the expense of the optimal performance,

Original: [Equation]
Revised: [Equation]

Original:  Given that the new features and old features are interchangeable due to their compatibility, the gallery images can be backﬁlled on-the-ﬂy to progressively improve the retrieval accuracy to achieve[Figure] the optimal performance of the new model,
Revised: [Figure] Given that the new features and old features are interchangeable due to their compatibility, the gallery images can be backfilled on-the-fly to progressively improve the retrieval accuracy to achieve the optimal performance of the new model,

Original: [Equation]
Revised: [Equation]

Original: The above process is termed as hot-refresh model upgrades, where G 20% new indicates the 20% of images in the gallery have been re-indexed by the new model φ new .
Revised: The above process is termed as hot-refresh model upgrades, where G 20% new indicates the 20% of images in the gallery have been re-indexed by the new model ϕ new .

Original: 3. M ODEL R EGRESSION WITH N EGATIVE F LIPS
Revised: 3. M ODEL R EGRESSION WITH N EGATIVE F LIPS

Original: Ideally, the performance of the retrieval system should be smoothly upgraded along with the gallery online backﬁlling, and the user experience would be continuously improved.
Revised: Ideally, the performance of the retrieval system should be smoothly upgraded along with the gallery online backfilling, and the user experience would be continuously improved.

Original: However, we observe the phenomenon of model regression especially in the early backﬁlling stages, that is, the accuracy degrades when part of the gallery images are refreshed, e.g. ,
Revised: However, we observe the phenomenon of model regression especially in the early backfilling stages, that is, the accuracy degrades when part of the gallery images are refreshed, e.g. ,

Original: [Equation]
Revised: [Equation]

Original: It is actually due to the negative ﬂips , where some query images correctly retrieved by the old model are incorrectly indexed by the new model during the upgrade process.
Revised: It is actually due to the negative flips , where some query images correctly retrieved by the old model are incorrectly indexed by the new model during the upgrade process.

Original: There are two main potential factors that cause the negative ﬂips: (1) new-to-old query-gallery positive pairs are less similar than new-to-new query-gallery negative pairs; (2) new-to-new positives are less similar than new-to-old negative pairs.
Revised: There are two main potential factors that cause the negative flips: (1) new-to-old query-gallery positive pairs are less similar than new-to-new query-gallery negative pairs; (2) new-to-new positives are less similar than new-to-old negative pairs.

Original: Given that the new model generally encodes more discriminative features than the old model for the overall performance gains, we focus on the former one in this paper, i.e. , there exists a query x q that satisﬁes
Revised: Given that the new model generally encodes more discriminative features than the old model for the overall performance gains, we focus on the former one in this paper, i.e. , there exists a query x q that satisfies

Original: [Equation]
Revised: [Equation]

Original: where x pos g is the positive gallery sample, x neg g is the negative gallery sample, and (cid:104)· , ·(cid:105) denotes the cosine similarity of two normalized feature vectors.
Revised: where x pos g is the positive gallery sample, x neg g is the negative gallery sample, and ⟨· , ·⟩ denotes the cosine similarity of two normalized feature vectors.

Original: In the case of Eq.
Revised: In the case of Eq.

Original: (3), φ new ( x q ) ∈ Q new , φ old ( x pos g ) ∈
Revised: (3), ϕ new ( x q ) ∈ Q new , ϕ old ( x pos g ) ∈

Original: G 80% old and φ new ( x neg g ) ∈ G 20% new .
Revised: G 80% old and ϕ new ( x neg g ) ∈ G 20% new .

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We introduce a regression-alleviating compatible training approach (Figure 2) for tackling the challenge of model regression in hot-refresh model upgrades of image retrieval systems, which has essential impacts on the user experience but was ignored by previous compatible training methods.
Revised: We introduce a regression-alleviating compatible training approach (Figure 2) for tackling the challenge of model regression in hot-refresh model upgrades of image retrieval systems, which has essential impacts on the user experience but was ignored by previous compatible training methods.

Original: Our key idea is to encourage the new-to-old positive pairs to be more similar than both new-to-old negative pairs and new-to-new negative pairs.
Revised: Our key idea is to encourage the new-to-old positive pairs to be more similar than both new-to-old negative pairs and new-to-new negative pairs.

Original: In addition, an uncertainty-based strategy is proposed to backﬁll the gallery in “the-poor-ﬁrst” manner, in order to fasten accuracy improvements.
Revised: In addition, an uncertainty-based strategy is proposed to backfill the gallery in “the-poor-first” manner, in order to fasten accuracy improvements.

Original: 4. R EVISIT C ONVENTIONAL C OMPATIBLE T RAINING
Revised: 4. R EVISIT C ONVENTIONAL C OMPATIBLE T RAINING

Original: Being cast as either the metric learning task (Budnik & Avrithis, 2020) or the transformation learning task (Wang et al., 2020), the core of existing compatible training methods is to encourage the new feature of an image to approach the old features of the intra-class images while keeping away from the old features of different classes.
Revised: Being cast as either the metric learning task (Budnik & Avrithis, 2020) or the transformation learning task (Wang et al., 2020), the core of existing compatible training methods is to encourage the new feature of an image to approach the old features of the intra-class images while keeping away from the old features of different classes.

Original: We treat the compatible training as an instance discriminationlike task ( i.e. , the positive pair is the new and old features of the same instance) for efﬁcient training in large-scale retrieval tasks.
Revised: We treat the compatible training as an instance discriminationlike task ( i.e. , the positive pair is the new and old features of the same instance) for efficient training in large-scale retrieval tasks.

Original: To verify that such an instance discrimination-like strategy does not hurt the retrieval accuracy as well as the feature compatibility, we conduct an experiment by using a pre-sampler for constructing positive pairs of intra-class instances for comparison.
Revised: To verify that such an instance discrimination-like strategy does not hurt the retrieval accuracy as well as the feature compatibility, we conduct an experiment by using a pre-sampler for constructing positive pairs of intra-class instances for comparison.

Original: As shown in
Revised: As shown in

Original: Table 6 of Appendix, similar results can be observed with or without a pre-sampler.
Revised: Table 6 of Appendix, similar results can be observed with or without a pre-sampler.

Original: The reason for the similar results might be that the most signiﬁcant objective of compatible training is to train the new model to map images to same features as the old model rather than constraining the distance between positive pairs, according to (Budnik & Avrithis, 2020).
Revised: The reason for the similar results might be that the most significant objective of compatible training is to train the new model to map images to same features as the old model rather than constraining the distance between positive pairs, according to (Budnik & Avrithis, 2020).

Original: Note that the negative pair is built by the new and old features of inter-class instances in the mini-batch, i.e. , the intra-class samples would not be treated as the negatives in our instance discrimination-like compatible training.
Revised: Note that the negative pair is built by the new and old features of inter-class instances in the mini-batch, i.e. , the intra-class samples would not be treated as the negatives in our instance discrimination-like compatible training.

Original: We denote T as the training set and B as the mini-batch.
Revised: We denote T as the training set and B as the mini-batch.

Original: Given an anchor image x ∈ T , we can formulate the backward-compatible regularization as a contrastive loss,
Revised: Given an anchor image x ∈ T , we can formulate the backward-compatible regularization as a contrastive loss,

Original: [Equation]
Revised: [Equation]

Original: [Equation] where τ is a temperature hyper-parameter and p ( x ) denotes the set of intra-class samples for x (including x ) in the mini-batch.
Revised: (5) where τ is a temperature hyper-parameter and p ( x ) denotes the set of intra-class samples for x (including x ) in the mini-batch.

Original: With the above regularization, φ new is trained to embed new features on the same latent space as the old features.
Revised: With the above regularization, ϕ new is trained to embed new features on the same latent space as the old features.

Original: However, they ignore the relation between new-old and new-new similarities, facing the risk of negative ﬂips during model upgrading.
Revised: However, they ignore the relation between new-old and new-new similarities, facing the risk of negative flips during model upgrading.

Original: 4. R EGRESSION - ALLEVIATING C OMPATIBLE T RAINING
Revised: 4. R EGRESSION - ALLEVIATING C OMPATIBLE T RAINING

Original: Regression-alleviating Compatibility Regularization.
Revised: Regression-alleviating Compatibility Regularization.

Original: To constrain the feature compatibility and reduce negative ﬂips at the same time, we improve the conventional backward-compatible loss by adding new-to-new negative pairs as an extra regularization term.
Revised: To constrain the feature compatibility and reduce negative flips at the same time, we improve the conventional backward-compatible loss by adding new-to-new negative pairs as an extra regularization term.

Original: Such an improved backward- compatible loss provides regression-alleviating regularization during training, that is formulated asL ra-
Revised: Such an improved backward- compatible loss provides regression-alleviating regularization during training, that is formulated as

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] 
Revised: [Equation] [Table]

Original: In such a way, the new and old compatible positive pairs are trained to achieve more prominent similarities than both new-to-old and new-to-new negative pairs.
Revised: In such a way, the new and old compatible positive pairs are trained to achieve more prominent similarities than both new-to-old and new-to-new negative pairs.

Original: The issue of negative ﬂips can be alleviated properly when the gallery backﬁlling proceeds, i.e. , the gallery is a mix-up of both old and new features.
Revised: The issue of negative flips can be alleviated properly when the gallery backfilling proceeds, i.e. , the gallery is a mix-up of both old and new features.

Original: We do not speciﬁcally constrain the new-to-new positive pairs to be closer than new-to-old negative pairs, as the new features are generally trained to be more discriminative than the old ones in order to achieve the model upgrading purpose.
Revised: We do not specifically constrain the new-to-new positive pairs to be closer than new-to-old negative pairs, as the new features are generally trained to be more discriminative than the old ones in order to achieve the model upgrading purpose.

Original: Thus, the new-to-new positives are naturally to achieve larger similarities than new-to-old negatives.
Revised: Thus, the new-to-new positives are naturally to achieve larger similarities than new-to-old negatives.

Original: Overall Training Objective.
Revised: Overall Training Objective.

Original: Besides the compatibility constraints, we also need the ordinary training regularization for image retrieval tasks to maintain the distinctiveness of the new feature representations.
Revised: Besides the compatibility constraints, we also need the ordinary training regularization for image retrieval tasks to maintain the distinctiveness of the new feature representations.

Original: Following the strong benchmark in (Weyand et al., 2020), we choose the classiﬁcation as a pretext task during training.
Revised: Following the strong benchmark in (Weyand et al., 2020), we choose the classification as a pretext task during training.

Original: Speciﬁcally, we adopt a fully-connected layer as the classiﬁer, denoted as ω new : f → { 1 , · · · , C } , where C is the number of classes of the new training dataset.
Revised: Specifically, we adopt a fully-connected layer as the classifier, denoted as ω new : f → { 1 , · · · , C } , where C is the number of classes of the new training dataset.

Original: The classiﬁcation loss for the pretext task is formulated as
Revised: The classification loss for the pretext task is formulated as

Original: [Equation]
Revised: [Equation]

Original: 7) where (cid:96) ce denotes the cross-entropy loss, y ( x ) is the ground-truth label for x , and ω new ◦ φ new ( x ) produces the class logits after a softmax operation.
Revised: (7) where ℓ ce denotes the cross-entropy loss, y ( x ) is the ground-truth label for x , and ω new ◦ ϕ new ( x ) produces the class logits after a softmax operation.

Original: The overall training objective function for our regression-alleviating compatible training framework is,
Revised: The overall training objective function for our regression-alleviating compatible training framework is,

Original: [Equation]
Revised: [Equation]

Original: where λ is the loss weight.
Revised: where λ is the loss weight.

Original: New models trained towards the above objectives enable better hotrefresh upgrades of image retrieval systems by properly alleviating the problem of model regression.
Revised: New models trained towards the above objectives enable better hotrefresh upgrades of image retrieval systems by properly alleviating the problem of model regression.

Original: 4. U NCERTAINTY - BASED B ACKFILLING
Revised: 4. U NCERTAINTY - BASED B ACKFILLING

Original: In order to further fasten the accuracy convergence during the hot-refresh model upgrades, we propose to refresh the gallery following “the-poor-ﬁrst” priority, i.e. , images with non-discriminative features are expected to be re-encoded ﬁrst.
Revised: In order to further fasten the accuracy convergence during the hot-refresh model upgrades, we propose to refresh the gallery following “the-poor-first” priority, i.e. , images with non-discriminative features are expected to be re-encoded first.

Original: Given the massive data in industrial applications, we need to quickly measure the distinctiveness of old gallery features in a relatively lightweight way.
Revised: Given the massive data in industrial applications, we need to quickly measure the distinctiveness of old gallery features in a relatively lightweight way.

Original: An uncertainty-based strategy is therefore introduced to measure the priority for backﬁlling the gallery.
Revised: An uncertainty-based strategy is therefore introduced to measure the priority for backfilling the gallery.

Original: We claim that the conﬁdence of class probabilities represents the feature distinctiveness to some extent.
Revised: We claim that the confidence of class probabilities represents the feature distinctiveness to some extent.

Original: Since the old and new latent spaces are directly comparable after compatible training, we can produce the class probabilities of the old gallery features by feeding them into the trained new classiﬁer ω new .
Revised: Since the old and new latent spaces are directly comparable after compatible training, we can produce the class probabilities of the old gallery features by feeding them into the trained new classifier ω new .

Original: Although the ground-truth classes of the gallery images are generally non-overlapped with the new training dataset and even unknown, we can assume that more discriminative features should have sharper class probability vectors.
Revised: Although the ground-truth classes of the gallery images are generally non-overlapped with the new training dataset and even unknown, we can assume that more discriminative features should have sharper class probability vectors.

Original: Inspired by the “uncertainty sampling” (Settles, 2009), we evaluate the sharpness of the probability vectors using the uncertainty estimation metrics.
Revised: Inspired by the “uncertainty sampling” (Settles, 2009), we evaluate the sharpness of the probability vectors using the uncertainty estimation metrics.

Original: Samples with larger uncertainty scores will be assigned higher priority to be backﬁlled.
Revised: Samples with larger uncertainty scores will be assigned higher priority to be backfilled.

Original: We consider three common ways of measuring uncertainty in this paper.
Revised: We consider three common ways of measuring uncertainty in this paper.

Original: Least Conﬁdence Uncertainty.
Revised: Least Confidence Uncertainty.

Original: It considers the gap between 100% and the probability of the predicted class as the uncertainty score for each sample, formulated as
Revised: It considers the gap between 100% and the probability of the predicted class as the uncertainty score for each sample, formulated as

Original: [Equation]
Revised: [Equation]

Original: where softmax [ · ] k denotes the probability of rank k -th conﬁdent class.
Revised: where softmax [ · ] k denotes the probability of rank k -th confident class.

Original: Margin of Conﬁdence Uncertainty.
Revised: Margin of Confidence Uncertainty.

Original: It measures the uncertainty of class probabilities by the difference between the top- 2 conﬁdent predictions,
Revised: It measures the uncertainty of class probabilities by the difference between the top- 2 confident predictions,

Original: [Equation]
Revised: [Equation]

Original: Entropy-based Uncertainty.
Revised: Entropy-based Uncertainty.

Original: The above metrics only consider the top- 1 or top- 2 classes, while the
Revised: The above metrics only consider the top- 1 or top- 2 classes, while the

Original: Entropy-based uncertainty measurement takes the overall probability distribution into consideration.
Revised: Entropy-based uncertainty measurement takes the overall probability distribution into consideration.

Original: [Equation]
Revised: [Equation]

Original: where C is the number of classes in new training set.
Revised: where C is the number of classes in new training set.

Original: The introduced two technical parts of our work, i.e. , regression-alleviating compatible training and uncertainty-based backﬁlling strategy, together contribute to alleviating the model regression problem in hot-refresh model upgrades.
Revised: The introduced two technical parts of our work, i.e. , regression-alleviating compatible training and uncertainty-based backfilling strategy, together contribute to alleviating the model regression problem in hot-refresh model upgrades.

Original: The former explicitly regularizes the models to reduce negative ﬂips at training time, while the latter can implicitly reduce negative ﬂips by refreshing the poor features (which show a higher risk for negative ﬂips) in priority at test time.
Revised: The former explicitly regularizes the models to reduce negative flips at training time, while the latter can implicitly reduce negative flips by refreshing the poor features (which show a higher risk for negative flips) in priority at test time.

Original: They can be considered as an entire solution (from training to testing) for hot-refresh model upgrades.
Revised: They can be considered as an entire solution (from training to testing) for hot-refresh model upgrades.

Original:  E XPERIMENTS 5. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 5. I MPLEMENTATION D ETAILS

Original: Training Data.
Revised: Training Data.

Original: Google Landmark v2 (Weyand et al., 2020), GLDv2 in short, is a large-scale public dataset for landmark retrieval.
Revised: Google Landmark v2 (Weyand et al., 2020), GLDv2 in short, is a large-scale public dataset for landmark retrieval.

Original: Following the setup in (Jeon, 2020), we use a clean subset of
Revised: Following the setup in (Jeon, 2020), we use a clean subset of

Original: GLDv2, dubbed GLDv2-clean, which consists of 1,580,470 samples with 81,313 classes.
Revised: GLDv2, dubbed GLDv2-clean, which consists of 1,580,470 samples with 81,313 classes.

Original: In this paper, we investigate three different kinds of data allocations for training the old model and the new model, covering most scenarios of model upgrades in practice.
Revised: In this paper, we investigate three different kinds of data allocations for training the old model and the new model, covering most scenarios of model upgrades in practice.

Original: (1) Expansion : the new training set is an expansion of the old training set.
Revised: (1) Expansion : the new training set is an expansion of the old training set.

Original: We randomly sample 30% of images as the old training set, and use the 100% of images as the new training set.
Revised: We randomly sample 30% of images as the old training set, and use the 100% of images as the new training set.

Original: (2) Open-data : the new training data are excluded from the old training data, but they share the same classes.
Revised: (2) Open-data : the new training data are excluded from the old training data, but they share the same classes.

Original: We randomly sample 30% of each class as the old training set, and use the rest 70% of each class as the new training set.
Revised: We randomly sample 30% of each class as the old training set, and use the rest 70% of each class as the new training set.

Original: Open-class : the new training data and classes are both excluded from the old ones.
Revised: Open-class : the new training data and classes are both excluded from the old ones.

Original: We randomly sample 30% of classes for the old model training, and images of the rest 70% of classes are utilized for the new model training.
Revised: We randomly sample 30% of classes for the old model training, and images of the rest 70% of classes are utilized for the new model training.

Original: The training data details can be found in Table 1.
Revised: The training data details can be found in Table 1.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  vanilla
Revised: of vanilla

Original: NFR
Revised: NFR

Original: [Figure]
Revised: [Figure]

Original: vanilla
Revised: vanilla

Original: [Figure]
Revised: [Figure]

Original: Testing Data.
Revised: Testing Data.

Original: We evaluate the trained models on three test sets, including GLDv2-test, Revisited
Revised: We evaluate the trained models on three test sets, including GLDv2-test, Revisited

Original: Oxford (ROxford), and Revisited Paris (RParis).
Revised: Oxford (ROxford), and Revisited Paris (RParis).

Original: GLDv2-test contains 761,757 gallery images and 750 query images.
Revised: GLDv2-test contains 761,757 gallery images and 750 query images.

Original: Following the test protocol in (Radenovi ´ c et al., 2018), we use the medium testing with 70 queries for both ROxford and RParis.
Revised: Following the test protocol in (Radenovi ´ c et al., 2018), we use the medium testing with 70 queries for both ROxford and RParis.

Original: ROxford (Philbin et al., 2007) contains 5,007 gallery images, and RParis (Philbin et al., 2008) consists of 6,357 gallery images.
Revised: ROxford (Philbin et al., 2007) contains 5,007 gallery images, and RParis (Philbin et al., 2008) consists of 6,357 gallery images.

Original: Evaluation Metric.
Revised: Evaluation Metric.

Original: Mean Average Precision ( mAP ) is adopted here for evaluating the retrieval performance.
Revised: Mean Average Precision ( mAP ) is adopted here for evaluating the retrieval performance.

Original: Following the evaluation protocols in (Budnik & Avrithis, 2020; Weyand et al., 2020), we use mAP@100 for GLDv2-test, and mAP@10 for ROxford and RParis.
Revised: Following the evaluation protocols in (Budnik & Avrithis, 2020; Weyand et al., 2020), we use mAP@100 for GLDv2-test, and mAP@10 for ROxford and RParis.

Original: Since mAP measures the overall accuracy with the consideration of both positive ﬂips and negative ﬂips, we deﬁne a metric, namely negative ﬂip rate ( NFR ), to measure the degree of model regression in hot-refresh model upgrades.
Revised: Since mAP measures the overall accuracy with the consideration of both positive flips and negative flips, we define a metric, namely negative flip rate ( NFR ), to measure the degree of model regression in hot-refresh model upgrades.

Original: Note that our NFR is speciﬁcally deﬁned for image retrieval tasks, different from the one introduced in (Yan et al., 2020).
Revised: Note that our NFR is specifically defined for image retrieval tasks, different from the one introduced in (Yan et al., 2020).

Original: Our NFR is formulated as
Revised: Our NFR is formulated as

Original: [Equation]
Revised: [Equation]

Original: where ˆ Q + k denotes the set of queries that correctly recall relative images from the old gallery in topk candidates.
Revised: where ˆ Q + k denotes the set of queries that correctly recall relative images from the old gallery in topk candidates.

Original: Q − k is the set of queries that fail to be recalled by top- k galleries during upgrading, where the gallery is a mix-up of both old and new features.
Revised: Q − k is the set of queries that fail to be recalled by top- k galleries during upgrading, where the gallery is a mix-up of both old and new features.

Original: NFR@1 is used in this paper.
Revised: NFR@1 is used in this paper.

Original: Architectures.
Revised: Architectures.

Original: We study two kinds of architecture combinations, (1) R50-R101 : ResNet-50 (He et al., 2016) for the old model, and ResNet-101 for the new model; (2) R50-R50 : ResNet-50 for both old and new models.
Revised: We study two kinds of architecture combinations, (1) R50-R101 : ResNet-50 (He et al., 2016) for the old model, and ResNet-101 for the new model; (2) R50-R50 : ResNet-50 for both old and new models.

Original: Due to space limitation, we discuss R50-R101 in the main paper, and the results of R50-R50 can be found in Appendix A.2.
Revised: Due to space limitation, we discuss R50-R101 in the main paper, and the results of R50-R50 can be found in Appendix A.2.

Original: Training details can be found in Appendix A.1.
Revised: Training details can be found in Appendix A.1.

Original: 5. R EGRESSION - ALLEVIATING C OMPATIBLE T RAINING
Revised: 5. R EGRESSION - ALLEVIATING C OMPATIBLE T RAINING

Original: We start with the architecture combination of R50-R101, i.e. , ResNet-50 for the old model, and
Revised: We start with the architecture combination of R50-R101, i.e. , ResNet-50 for the old model, and

Original: ResNet-101 for the new one.
Revised: ResNet-101 for the new one.

Original: To verify that our regression-alleviating compatibility regularization (Eq.
Revised: To verify that our regression-alleviating compatibility regularization (Eq.

Original: (6)) can properly alleviate the issue of model regression, we compare our method to the vanilla compatible training, i.e. , L vanilla ( x ) =
Revised: (6)) can properly alleviate the issue of model regression, we compare our method to the vanilla compatible training, i.e. , L vanilla ( x ) =

Original: L cls ( x ) + λ L comp ( x ) , where L cls is deﬁned in Eq.
Revised: L cls ( x ) + λ L comp ( x ) , where L cls is defined in Eq.

Original: (7), and
Revised: (7), and

Original: L comp is deﬁned in Eq.
Revised: L comp is defined in Eq.

Original: Our uncertainty-based backﬁlling strategy is not used in this section in order to fairly compare with the vanilla settings.
Revised: Our uncertainty-based backfilling strategy is not used in this section in order to fairly compare with the vanilla settings.

Original: The only variable here is the compatibility regularization, i.e. , L comp for vanilla and L ra-comp for ours .
Revised: The only variable here is the compatibility regularization, i.e. , L comp for vanilla and L ra-comp for ours .

Original: We evaluate the models trained with expansion, open-data and open-class datasets (see Table 1) by testing on three different test sets, i.e. , GLDv2test, ROxford, and RParis, as illustrated in Figure 3, 4, 5, respectively.
Revised: We evaluate the models trained with expansion, open-data and open-class datasets (see Table 1) by testing on three different test sets, i.e. , GLDv2test, ROxford, and RParis, as illustrated in Figure 3, 4, 5, respectively.

Original: It can be observed that the models trained with vanilla compatibility regularization ( i.e. , “vanilla”) suffer from the model regression signiﬁcantly.
Revised: It can be observed that the models trained with vanilla compatibility regularization ( i.e. , “vanilla”) suffer from the model regression significantly.

Original: In contrast, the models trained with our regression-alleviating compatibility regularization ( i.e. , “ours”) mitigate the model regression to some extent by properly reducing the negative ﬂips.
Revised: In contrast, the models trained with our regression-alleviating compatibility regularization ( i.e. , “ours”) mitigate the model regression to some extent by properly reducing the negative flips.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. U NCERTAINTY - BASED B ACKFILLING
Revised: 5. U NCERTAINTY - BASED B ACKFILLING

Original: The uncertainty-based backﬁlling strategy is proposed to refresh the gallery following “the-poorﬁrst” priority, in order to fasten the accuracy improvements of the image retrieval during hot-refresh upgrading.
Revised: The uncertainty-based backfilling strategy is proposed to refresh the gallery following “the-poorfirst” priority, in order to fasten the accuracy improvements of the image retrieval during hot-refresh upgrading.

Original: The uncertainty-based backﬁlling is plug-and-play with different models.
Revised: The uncertainty-based backfilling is plug-and-play with different models.

Original: We discuss different backﬁlling strategies on ROxford, as shown in Figure 6, and draw the following conclusions: (1) The Least Conﬁdence and Margin of Conﬁdence generally achieve better performance than Entropy-based uncertainty, indicating that the probabilities of top- 2 conﬁdent classes are more important than the overall classes; (2) The uncertainty-based backﬁlling strategy can fasten the mAP convergence on top of either vanilla method or our RACT method; (3) Since the uncertainty-based backﬁlling strategy attempts to refresh the poor features (which show a higher risk for negative ﬂips) in priority, it can indirectly alleviate the model regression problem.
Revised: We discuss different backfilling strategies on ROxford, as shown in Figure 6, and draw the following conclusions: (1) The Least Confidence and Margin of Confidence generally achieve better performance than Entropy-based uncertainty, indicating that the probabilities of top- 2 confident classes are more important than the overall classes; (2) The uncertainty-based backfilling strategy can fasten the mAP convergence on top of either vanilla method or our RACT method; (3) Since the uncertainty-based backfilling strategy attempts to refresh the poor features (which show a higher risk for negative flips) in priority, it can indirectly alleviate the model regression problem.

Original: How to achieve faster and smoother mAP improvements during backﬁlling is a new task, and ﬁrst investigated in this paper.
Revised: How to achieve faster and smoother mAP improvements during backfilling is a new task, and first investigated in this paper.

Original: 5. S EQUENTIAL MODEL UPGRADES
Revised: 5. S EQUENTIAL MODEL UPGRADES

Original: In real-world applications, model upgrades are often conducted sequentially to achieve continuous improvements.
Revised: In real-world applications, model upgrades are often conducted sequentially to achieve continuous improvements.

Original: To simulate such a scenario, we split the training data into 30%, 60%, 80%, and 100% for training the old model, the upgraded model of the 1st generation, the 2nd generation, and the 3rd generation, respectively.
Revised: To simulate such a scenario, we split the training data into 30%, 60%, 80%, and 100% for training the old model, the upgraded model of the 1st generation, the 2nd generation, and the 3rd generation, respectively.

Original: As shown in Figure 7, our method can achieve consistent improvements over the vanilla baseline in terms of hot-refresh model upgrades (solid line).
Revised: As shown in Figure 7, our method can achieve consistent improvements over the vanilla baseline in terms of hot-refresh model upgrades (solid line).

Original: Another kind of model upgrading mechanism (dotted line) introduced by (Shen et al., 2020), denoted as “no-
Revised: Another kind of model upgrading mechanism (dotted line) introduced by (Shen et al., 2020), denoted as “norefresh” model upgrades, deploys the compatible new model without refreshing the gallery features.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: refresh” model upgrades, deploys the compatible new model without refreshing the gallery features.

Original: We observe the considerable performances gaps between our hot-refresh model upgrades and the no-refresh model upgrades.
Revised: We observe the considerable performances gaps between our hot-refresh model upgrades and the no-refresh model upgrades.

Original: It is actually due to the fact that the poor discriminativeness of the old gallery features highly limits the retrieval accuracy of the new model in no-refresh model upgrades
Revised: It is actually due to the fact that the poor discriminativeness of the old gallery features highly limits the retrieval accuracy of the new model in no-refresh model upgrades

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: This paper for the ﬁrst time studies the issue of model regression during the hot-refresh model upgrades for image retrieval systems.
Revised: This paper for the first time studies the issue of model regression during the hot-refresh model upgrades for image retrieval systems.

Original: We hope it can arouse more public attention to the bottleneck of efﬁcient model upgrades in industrial applications.
Revised: We hope it can arouse more public attention to the bottleneck of efficient model upgrades in industrial applications.

Original: Despite that the introduced regression-alleviating compatibility regularization can properly reduce the negative ﬂips to some extent, there is still a long way to go for completely wiping out the problem of model regression.
Revised: Despite that the introduced regression-alleviating compatibility regularization can properly reduce the negative flips to some extent, there is still a long way to go for completely wiping out the problem of model regression.

Original: Speciﬁcally, several open questions are worth investigating.
Revised: Specifically, several open questions are worth investigating.

Original: (1) In this paper, we only consider the negative ﬂips during the gallery backﬁlling procedure, however, there may also exist negative ﬂips after the backﬁlling is ﬁnished.
Revised: (1) In this paper, we only consider the negative flips during the gallery backfilling procedure, however, there may also exist negative flips after the backfilling is finished.

Original: (2) We investigate regression-alleviating regularization in the form of a plain contrastive loss.
Revised: (2) We investigate regression-alleviating regularization in the form of a plain contrastive loss.

Original: It may further help if integrating the regression-alleviating regularization into more advanced backward-compatible losses.
Revised: It may further help if integrating the regression-alleviating regularization into more advanced backward-compatible losses.

Original: (3) We introduce to backﬁll the gallery in “the-poor-ﬁrst” priority with uncertainty estimation.
Revised: (3) We introduce to backfill the gallery in “the-poor-first” priority with uncertainty estimation.

Original: It is also a new task for model upgrades.
Revised: It is also a new task for model upgrades.

Original: Future works are required for achieving faster and more stable accuracy improvements.
Revised: Future works are required for achieving faster and more stable accuracy improvements.

Original: We would also like to discuss the limitations of the introduced method.
Revised: We would also like to discuss the limitations of the introduced method.

Original: The raw images of the gallery need to be stored for supporting the hot-refresh model upgrades, which may not be satisﬁed due to privacy issues.
Revised: The raw images of the gallery need to be stored for supporting the hot-refresh model upgrades, which may not be satisfied due to privacy issues.

Original: We considered forward-compatible learning (Meng et al., 2021;
Revised: We considered forward-compatible learning (Meng et al., 2021;

Original: Wang et al., 2020) as a potential solution for dealing with this limitation, which is expected to upgrade the gallery features with the input of only the old features rather than raw images.
Revised: Wang et al., 2020) as a potential solution for dealing with this limitation, which is expected to upgrade the gallery features with the input of only the old features rather than raw images.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We demonstrate the core of the algorithm in the form of pseudo codes, as shown in Alg. 1 and 2.
Revised: We demonstrate the core of the algorithm in the form of pseudo codes, as shown in Alg. 1 and 2.

Original: The training details and hyper-parameters can be found at Appendix A.1.
Revised: The training details and hyper-parameters can be found at Appendix A.1.

Original: The datasets are all publicly available and the training data splits are carefully illustrated in Table 1.
Revised: The datasets are all publicly available and the training data splits are carefully illustrated in Table 1.

Original: The full code will be released upon acceptance.
Revised: The full code will be released upon acceptance

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: A CKNOWLEDGEMENT

Revised: This work was supported by NSFC project Grant No.

Revised: U1833101, SZSTI Grant No.

Revised: JCYJ20190809172201639 and WDZC20200820200655001, the Joint Research Center of Tencent and Tsinghua.

Original: A PPENDIX A. T RAINING D ETAILS
Revised: A. T RAINING D ETAILS

Original: The networks are pre-trained on the ImageNet dataset (Deng et al., 2009).
Revised: The networks are pre-trained on the ImageNet dataset (Deng et al., 2009).

Original: The images are resized to 224 × 224 for both training and testing.
Revised: The images are resized to 224 × 224 for both training and testing.

Original: During training, random data augmentation is applied to each image before it is fed into the network, including randomly ﬂipping and cropping.
Revised: During training, random data augmentation is applied to each image before it is fed into the network, including randomly flipping and cropping.

Original: We adopt 6 Tesla V100 GPUs for training, and the batch size per GPU is set to 80 for ResNet-50 and 64 for
Revised: We adopt 6 Tesla V100 GPUs for training, and the batch size per GPU is set to 80 for ResNet-50 and 64 for

Original: ResNet-101.
Revised: ResNet-101.

Original: For all the experiments, Adam optimizer is adopted to optimize the training model with a weight decay of 0.0001.
Revised: For all the experiments, Adam optimizer is adopted to optimize the training model with a weight decay of 0.0001.

Original: The initial learning rate is set to 0.01 and is decreased to 1/10 of its previous value every 30 epochs in the total 90 epochs.
Revised: The initial learning rate is set to 0.01 and is decreased to 1/10 of its previous value every 30 epochs in the total 90 epochs.

Original: The old model is trained with only a classiﬁcation loss in Eq.
Revised: The old model is trained with only a classification loss in Eq.

Original: (7), and the new model is trained towards the objective in Eq.
Revised: (7), and the new model is trained towards the objective in Eq.

Original: The temperature τ in Eq.
Revised: The temperature τ in Eq.

Original: (6) is empirically set as 0.05, and the loss weight λ in Eq.
Revised: (6) is empirically set as 0.05, and the loss weight λ in Eq.

Original: (8) is set as 1.0.
Revised: (8) is set as 1.0.

Original: [Figure]
Revised: [Figure]

Original: We observe that, despite the overall accuracy ( i.e. , mAP) has gradually increased without regression, vanilla shows signiﬁcantly larger NFR than ours.
Revised: We observe that, despite the overall accuracy ( i.e. , mAP) has gradually increased without regression, vanilla shows significantly larger NFR than ours.

Original: Noted that even if the overall mAP has increased, every new mistake on the old positive queries feels like a step backwards, still leading to a perceived regression in user experience.
Revised: Noted that even if the overall mAP has increased, every new mistake on the old positive queries feels like a step backwards, still leading to a perceived regression in user experience.

Original: Reducing NFR can be as important as improving mAP in practice.
Revised: Reducing NFR can be as important as improving mAP in practice.

Original: [Figure]
Revised: [Figure]

Original: A. M ODEL U PGRADES OF THE S AME A RCHITECTURE
Revised: A. M ODEL U PGRADES OF THE S AME A RCHITECTURE

Original: Regression-alleviating Compatible Training.
Revised: Regression-alleviating Compatible Training.

Original: We further evaluate our method when the new model has the same architecture as the old model, e.g. , ResNet-50 for both old and new models.
Revised: We further evaluate our method when the new model has the same architecture as the old model, e.g. , ResNet-50 for both old and new models.

Original: As illustrated in Figure 8, the retrieval system with vanilla models is less susceptible to the model regression problem compared to the R50-R101 setup (Figure 3) when testing on GLDv2-test.
Revised: As illustrated in Figure 8, the retrieval system with vanilla models is less susceptible to the model regression problem compared to the R50-R101 setup (Figure 3) when testing on GLDv2-test.

Original: This is because that old and new models of the same architecture start from the same weights, i.e. , ImageNet pre-training.
Revised: This is because that old and new models of the same architecture start from the same weights, i.e. , ImageNet pre-training.

Original: They have natural prediction consistency in the beginning, making the feature
Revised: They have natural prediction consistency in the beginning, making the feature

Original: [Figure]
Revised: [Figure]

Original: vanilla
Revised: vanilla

Original: [Figure]
Revised: [Figure]

Original: ours

Original: NFR of ours
Revised: NFR of ours

Original: [Figure]
Revised: [Figure]

Original: old model 83.

Original: [Figure]
Revised: [Figure]

Original: due compatibility easier.
Revised: compatibility easier.

Original: Even so, we argue that vanilla models show evident larger NFR than our model, indicating an underlying model regression.
Revised: Even so, we argue that vanilla models show evident larger NFR than our model, indicating an underlying model regression.

Original: The results of ROxford and RParis are in Figures 9, 10.
Revised: The results of ROxford and RParis are in Figures 9, 10.

Original: Uncertainty-based Backﬁlling.
Revised: Uncertainty-based Backfilling.

Original: We investigate the uncertainty-based backﬁlling on the setup of
Revised: We investigate the uncertainty-based backfilling on the setup of

Original: R50-R50, as shown in Figure 11.
Revised: R50-R50, as shown in Figure 11.

Original: Backﬁlling the gallery with uncertainty-based priority can effectively fasten the performance improvements of image retrieval systems, as well as implicitly alleviate the model regression.
Revised: Backfilling the gallery with uncertainty-based priority can effectively fasten the performance improvements of image retrieval systems, as well as implicitly alleviate the model regression.

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. A DDITIONAL E XPERIMENTS
Revised: A DDITIONAL E XPERIMENTS

Original: Analysis of Uncertainty-based Backﬁlling.
Revised: Analysis of Uncertainty-based Backfilling.

Original: We further investigate the uncertainty-based backﬁlling with Margin of Conﬁdence metric on RParis and GLDv2-test, as shown in Figure 12, where the uncertainty-based backﬁlling generally achieves faster mAP convergence with either the vanilla models or our models.
Revised: We further investigate the uncertainty-based backfilling with Margin of Confidence metric on RParis and GLDv2-test, as shown in Figure 12, where the uncertainty-based backfilling generally achieves faster mAP convergence with either the vanilla models or our models.

Original: Similar conclusions with the ROxford (discussed in Section 5.3) can be drawn, i.e. , the uncertainty-based backﬁlling strategy can indirectly alleviate the model regression problem since the poor features with a higher risk for negative ﬂips are refreshed in priority.
Revised: Similar conclusions with the ROxford (discussed in Section 5.3) can be drawn, i.e. , the uncertainty-based backfilling strategy can indirectly alleviate the model regression problem since the poor features with a higher risk for negative flips are refreshed in priority.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: However, we still face some failure cases with the backﬁlling strategy, as shown in Figure 13.
Revised: However, we still face some failure cases with the backfilling strategy, as shown in Figure 13.

Original: We argue that the uncertainty-based strategy may not work when the new and old features are not compatible enough, since we use the trained and ﬁxed new classiﬁer to produce class probabilities for the old gallery features under the assumption that they lay on the almost same latent space.
Revised: We argue that the uncertainty-based strategy may not work when the new and old features are not compatible enough, since we use the trained and fixed new classifier to produce class probabilities for the old gallery features under the assumption that they lay on the almost same latent space.

Original: To further investigate the phenomenon of Figure 13, we observe that the new-to-old performance (0%) is inferior to the old-to-old performance (brown star) in Figure 5(b) of the main paper, showing the unsatisfactory feature compatibility.
Revised: To further investigate the phenomenon of Figure 13, we observe that the new-to-old performance (0%) is inferior to the old-to-old performance (brown star) in Figure 5(b) of the main paper, showing the unsatisfactory feature compatibility.

Original: More advanced compatible constraints have the potential to solve this issue but is not the core of this paper.
Revised: More advanced compatible constraints have the potential to solve this issue but is not the core of this paper.

Original: The introduced uncertainty-based backﬁlling method is simple but consistently effective in most cases.
Revised: The introduced uncertainty-based backfilling method is simple but consistently effective in most cases.

Original: We acknowledge that the method is not perfect enough to handle all the deployment scenarios, i.e. , it fails when the feature compatibility is not good enough (generally when the new-to-old retrieval performance is lower than old-to-old retrieval performance).
Revised: We acknowledge that the method is not perfect enough to handle all the deployment scenarios, i.e. , it fails when the feature compatibility is not good enough (generally when the new-to-old retrieval performance is lower than old-to-old retrieval performance).

Original: However, we give the ﬁrst attempt on a totally new task, showcasing the great potential for further work that incorporates model upgrades with ordered gallery images.
Revised: However, we give the first attempt on a totally new task, showcasing the great potential for further work that incorporates model upgrades with ordered gallery images.

Original: Compatibility Evaluation.
Revised: Compatibility Evaluation.

Original: The evaluation of compatibility can be observed at 0% in all the ﬁgures and tables for model upgrades.
Revised: The evaluation of compatibility can be observed at 0% in all the figures and tables for model upgrades.

Original: Speciﬁcally, at 0% of model upgrades, the new features of queries need to be directly indexed by the old gallery feature, which is an ordinary benchmark for conventional compatible learning (Shen et al., 2020).
Revised: Specifically, at 0% of model upgrades, the new features of queries need to be directly indexed by the old gallery feature, which is an ordinary benchmark for conventional compatible learning (Shen et al., 2020).

Original: To make it clear [Table] we listed the above results in
Revised: To make it clear, we listed the above results in

Original: Table 2.
Revised: Table 2.

Original: We can observe that our method can achieve better compatible performance than the vanilla baseline, indicating that we do not sacriﬁce the compatibility for reducing negative ﬂips
Revised: We can observe that our method can achieve better compatible performance than the vanilla baseline, indicating that we do not sacrifice the compatibility for reducing negative flips.

Original: [Table]
Revised: [Table]

Original: A. A BLATION S TUDIES
Revised: A. A BLATION S TUDIES

Original: Effects of temperature τ .
Revised: Effects of temperature τ .

Original: The temperature τ is empirically set as 0.05 in this work.
Revised: The temperature τ is empirically set as 0.05 in this work.

Original: We discuss different values of τ in Table 3.
Revised: We discuss different values of τ in Table 3.

Original: A similar increasing trend of retrieval performance upgrades can be observed when τ varies from 0.05 to 0.1, while model regression at 20% is observed when τ = 0 .
Revised: A similar increasing trend of retrieval performance upgrades can be observed when τ varies from 0.05 to 0.1, while model regression at 20% is observed when τ = 0 .

Original: The 100% new-to-new performances are sub-optimal when τ = 0 .
Revised: The 100% new-to-new performances are sub-optimal when τ = 0 .

Original: 1 and τ = 0 .
Revised: 1 and τ = 0 .

Original: In all, 0.05 is a suitable value for the contrastive temperature, and all the experiments in this paper adopt the same temperature value for a fair comparison.
Revised: In all, 0.05 is a suitable value for the contrastive temperature, and all the experiments in this paper adopt the same temperature value for a fair comparison.

Original: [Table]
Revised: [Table]

Original: Effects of loss weight λ .
Revised: Effects of loss weight λ .

Original: The loss weight λ is set as 1.0 in this work for simplicity.
Revised: The loss weight λ is set as 1.0 in this work for simplicity.

Original: We study the effects of different λ in Table 4.
Revised: We study the effects of different λ in Table 4.

Original: We can observe that the value of λ controls the trade-off between new feature discriminativeness and new-to-old feature compatibility, i
Revised: We can observe that the value of λ controls the trade-off between new feature discriminativeness and new-to-old feature compatibility, i

Original: [Table]
Revised: [Table]

Original: e. , models with smaller λ achieve better performance at 100% while models with larger λ achieve better performance at 0%.
Revised: e. , models with smaller λ achieve better performance at 100% while models with larger λ achieve better performance at 0%.

Original: Despite the start and end points vary from different models, a similar increasing trend of the performances during backﬁlling can be observed.
Revised: Despite the start and end points vary from different models, a similar increasing trend of the performances during backfilling can be observed.

Original: [Table]
Revised: [Table]

Original: Effects of batch size
Revised: Effects of batch size.

Original: We use a batch size of 80 per GPU for the experiments with ResNet-50.
Revised: We use a batch size of 80 per GPU for the experiments with ResNet-50.

Original: As the batch size may affect the number of negative samples for contrastive loss, we conduct ablation studies by modifying the batch size, as illustrated in Table 5.
Revised: As the batch size may affect the number of negative samples for contrastive loss, we conduct ablation studies by modifying the batch size, as illustrated in Table 5.

Original: We observe that our method is robust to the number of batch size.
Revised: We observe that our method is robust to the number of batch size.

Original: [Table]
Revised: [Table]

Original: The results are reported on GLDv2-test with the architectures of R50-R50.
Revised: The results are reported on GLDv2-test with the architectures of R50-

Original: We fail to try larger batchsizes to
Revised: We fail to try larger batch

Original: Effects of positive sampler.
Revised: Effects of positive sampler.

Original: As mentioned in the method, for more efﬁcient training, we treat the compatible training as an instance discrimination-like task when training with the large-scale retrieval dataset, i.e. , GLDv2.
Revised: As mentioned in the method, for more efficient training, we treat the compatible training as an instance discrimination-like task when training with the large-scale retrieval dataset, i [Table] e. , GLDv2.

Original: Speciﬁcally, the positive pair consists of the new and old features of the same instance.
Revised: Specifically, the positive pair consists of the new and old features of the same instance.

Original: To verify that the mentioned instance discrimination-like compatible training does not harm the retrieval performance, we conduct an ablation study by using a positive presampler to randomly sample an intra-class sample for each anchor before each iteration.
Revised: To verify that the mentioned instance discrimination-like compatible training does not harm the retrieval performance, we conduct an ablation study by using a positive presampler to randomly sample an intra-class sample for each anchor before each iteration.

Original: As shown in Table 6, similar results can be achieved by using either the positive pre-sampler or not.
Revised: As shown in Table 6, similar results can be achieved by using either the positive pre-sampler or not.

Original: The reason for not seeing an improvement with the positive sampler might be that, in the task of compatible
Revised: The reason for not seeing an improvement with the positive sampler might be that, in the task of compatible

Original: [Table]
Revised: [Table]

Original: training, the most signiﬁcant objective is to train the new model to map images to same features as the old model, rather than constraining the distance between positive pairs.
Revised: training, the most significant objective is to train the new model to map images to same features as the old model, rather than constraining the distance between positive pairs.

Original: Such a phenomenon has also been observed by the related work (Budnik & Avrithis, 2020).
Revised: Such a phenomenon has also been observed by the related work (Budnik & Avrithis, 2020).

Original: As shown in Table 3 of (Budnik & Avrithis, 2020), the method of “regression” (forces the old feature and new feature of the same image similar using cosine similarity) wins the asymmetric test (cross-model test), compared to other methods that regularize the positive pairs.
Revised: As shown in Table 3 of (Budnik & Avrithis, 2020), the method of “regression” (forces the old feature and new feature of the same image similar using cosine similarity) wins the asymmetric test (cross-model test), compared to other methods that regularize the positive pairs.

Original: Moreover, the training dataset, GLDv2, consists of 1,580,470 samples with 81,313 classes, i.e. , 19 samples per class, showing limited intra-class variations, which might also lead to similar results with or without such a positive sampler.
Revised: Moreover, the training dataset, GLDv2, consists of 1,580,470 samples with 81,313 classes, i.e. , 19 samples per class, showing limited intra-class variations, which might also lead to similar results with or without such a positive sampler

Original: The trade-off between new-to-old negatives and new-to-new negatives in L ra-comp .
Revised: The trade-off between new-to-old negatives and new-to-new negatives in L ra-comp .

Original: We inves- tigate the effects of balancing new-to-old negatives and new-to-new negatives in L ra-comp .
Revised: We inves- tigate the effects of balancing new-to-old negatives and new-to-new negatives in L ra-comp .

Original: Since it does not make sense to directly impose weighted factors on the sample distances ( i.e. , (cid:104)· , ·(cid:105) ) in our original loss function (Eq.
Revised: Since it does not make sense to directly impose weighted factors on the sample distances ( i.e. , ⟨· , ·⟩ ) in our original loss function (Eq.

Original: (6)), we ﬁrst split our original uniﬁed contrastive loss into a combination of two terms, formulated as
Revised: , we first split our original unified contrastive loss into a combination of two terms, formulated as

Original: [Equation]
Revised: [Equation]

Original: where η is a weighting factor.
Revised: where η is a weighting factor.

Original: When setting η = 1 , similar results can be observed as the original setup.
Revised: When setting η = 1 , similar results can be observed as the original setup.

Original: When changing η from 0.2 to 5.0, we ﬁnd that a larger η may lead to better alleviation of model regression but lower new-to-old compatible performance (0%).
Revised: When changing η from 0.2 to 5.0, we find that a larger η may lead to better alleviation of model regression but lower new-to-old compatible performance (0%).

Original: We ﬁnd η = 1 . 

Revised: We find η = 1 .

Original: a better trade-off between feature compatibility and alleviating model regression, showing that our uniﬁed version (Eq.
Revised: a better trade-off between feature compatibility and alleviating model regression, showing that our unified version (Eq.

Original: (6)) of contrastive loss is reasonable.
Revised: (6)) of contrastive loss is reasonable.

Original: Detailed results are illustrated in Figure 14.
Revised: Detailed results are illustrated in Figure 14.

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Equation]

Original: [Table] [Equation] [Table] [Figure]
Revised: -R50) on ROxford in terms of mAP( ↑ ).

Original: Different temperatures for new-to-old negatives and new-to-new negatives in L ra-comp .
Revised: Different temperatures for new-to-old negatives and new-to-new negatives in L ra-comp .

Original: We empirically set the temperature to 0.05.
Revised: We empirically set the temperature to 0.05.

Original: To further analyze the effects of different temperatures in two types of negative pairs, we conduct experiments as shown in Table 7.
Revised: To further analyze the effects of different temperatures in two types of negative pairs, we conduct experiments as shown in Table 7.

Original: Fewer effects can be observed when changing the temperature, so we use 0.05 for both for brevity.
Revised: Fewer effects can be observed when changing the temperature, so we use 0.05 for both for brevity.

Original: An alternative form of compatible regularizations - triplet loss.
Revised: An alternative form of compatible regularizations - triplet loss.

Original: In the main paper, we introduce to regularize the compatibility of the learned representations in the form of a contrastive loss.An [Figure] [Table]
Revised: In the main paper, we introduce to regularize the compatibility of the learned representations in the form of a contrastive loss.

Original: old

Original: [Table]

Original: [Table]

Original: [Table] Sampler old [Table] alternative option for formulating the regularization is in the form of a triplet loss, where several triplets combining positives and negatives from old and new models are considered at the same time.
Revised: An alternative option for formulating the regularization is in the form of a triplet loss, where several triplets combining positives and negatives from old and new models are considered at the same time.

Original: Speciﬁcally, the vanilla version of backward-compatible regularization is formulated as,
Revised: Specifically, the vanilla version of backward-compatible regularization is formulated as,

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Original:  where the margin m is empirically set to 0 .
Revised: [Table] Sampler [Table] The results are where the margin m is empirically set to 0 .

Original: 8 and p ( x ) denotes the set of intra-class samples for x (including x ) in the mini-batch.
Revised: 8 and p ( x ) denotes the set of intra-class samples for x (including x ) in the mini-batch.

Original: The regression-alleviating compatibility regularization therefore becomes
Revised: The regression-alleviating compatibility regularization therefore becomes

Original: [Equation]
Revised: [Equation]

Original: The comparisons between vanilla method and our regression-alleviating method in terms of openclass data split on three different datasets can be found in Figure 15.
Revised: The comparisons between vanilla method and our regression-alleviating method in terms of openclass data split on three different datasets can be found in Figure 15.

Original: We observe that our method can properly reduce negative ﬂips on GLDv2 and RParis datasets, but fail in ROxford.
Revised: We observe that our method can properly reduce negative flips on GLDv2 and RParis datasets, but fail in ROxford.

Original: Both vanilla and ours show inferior performance than the contrastive counterparts, since the uniﬁed form of a contrastive loss can establish a global structure among new-to-old and new-to-new pairs while dividing them into several triplets cannot.
Revised: Both vanilla and ours show inferior performance than the contrastive counterparts, since the unified form of a contrastive loss can establish a global structure among new-to-old and new-to-new pairs while dividing them into several triplets cannot.

Original: We also investigate the effects of a positive pre-sampler when conducting experiments with triplet loss.
Revised: We also investigate the effects of a positive pre-sampler when conducting experiments with triplet loss.

Original: As demonstrated in Table 8, similar results can be found with or without such a sampler, drawing the same conclusion with the experiments of contrastive loss.
Revised: As demonstrated in Table 8, similar results can be found with or without such a sampler, drawing the same conclusion with the experiments of contrastive loss.

Original: A. R ELEVANT T OPICS
Revised: A. R ELEVANT T OPICS

Original: We discuss three relevant topics that also need an old model for training the new model, including compatible learning (Shen et al., 2020;
Revised: We discuss three relevant topics that also need an old model for training the new model, including compatible learning (Shen et al., 2020;

Original: Meng et al., 2021; Budnik & Avrithis, 2020), knowledge distillation (Hinton et al., 2015; Li et al., 2020a; Yan et al., 2020), and continual learning (Li &
Revised: Meng et al., 2021; Budnik & Avrithis, 2020), knowledge distillation (Hinton et al., 2015; Li et al., 2020a; Yan et al., 2020), and continual learning (Li &

Original: Hoiem, 2017; Wang et al., 2021).
Revised: Hoiem, 2017; Wang et al., 2021).

Original: Compare to Compatible Learning.
Revised: Compare to Compatible Learning.

Original: Conventional compatible learning methods (Shen et al., 2020; Meng et al., 2021; Budnik & Avrithis, 2020) regularize the feature compatibility by enforcing the new-to-old positive pairs to be more similar than new-to-old negative pairs.
Revised: Conventional compatible learning methods (Shen et al., 2020; Meng et al., 2021; Budnik & Avrithis, 2020) regularize the feature compatibility by enforcing the new-to-old positive pairs to be more similar than new-to-old negative pairs.

Original: They did not consider the task of hot-refresh model upgrades thus overlooked the problem of negative ﬂips.
Revised: They did not consider the task of hot-refresh model upgrades thus overlooked the problem of negative flips.

Original: We take the state-of-the-art BCT (Shen et al., 2020) as an example.
Revised: We take the state-of-the-art BCT (Shen et al., 2020) as an example.

Original: The comparisons can be found in
Revised: The comparisons can be found in

Original: Figures 16 and 17.
Revised: Figures 16 and 17.

Original: As it only focuses on the backﬁll-free model upgrades, it cannot alleviate the negative ﬂips, suffering from the problem of model regression.
Revised: As it only focuses on the backfill-free model upgrades, it cannot alleviate the negative flips, suffering from the problem of model regression.

Original: Note that BCT adopts classiﬁcation loss as compatible constraints (via feeding the new features into the old classiﬁer), where it is hard to explicitly integrate the regularization of new-to-new negative pairs into the form of BCT (classiﬁcation) loss.
Revised: Note that BCT adopts classification loss as compatible constraints (via feeding the new features into the old classifier), where it is hard to explicitly integrate the regularization of new-to-new negative pairs into the form of BCT (classification) loss.

Original: So we use our regression-alleviating compatible loss on top of BCT loss, and achieve state-of-the-art compatible performance at the same time alleviating model regression.
Revised: So we use our regression-alleviating compatible loss on top of BCT loss, and achieve state-of-the-art compatible performance at the same time alleviating model regression.

Original: Compare to Knowledge Distillation.
Revised: Compare to Knowledge Distillation.

Original: Although we both regularize the prediction consistency between two different models, we have entirely different purposes.
Revised: Although we both regularize the prediction consistency between two different models, we have entirely different purposes.

Original: Speciﬁcally, distillation aims at training a new model that inherits the knowledge of the teacher (old) model in order to improve the overall accuracy of the new model, and the new model will be deployed individually during inference.
Revised: Specifically, distillation aims at training a new model that inherits the knowledge of the teacher (old) model in order to improve the overall accuracy of the new model, and the new model will be deployed individually during inference.

Original: In contrast, our task requires cross-model retrieval via learning compatible features at the same time reducing negative ﬂips.
Revised: In contrast, our task requires cross-model retrieval via learning compatible features at the same time reducing negative flips.

Original: Well noted that learning new features that are interoperable with old features is not equal to inheriting the knowledge of old features.
Revised: Well noted that learning new features that are interoperable with old features is not equal to inheriting the knowledge of old features.

Original: Moreover, we have different experimental setups.
Revised: Moreover, we have different experimental setups.

Original: In knowledge distillation, the old model is generally a stronger network with more powerful representation discrimination.
Revised: In knowledge distillation, the old model is generally a stronger network with more powerful representation discrimination.

Original: In our model upgrading task, the old model is generally a poorer network with inferior representation discrimination.
Revised: In our model upgrading task, the old model is generally a poorer network with inferior representation discrimination.

Original: To verify that existing distillation-based algorithms are inapplicable for hot-refresh model upgrades of retrieval tasks, we compared to related work, namely Focal Distillation (FD) (Yan et al., 2020), which was specially designed for alleviating the model regression of classiﬁcation tasks via knowledge distillation.
Revised: To verify that existing distillation-based algorithms are inapplicable for hot-refresh model upgrades of retrieval tasks, we compared to related work, namely Focal Distillation (FD) (Yan et al., 2020), which was specially designed for alleviating the model regression of classification tasks via knowledge distillation.

Original: As shown in Figures 16 and 17, directly employing Focal Distillation in our task achieves unsatisfactory retrieval performances and is hard to achieve the goal of model upgrades.
Revised: As shown in Figures 16 and 17, directly employing Focal Distillation in our task achieves unsatisfactory retrieval performances and is hard to achieve the goal of model upgrades.

Original: It is actually due to the fact that it inherits the knowledge of the inferior old model too much, which limits its capability of learning discriminative new features.
Revised: It is actually due to the fact that it inherits the knowledge of the inferior old model too much, which limits its capability of learning discriminative new features.

Original: Compare to Continual Learning.
Revised: Compare to Continual Learning.

Original: Our work is also related to continual learning (Li & Hoiem, 2017; Wang et al., 2021), which aims at training a new model that inherits the capability of the old task.
Revised: Our work is also related to continual learning (Li & Hoiem, 2017; Wang et al., 2021), which aims at training a new model that inherits the capability of the old task.

Original: We mainly differ in two ways.
Revised: We mainly differ in two ways.

Original: (1) Application: Hot-refresh model upgrade requires feature compatibility for cross-modal retrieval, i.e. , the query is embedded by the new model while the gallery is a mix of both new and old features.
Revised: (1) Application: Hot-refresh model upgrade requires feature compatibility for cross-modal retrieval, i.e. , the query is embedded by the new model while the gallery is a mix of both new and old features.

Original: Continual learning requires the new model to remember the capability of the old model task during training, however, the new model is individually deployed without the “interaction” with the old model.
Revised: Continual learning requires the new model to remember the capability of the old model task during training, however, the new model is individually deployed without the “interaction” with the old model.

Original: Moreover, since continual learning does not need to conduct cross-model deployment, there’s no consideration of negative ﬂips.
Revised: Moreover, since continual learning does not need to conduct cross-model deployment, there’s no consideration of negative flips.

Original: (2) Purpose:
Revised: (2) Purpose:

Original: Hot-refresh model upgrade requires the new model to outperform the old model and the new features to be interoperable with old features.
Revised: Hot-refresh model upgrade requires the new model to outperform the old model and the new features to be interoperable with old features.

Original: Continual learning only requires the new model to inherit the old task knowledge, does not expect the new model to outperform the old model on the old task.
Revised: Continual learning only requires the new model to inherit the old task knowledge, does not expect the new model to outperform the old model on the old task.

Original: To verify that existing continual learning algorithms are inapplicable for hot-refresh model upgrades of retrieval tasks, we compared to related work, namely Learning without Forget (LwF) (Li &
Revised: To verify that existing continual learning algorithms are inapplicable for hot-refresh model upgrades of retrieval tasks, we compared to related work, namely Learning without Forget (LwF) (Li &

Original: Hoiem, 2017).
Revised: Hoiem, 2017).

Original: As shown in Figure 18, signiﬁcant model regression and unsatisfactory retrieval performances are observed.
Revised: As shown in Figure 18, significant model regression and unsatisfactory retrieval performances are observed.

Original: Existing continual learning algorithms cannot tackle the model regression problem of hot-refresh model upgrades.
Revised: Existing continual learning algorithms cannot tackle the model regression problem of hot-refresh model upgrades.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: are reported in terms of mAP( ↑ ).
Revised: terms of mAP( ↑ ).