Original: RecursiveMix: Mixed Learning with History
Revised: RecursiveMix: Mixed Learning with History

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Mix-based augmentation has been proven fundamental to the generalization of deepvision models.
Revised: Mix-based augmentation has been proven fundamental to the generalization of deep vision models.

Original: However, current augmentations only mix samples from the currentdata batch during training, which ignores the possible knowledge accumulated inthe learning history.
Revised: However, current augmentations only mix samples from the current data batch during training, which ignores the possible knowledge accumulated in the learning history.

Original: In this paper, we propose a recursive mixed-sample learningparadigm, termed “RecursiveMix” (RM), by exploring a novel training strategythat leverages the historical input-prediction-label triplets.
Revised: In this paper, we propose a recursive mixed-sample learning paradigm, termed “RecursiveMix” (RM), by exploring a novel training strategy that leverages the historical input-prediction-label triplets.

Original: More specifically, weiteratively resize the input image batch from the previous iteration and paste itinto the current batch while their labels are fused proportionally to the area ofthe operated patches.
Revised: More specifically, we iteratively resize the input image batch from the previous iteration and paste it into the current batch while their labels are fused proportionally to the area of the operated patches.

Original: Furthermore, a consistency loss is introduced to align theidentical image semantics across the iterations, which helps the learning of scaleinvariant feature representations.
Revised: Furthermore, a consistency loss is introduced to align the identical image semantics across the iterations, which helps the learning of scaleinvariant feature representations.

Original: Based on ResNet-50, RM largely improvesclassification accuracy by ∼ 3.2% on CIFAR-100 and ∼ 2.8% on ImageNet withnegligible extra computation/storage costs.
Revised: Based on ResNet-50, RM largely improves classification accuracy by ∼ 3.2% on CIFAR-100 and ∼ 2.8% on ImageNet with negligible extra computation/storage costs.

Original: In the downstream object detectiontask, the RM pretrained model outperforms the baseline by 2.1 AP points andsurpasses CutMix by 1.4 AP points under the ATSS detector on COCO.
Revised: In the downstream object detection task, the RM-pretrained model outperforms the baseline by 2.1 AP points and surpasses CutMix by 1.4 AP points under the ATSS detector on COCO.

Original: In semanticsegmentation, RM also surpasses the baseline and CutMix by 1.9 and 1.1 mIoUpoints under UperNet on ADE20K, respectively.
Revised: In semantic segmentation, RM also surpasses the baseline and CutMix by 1.9 and 1.1 mIoU points under UperNet on ADE20K, respectively.

Revised: Codes and pretrained models are available at https://github.com/implus/RecursiveMix .

Original:  Introduction
Revised:  Introduction

Original: Deep convolutional neural networks (CNN) have made great progress in many computer vision taskssuch as image classification [26, 29, 60], object detection [25, 69, 40], semantic segmentation [70, 65],etc.
Revised: Deep convolutional neural networks (CNN) have made great progress in many computer vision tasks such as image classification [26, 29, 62], object detection [25, 71, 42], semantic segmentation [72, 67], etc.

Original: Despite achieving better performances, the models inevitably become more complex andincur over-fitting risks.
Revised: Despite achieving better performances, the models inevitably become more complex and incur over-fitting risks.

Original: To overcome this problem, methods like regularization [51, 30] and dataaugmentation [15, 68, 66] are developed consequently.
Revised: To overcome this problem, methods like regularization [53, 30] and data augmentation [15, 70, 68] are developed consequently.

Original: Recently, mixed-based data augmentations [68,66, 57, 58, 34, 13] have been proven to enhance the model generalization capability.
Revised: Recently, mixed-based data augmentations [70, 68, 59, 60, 34, 13] have been proven to enhance the model generalization capability.

Original: Following thepopular Mixup [68] and CutMix [66] which optimize networks based on mixed pixels and fusedlabels of two images, improvement has been made on aspects of the mix level [57, 18, 39] and mixregion [34, 13, 58].
Revised: Following the popular Mixup [70] and CutMix [68], which optimize networks based on mixed pixels and fused labels of two images, improvement has been made on aspects of the mix level [59, 18, 39] and mix region [34, 13, 60].

Original: However, existing methods only mix samples from the current data batch.
Revised: However, existing methods only mix samples from the current data batch.

Original: As aresult, they fail to capitalize on the potential knowledge in their learning history.
Revised: As a result, they fail to capitalize on the potential knowledge in their learning history.

Original: In order to leverage the historical knowledge, we propose an efficient and effective augmentationapproach termed “RecursiveMix” (RM) that makes use of the historical input-prediction-label triplets.
Revised: In order to leverage the historical knowledge, we propose an efficient and effective augmentation approach termed “RecursiveMix” (RM) that makes use of the historical input-prediction-label triplets.

Original: Different from the conventional mix-based approaches which only fuse the training images in thecurrent batch, RM further leverages the last mixed inputs and labels to form new mixed ones ina recursive paradigm.
Revised: Different from the conventional mix-based approaches which only fuse the training images in the current batch, RM further leverages the last mixed inputs and labels to form new mixed ones in a recursive paradigm.

Original: It aims at continuously reusing the augmented data in previous iterationsalong with its augmented supervisions (Fig. 1). In addition, a consistency loss is introduced to alignthe identical spatial semantics between the corresponding regions of interest (RoI) in new mixedinputs and the entire historical inputs in the last iteration. As illustrated in Fig. 2, thanks to therecursive design via historical input-prediction-label triplets, RM has three obvious advantages overits competitive counterparts [66, 68]: it 1) introduces more data diversity by considerably enlargingthe input data space; 2) can explicitly and sufficiently explore the multi-scale/-space views of eachinstance and 3) aptly leverages the identical semantic relationship between two inputs across iterations,which enables the network to learn the space-aware semantic representations explicitly.
Revised: It aims at continuously reusing the augmented data in previous iterations along

Original: Further,

Original: Iteration 𝑡Iteration 𝑡 +
Revised: Iteration 𝑡Iteration 𝑡 +

Original: Model
Revised: Model

Original: Model
Revised: Model

Original: Model 𝝀 𝒕 𝟏−𝝀 𝒕+ 𝝀 𝒕 𝝀 𝒕+ 𝟏−𝝀 𝒕
Revised: Model 𝝀 𝒕 𝟏−𝝀 𝒕+ 𝝀 𝒕 𝝀 𝒕+ 𝟏−𝝀 𝒕

Original: [Figure]
Revised: [Figure]

Original: 𝝀 𝒕+ …… ……
Revised: 𝝀 𝒕+ …… ……

Original: Iteration 𝑡
Revised: Iteration 𝑡

Original: Supervise ResizeFill
Revised: Supervise ResizeFill

Original: Iteration 𝑡 +
Revised: Iteration 𝑡 +

Original: Model 𝟏−𝝀 𝒕+ 𝝀 𝒕+ 𝝀 𝒕+
Revised: Model 𝟏−𝝀 𝒕+ 𝝀 𝒕+ 𝝀 𝒕+

Original: One-hot Label 𝑦 𝑡−1 Cat
Revised: One-hot Label 𝑦 𝑡−1 Cat

Original: One-hot Label 𝑦 𝑡 Dog
Revised: One-hot Label 𝑦 𝑡 Dog

Original: One-hot Label 𝑦 𝑡+1 Bird
Revised: One-hot Label 𝑦 𝑡+1 Bird

Original: One-hot Label 𝑦 𝑡+2 Horse ……
Revised: One-hot Label 𝑦 𝑡+2 Horse ……

Original: [Figure] Prediction
Revised: Historical Prediction

Original: KL divergence KL divergence
Revised: KL divergence KL divergence

Original: KL divergence
Revised: KL divergence

Original: GAP GAP RoIAlign
Revised: GAP GAP RoIAlign

Original: RoIAlign
Revised: RoIAlign

Original: [Figure]
Revised: GAP RoIAlign GAP ℋ ℋ ℋ ℋ ℋ′ ℋ′ ℋ′ with its augmented supervisions (Fig. 1). In addition, a consistency loss is introduced to align two spatial representations derived from the identical instance.

Original: ℋ ℋ ℋ ℋ ℋ′ ℋ′ ℋ′ the space-aware features can potentially benefit the downstream tasks such as object detection andsegmentation as they are more sensitive to the spatial understanding of the image.
Revised: RM shows significant advantages over its competitive counterparts [68, 70] on classification tasks and its space-aware features can potentially benefit the downstream tasks such as object detection and segmentation as they are more sensitive to the spatial understanding of the image.

Original: The advancedperformance of RM will be illustrated later in our  experiments.
Revised: The advanced performance of RM will be illustrated later in our methods and experiments.

Original: [Figure]
Revised: [Figure]

Original: Another notable property of the proposed RM is that ithardly increases the training/inference budget (Table 11)but only consumes an extremely negligible amount of storage complexity (only a mini-batch of the historical input,prediction, and label) and a lightweight additional module(only an RoI [25] operator and an optional fully connectedlayer).
Revised: Another notable property of the proposed RM is that it hardly increases the training/inference budget (Table 11) but only consumes an extremely negligible amount of storage complexity (only a mini-batch of the historical input, prediction, and label) and a lightweight additional module (only an RoI [25] operator and an optional fully connected layer).

Original: Experimental evidence in CIFAR and ImageNetclassification benchmarks demonstrate its effectiveness.
Revised: Experimental evidence in CIFAR and ImageNet classification benchmarks demonstrate its effectiveness.

Original: Specifically, RM shows a gain of absolutely 3.2% pointsover the baseline model and outperforms CutMix [66] by1.1% points in the CIFAR-100 dataset under DenseNet161 [29].
Revised: Specifically, RM shows a gain of absolutely 3.2% points over the baseline model and outperforms CutMix [68] by 1.1% points in the CIFAR-100 dataset under DenseNet161 [29].

Original: In the ImageNet dataset, RM improves by 2.9%and 2.7% points based on the ResNet-50 [26] and PVTv2B1 [59] baselines while outperforming CutMix by 0.6%and 0.7% points, respectively.
Revised: In the ImageNet dataset, RM improves by 2.9% and 2.7% points based on the ResNet-50 [26] and PVTv2B1 [61] baselines while outperforming CutMix by 0.6% and 0.7% points, respectively.

Original: In terms of the downstreamtasks, RM pretrained models improve the baseline by 1 ∼ 3box AP points (object detection) and 1 ∼ 2 mask AP points(instance segmentation) on the COCO dataset and 1 ∼ 3mIoU (semantic segmentation) on ADE20K while consistently outperforming models pretrained by CutMix.
Revised: In terms of the downstream tasks, RM-pretrained models improve the baseline by 1 ∼box AP points (object detection) and 1 ∼ 2 mask AP points (instance segmentation) on the COCO dataset and 1 ∼mIoU (semantic segmentation) on ADE20K while consistently outperforming models pretrained by CutMix.

Original: Notably, under the one-stage detector ATSS [69], the RMpretrained model under ResNet-50 outperforms the baseline and CutMix by 2.1 AP and 1.4 AP points, respectively.
Revised: Notably, under the one-stage detector ATSS [71], the RMpretrained model under ResNet-50 outperforms the baseline and CutMix by 2.1 AP and 1.4 AP points, respectively.

Original:  Related Work
Revised:  Related Work

Original: Regularization Methods.
Revised: Regularization Methods.

Original: There are many other types of regularization techniques that focus on thefeature [51, 20, 30], data [15, 8], label [47], data-label pair [68, 66, 34] and feature-label pair [57].
Revised: There are many other types of regularization techniques that focus on the feature [53, 20, 30], data [15, 8], label [49], data-label pair [70, 68, 34] and feature-label pair [59].

Original: Dropout [51], DropBlock [20] and Stochastic Depth [30] introduce the stochastic feature dropduring training in an element-wise, block-wise, and path-wise way, respectively.
Revised: Dropout [53], DropBlock [20] and Stochastic Depth [30] introduce the stochastic feature drop during training in an element-wise, block-wise, and path-wise way, respectively.

Original: Cutout [15] and
Revised: Cutout [15] and

Original: GridMask [8] randomly/systematically erase regional pixels on images while Label Smoothing [47]tries to prevent the classifiers from being too confident on a certain category by slightly modifyingthe training labels into a soft distribution [27].
Revised: GridMask [8] randomly/systematically erase regional pixels on images while Label Smoothing [49] tries to prevent the classifiers from being too confident on a certain category by slightly modifying the training labels into a soft distribution [27].

Original: Mixup [68] and CutMix [66] both combine twosamples, where the corresponding training label is given by the linear interpolation of two one-hotlabels.
Revised: Mixup [70] and CutMix [68] both combine two samples, where the corresponding training label is given by the linear interpolation of two one-hot labels.

Original: They differ in the detailed combining strategy of the input images: Mixup employs pairwise linear combination but CutMix adopts a regional crop-and-paste operation.
Revised: They differ in the detailed combining strategy of the input images: Mixup employs pairwise linear combination but CutMix adopts a regional crop-and-paste operation.

Original: SaliencyMix [56],
Revised: SaliencyMix [58],

Original: PuzzleMix [34], Co-Mixup [33], and SuperMix [13] further exploit the salient regions and localstatistics for optimal Mixup.
Revised: PuzzleMix [34], Co-Mixup [33], and SuperMix [13] further exploit the salient regions and local statistics for optimal Mixup.

Original: Attentive CutMix [58] and SnapMix [31] take the activation map ofthe correct label as a confidence indicator for selecting the semantically meaningful mix regions.
Revised: Attentive CutMix [60] and SnapMix [31] take the activation map of the correct label as a confidence indicator for selecting the semantically meaningful mix regions.

Original: TransMix [5] requires the attention map of the self-attention module to re-weight the targets.
Revised: TransMix [5] requires the attention map of the self-attention module to re-weight the targets.

Original: Manifold
Revised: Manifold

Original: Mixup [57], PatchUp [18] and MoEx [39] perform a feature-level interpolation over two samples toprevent overfitting the intermediate representations.
Revised: Mixup [59], PatchUp [18] and MoEx [39] perform a feature-level interpolation over two samples to prevent overfitting the intermediate representations.

Original: Recently, StyleMix [28] employs style transferto enrich the generated mixed images.
Revised: Recently, StyleMix [28] employs style transfer to enrich the generated mixed images.

Original: The proposed RecursiveMix (RM) shares similarities with
Revised: The proposed RecursiveMix (RM) shares similarities with

Original: CutMix by combining data samples in a regional replacement manner, but differs essentially from allthe existing regularizers in that RM is the first to exploit the historical information.
Revised: CutMix by combining data samples in a regional replacement manner, but differs essentially from all the existing regularizers in that RM is the first to exploit the historical information.

Original: In the experimentpart, we will show the comparisons between the proposed RM and other regularization methods.
Revised: In the experiment part, we will show the comparisons between the proposed RM and other regularization methods.

Original: Contrastive Learning.
Revised: Contrastive Learning.

Original: Contrastive learning [22] aims to minimize the distances between the positivepairs by consistency losses [22] which measure their similarities in a representation space or aprobability distribution space.
Revised: Contrastive learning [22] aims to minimize the distances between the positive pairs by consistency losses [22] which measure their similarities in a representation space or a probability distribution space.

Original: In recent years, contrastive learning has made a lot of progress inthe fields of computer vision [24, 49, 2, 55, 1, 9, 64, 48, 4, 10, 19, 11, 21, 38] and natural languageprocessing [62, 46, 32].
Revised: In recent years, contrastive learning has made a lot of progress in the fields of computer vision [24, 51, 2, 57, 1, 9, 66, 50, 4, 10, 19, 11, 21, 38] and natural language processing [64, 48, 32].

Original: Positive pairs can be constructed in many ways.
Revised: Positive pairs can be constructed in many ways.

Original: In the fields of selfsupervised learning and semi-supervised learning, many works [2, 24, 9, 4, 21] have used twoaugmented inputs of one instance to form positive pairs.
Revised: In the fields of selfsupervised learning and semi-supervised learning, many works [2, 24, 9, 4, 21] have used two augmented inputs of one instance to form positive pairs.

Original: In terms of generating positive pairs withdifferent model structures or weights, [62, 50] employ Dropout [51] twice on one model, and [24, 55]make a difference in weights through Exponential Moving Average (EMA) and separate projectionheads.
Revised: In terms of generating positive pairs with different model structures or weights, [64, 52] employ Dropout [53] twice on one model, and [24, 57] make a difference in weights through Exponential Moving Average (EMA) and separate projection heads.

Original: The above methods essentially require Siamese networks [3] or the need to pass one instancethrough a model twice to obtain positive pairs, which involves additional computational costs andlarge architectural designs.
Revised: The above methods essentially require Siamese networks [3] or the need to pass one instance through a model twice to obtain positive pairs, which involves additional computational costs and large architectural designs.

Original: Another way is to record historical representations or predictions andcompare them with their current outputs [64, 35, 38, 63].
Revised: Another way is to record historical representations or predictions and compare them with their current outputs [66, 35, 38, 65].

Original: However, using a memory bank to store keystatistics for all instances consumes a huge memory cost and is not applicable to even bigger datasets.
Revised: However, using a memory bank to store key statistics for all instances consumes a huge memory cost and is not applicable to even bigger datasets.

Original: Our method employs the idea of contrastive learning from the learning history while consumingextremely negligible storage and computational complexity.
Revised: Our method employs the idea of contrastive learning from the learning history while consuming extremely negligible storage and computational complexity.

Original: Learning with History.
Revised: Learning with History.

Original: In the training history, the accumulated elements are a large amount ofwealth for the improvement of learning deep models.
Revised: In the training history, the accumulated elements are a large amount of wealth for the improvement of learning deep models.

Original: Existing methods have explored the usageacross model parameters [4, 10, 24, 19, 11, 21], gradients [52, 17, 67, 36, 45, 42], predictions [35, 38],and all-level feature maps [61, 63, 64, 24] applied during the training process.
Revised: Existing methods have explored the usage across model parameters [4, 10, 24, 19, 11, 21], gradients [54, 17, 69, 36, 47, 44], predictions [35, 38], and all-level feature maps [63, 65, 66, 24] applied during the training process.

Original: More related worksrefer to the Supplementary Material.
Revised: More related works refer to the Supplementary Material.

Original: Different from existing practices, we exploit a novel aspect ofthe historical input-prediction-label triplets to improve the generalization of vision models.
Revised: Different from existing practices, we exploit a novel aspect of the historical input-prediction-label triplets to improve the generalization of vision models.

Original:  Method
Revised:  Method

Original: In this section, we introduce the proposed RecursiveMix (RM) approach and discuss its properties.
Revised: In this section, we introduce the proposed RecursiveMix (RM) approach and discuss its properties.

Original: 3.1 RecursiveMix
Revised: 3.1 RecursiveMix

Original:  We attempt to exploit the historical input-prediction-label triplets of a learning instance to improvethe generalization of vision models.
Revised: [Figure] We attempt to exploit the historical input-predictionlabel triplets of a learning instance to improve the generalization of vision models.

Original: A practical idea is to reuse the past input-prediction-label tripletsto construct diversely mixed training samples with the current batch (see Fig. 1).
Revised: A practical idea is to reuse the past input-prediction-label triplets to construct diversely mixed training samples with the current batch (see Fig. 1).

Original: To be specific, thehistorical input images are resized and then filled into the current ones where the labels are fusedproportionally to the area of the mixed patches.
Revised: To be specific, the historical input images are resized and then filled into the current ones where the labels are fused proportionally to the area of the mixed patches.

Original: Interestingly, such an operation formulates an exactrecursive paradigm where each instance can have a gradually shrunken size of view during training,thus we term the method “RecursiveMix” (RM).
Revised: Interestingly, such an operation formulates an exact recursive paradigm where each instance can have a gradually shrunken size of view during training, thus we term the method “RecursiveMix” (RM).

Original: Specifically at iteration t , the preliminary of RMis to generate a new training sample ( (cid:101) x t , (cid:101) y t ) by combining the current sample ( x t , y t ) and the pasthistorical one ( x h , y h ) .
Revised: Specifically at iteration t , the preliminary of RM is to generate a new training sample ( (cid:101) x t , (cid:101) y t ) by combining the current sample ( x t , y t ) and the past historical one ( x h , y h ) .

Original: The new training sample ( (cid:101) x t , (cid:101) y t ) participates in training with the originalloss objectives L CE and then updates the historical pair ( x h , y h ) iteratively:
Revised: The new training sample ( (cid:101) x t , (cid:101) y t ) participates in training with the original loss objectives L CE and then updates the historical pair ( x h , y h ) iteratively:

Original: [Equation]
Revised: [Equation]

Original: where M ∈ { 0 , 1 } W × H is the binary mask indicating the place for the historical input to fill, 1 is thebinary mask filled with ones and ⊙ is the element-wise product.
Revised: where M ∈ { 0 , 1 } W × H is the binary mask indicating the place for the historical input to fill, 1 is the binary mask filled with ones and ⊙ is the element-wise product.

Original: The combination ratio λ ( λ t indicates[Figure] λ at moment t ) is sampled from the uniform distribution U [0 , α ] where α defaults to 0.5.
Revised: The combination ratio λ ( λ t indicates λ at moment t ) is sampled from the uniform distribution U [0 , α ] where α defaults to 0.5.

Original: This setting is differentfrom the symmetric beta distribution of Mixup [68] and CutMix [66]. Because in our case, λ  denotes the proportion of the historical images, which is not suggested tobe quite large.
Revised: This setting is different from the symmetric beta distribution in Mixup [70] and CutMix [68] because λ is not suggested to be quite large for it denotes the proportion of the historical images in our case.

Original: Otherwise, the information of the newdata points has a risk of being discarded.
Revised: Otherwise, the information of the new data points has a risk of being discarded.

Original: The function ResizeFill ( x h , M ) means resizing the image x h and filling it back exactly into its rectangle sub-region where M = 1 .
Revised: The function ResizeFill ( x h , M ) means resizing the image x h and filling it back exactly into its rectangle sub-region where M = 1 .

Original: The resized height and width are also determined by the rectangle area.
Revised: The resized height and width are also determined by the rectangle area.

Original: This design is to makethe recursive mechanism reasonable since the originalcut operation in CutMix [66] could possibly lose former information, which would cause the inconsistencyproblem with the label (see Fig. 3).
Revised: This design is to make the recursive mechanism reasonable since the original cut operation in CutMix [68] could possibly lose former information, which would cause the inconsistency problem with the label (see Fig. 3).

Original: The binary mask M is generated by filling in with 1 inside the sampled bounding box coordinates
Revised: The binary mask M is generated by filling in with 1 inside the sampled bounding box coordinates

Original: B = ( r x , r y , r w , r h ) , otherwise defaulting to 0.
Revised: B = ( r x , r y , r w , r h ) , otherwise defaulting to 0.

Original: Similar to CutMix [66], the size and position of therectangular region B are uniformly sampled according to:
Revised: Similar to CutMix [68], the size and position of the rectangular region B are uniformly sampled according to:

Original: [Equation]
Revised: [Equation]

Original: As depicted in Fig.
Revised: As depicted in Fig.

Original: 2, the resize-filled part within the current input is semantically consistent with thehistorical input regardless of proportion and scale.
Revised: 2, the resize-filled part within the current input is semantically consistent with the historical input regardless of proportion and scale.

Original: Inspired by contrastive learning [22, 24, 4], weoptimize the KL divergence between the cross-iteration predictions of the two corresponding regions.
Revised: Inspired by contrastive learning [22, 24, 4], we optimize the KL divergence between the cross-iteration predictions of the two corresponding regions.

Original: To be specific, we record the historical prediction p h which is updated by the original outputs afternetwork function F ( · ) , global average pooling GAP ( · ) , and a linear layer H ( · ) in the last iteration:
Revised: To be specific, we record the historical prediction p h which is updated by the original outputs after network function F ( · ) , global average pooling GAP ( · ) , and a linear layer H ( · ) in the last iteration:

Original: [Equation]
Revised: [Equation]

Original: In the current iteration, we obtain the corresponding prediction of local features by 1×1 RoIAlign [25]aligning with the computed coordinates B of RecursiveMix:
Revised: In the current iteration, we obtain the corresponding prediction of local features by 1×1 RoIAlign [25] aligning with the computed coordinates B of RecursiveMix:

Original: [Equation]
Revised: [Equation]

Original: where F denotes the backbone function and H , H ′ denote the final linear classification layer.
Revised: where F denotes the backbone function and H , H ′ denote the final linear classification layer.

Original: Bydefault, the parameters are not shared between layers H and H ′ , which is purely suggested by theexperiments in Table 4a.
Revised: By default, the parameters are not shared between layers H and H ′ , which is purely suggested by the experiments in Table 4a.

Original: The total loss function to train our model is:
Revised: The total loss function to train our model is:

Original: [Equation]
Revised: [Equation]

Original: where L CE denotes the cross-entropy loss and L KL denotes the consistency loss.
Revised: where L CE denotes the cross-entropy loss and L KL denotes the consistency loss.

Original: Note that p his the historical prediction from the last iteration, and the consistency loss weight ω is set to 0.1unless otherwise stated.
Revised: Note that p h is the historical prediction from the last iteration, and the consistency loss weight ω is set to 0. unless otherwise stated.

Original: In addition, we also use the mixed ratio λ representing the proportion of theresize-filled historical image to weight L KL .
Revised: In addition, we also use the mixed ratio λ representing the proportion of the resize-filled historical image to weight L KL .

Original: It makes sense that the confidence of L KL relates to λas a smaller λ (smaller image size to be filled) usually causes the RoI feature to lose more spatialsemantic information.
Revised: It makes sense that the confidence of L KL relates to λ as a smaller λ (smaller image size to be filled) usually causes the RoI feature to lose more spatial semantic information.

Original: 3.2 Discussion
Revised: 3.2 Discussion

Original: Similar to CutMix/Mixup, RM is simple and hardly introduces additional computational cost (see
Revised: Similar to CutMix/Mixup, RM is simple and hardly introduces additional computational costs (see

Original: Table 11).
Revised: Table 11).

Original: During training, only a mini-batch of the historical input, label, and prediction (i.e.,( x h , y h , p h ) ) needs to be stored and updated iteratively.
Revised: During training, only a mini-batch of the historical input, label, and prediction (i.e., ( x h , y h , p h ) ) needs to be stored and updated iteratively.

Original: The RoI [25] operator and an individual fullyconnected layer are considerably lightweight and free of deployment.
Revised: The RoI [25] operator and an individual fully connected layer are considerably lightweight and free of deployment.

Original: The additional storage cost isextremely negligible compared to the entire model, data, and feature maps, thus making RM veryefficient to train any network architecture (Table 11).
Revised: The additional storage cost is extremely negligible compared to the entire model, data, and feature maps, thus making RM very efficient to train any network architecture.

Original: As illustrated in Fig. 2, introducing the historical design of input, label, and prediction into augmentingthe mixed training pairs recursively and bridging the spatial semantics consistency has three obviousadvantages: 1) RM encourages more diversity in data space as it exhaustively explores exponentialcombinations of multiple instances along with their corresponding richer training signals within asingle iteration. 2) RM can explicitly provide continuous training supervisions for an instance with
Revised: As illustrated in Fig. 2, introducing the historical design of input, label, and prediction into augmenting the mixed training pairs recursively and bridging the spatial semantics consistency has three obvious

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure][Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table]
Revised: advantages: 1) RM encourages more diversity in data space as it exhaustively explores exponential combinations of multiple instances along with their corresponding richer training signals within a single iteration.

Original: [Figure] [Figure] [Figure] multi-scale and spatial-variant views.
Revised: 2) RM can explicitly provide continuous training supervisions for an instance with multi-scale and spatial-variant views.

Original: 3) RM makes use of the consistency ofthe inputs from twoadjacent iterations through contrastive learning via the consistency loss, which helps the model learnspatial-correlative semantic representation.
Revised: 3) RM makes use of the identical part of inputs from two adjacent iterations through contrastive learning via the consistency loss, which helps the model learn spatial-correlative semantic representation.

Original: These properties thus lead to the better generalizationof computer vision models and can potentially benefit downstream tasks, e.g., object detection andsemantic segmentation with enhanced spatial representation ability.
Revised: These properties thus lead to the better generalization of computer vision models and can potentially benefit downstream tasks, e.g., object detection and semantic segmentation with enhanced spatial representation ability.

Original: Interestingly these properties canbe simultaneously achieved by a simple operation of iteratively using the historical input-predictionlabel triplets.
Revised: Interestingly these properties can be simultaneously achieved by a simple operation of iteratively using the historical input-predictionlabel triplets.

Original:  Experiment
Revised:  Experiment

Original: In this section, we evaluate RM on image recognition tasks to show the effectiveness of the usage ofhistorical input-prediction-label triplets.
Revised: In this section, we evaluate RM on image recognition tasks to show the effectiveness of the usage of historical input-prediction-label triplets.

Original: More details can be found in the Supplementary Material.
Revised: More details can be found in the Supplementary Material.

Original: 4.1 CIFAR Classification
Revised: 4.1 CIFAR Classification

Original: Dataset.
Revised: Dataset.

Original: The two CIFAR datasets [37] consist of colored natural scene images, each with 32×32pixels in total.
Revised: The two CIFAR datasets [37] consist of colored natural scene images, each with 32× pixels in total.

Original: The train and test sets have 50K images and 10K images respectively.
Revised: The train and test sets have 50K images and 10K images respectively.

Original: CIFAR-10 has10 classes and CIFAR-100 has 100.
Revised: CIFAR-10 has 10 classes and CIFAR-100 has 100.

Original: Setup.
Revised: Setup.

Original: We conduct two major training settings: 200-epoch and 300-epoch training, respectively.
Revised: We conduct two major training settings: 200-epoch and 300-epoch training, respectively.

Original: For200-epoch training, we employ SGD with a momentum of 0.9, a weight decay of 5 × 10 − 4 , and [-
-]
Revised: For 200-epoch training, we employ SGD with a momentum of 0.9, a weight decay of 5 × 10 − 4 , and [-
-][Table] [Table] [Table]

Revised: [Table]

Original: GPUs with a mini-batch size of 64 on each to optimize the models.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: 2 GPUs with a mini-batch size of 64 on each to optimize the models.

Original: The learning rate is set to 0.1with a linear warmup [26] for five epochs and a cosine decay schedule [44].
Revised: The learning rate is set to 0. with a linear warmup [26] for five epochs and a cosine decay schedule [46].

Original: For 300-epoch training,we align all the hyperparameters with the official CutMix [66] for fair comparisons.
Revised: For 300-epoch training, we align all the hyperparameters with the official CutMix [68] for fair comparisons.

Original: Following [66],the averaged best performances are reported via three trials of experiments.[Table] [Table] [Table]
Revised: Following [68], the averaged best performances are reported via three trials of experiments.

Original: [Table]

Original: [Table]

Original: [Table]

Original: Comparisons with State-of-the-art Regularizers.
Revised: Comparisons with State-of-the-art Regularizers.

Original: Based on PyramidNet-200, ˜ α =240 [23], Table 1shows the comparisons against the state-of-the-art data augmentation and regularization approachesunder 300 epochs.
Revised: Based on PyramidNet-200, ˜ α =240 [23], Tableshows the comparisons against the state-of-the-art data augmentation and regularization approaches under 300 epochs.

Original: The proposed RM achieves a 2.35% Top-1 classification error in CIFAR-10, 1.5%better than the baseline (3.85%).
Revised: The proposed RM achieves a 2.35% Top-1 classification error in CIFAR-10, 1.5% better than the baseline (3.85%).

Original: It outperforms the two popular regularizers Mixup and CutMix by0.74% and 0.53% points, respectively.
Revised: It outperforms the two popular regularizers Mixup and CutMix by 0.74% and 0.53% points, respectively.

Original: Performance under Various Network Backbones.
Revised: Performance under Various Network Backbones.

Original: The effectiveness of RM is further validatedacross a variety of network architectures in CIFAR-100 under the 200-epoch training setting, including
Revised: The effectiveness of RM is further validated across a variety of network architectures in CIFAR-100 under the 200-epoch training setting, including

Original: ResNet [26], DenseNet [29], and PyramidNet [23].
Revised: ResNet [26], DenseNet [29], and PyramidNet [23].

Original: From Table 3, we observe that RM has a consistentimprovement in accuracy against the baselines (+1.5 ∼ 3.5%) and other competitive counterparts(+0.3 ∼ 1.1%).
Revised: From Table 3, we observe that RM has a consistent improvement in accuracy against the baselines (+1.5 ∼ 3.5%) and other competitive counterparts (+0.3 ∼ 1.1%).

Original: Notably, for DenseNet-161, RM shows an absolute gain of 3.2% points over thebaseline model and outperforms the competitive CutMix by 1.1%.
Revised: Notably, for DenseNet-161, RM shows an absolute gain of 3.2% points over the baseline model and outperforms the competitive CutMix by 1.1%.

Original: Ablation Study on hyperparameter α .
Revised: Ablation Study on Hyperparameter α .

Original: To reveal the impact of α , we conduct ablation study byvarying α ∈ { 0 . 1 , 0 . 2 , 0 .3 , 0 . 4 , 0 .
Revised: To reveal the impact of α , we conduct ablation study by varying α ∈ { 0 . 1 , 0 . 2 , 0 .

Original: 7 , 0 . 8 , 0 . 9 } in CIFAR-10 with PyramidNet-200, ˜α
Revised: 7 , 0 . 8 , 0 . 9 } in CIFAR-10 with PyramidNet-200, ˜

Original: =240backbone under 300-epoch setting and CIFAR-100 with ResNet-18 [26] backbone under 200-epochsetting, respectively.
Revised: α = backbone under 300-epoch setting and CIFAR-100 with ResNet-18 [26] backbone under 200-epoch setting, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, RM improves upon the baseline (3.85% and 21.70%) for allconsidered α values.
Revised: 4, RM improves upon the baseline (3.85% and 21.70%) for all considered α values.

Original: The best performance is achieved when α = 0 .
Revised: The best performance is achieved when α = 0 .

Original: Ablation Study on hyperparameter ω .
Revised: Ablation Study on Hyperparameter ω .

Original: Next, we examine what is the best value for the loss weightω .
Revised: Next, we examine what is the best value for the loss weight ω .

Original: We adopt the same experimental setting on CIFAR-10 and CIFAR-100 with that on α .
Revised: We adopt the same experimental setting on CIFAR-10 and CIFAR-100 as that on α .

Original: By fixingα to 0.5, we change ω ∈ { 0 , 0 . 01 , 0 . 05 , 0 .
Revised: By fixing α to 0.5, we change ω ∈ { 0 , 0 . 01 , 0 . 05 , 0 .

Original: 5 , 1 } and report their performancesin average value and standard deviation over 3 runs on each.
Revised: 5 , 1 } and report their performances in average value and standard deviation over 3 runs on each.

Original: Notably, when ω =0, the RM modelis optimized without the consistency loss, but only leverages the historical input-label pairs.
Revised: Notably, when ω =0, the RM model is optimized without the consistency loss, but only leverages the historical input-label pairs.

Original: Fig. 5shows that the best performance is achieved when ω = 0 .
Revised: Fig.shows that the best performance is achieved when ω = 0 .

Original: 1 , while the performance is not sensitive(i.e. , around 0.3%) in CIFAR-10 to a wide range of ω ∈ [0 . 1 , 1] .
Revised: 1 , while the performance is not sensitive (i.e., around 0.3%) in CIFAR-10 to a wide range of ω ∈ [0 . 1 , 1] .

Original: Ablation Study on RM Components.
Revised: Ablation Study on RM Components.

Original: To illustrate the performance improvement brought by each
Revised: To illustrate the performance improvement brought by each

Original: RM component individually, we first modify CutMix with the same resize strategy of RM, instead ofthe original cut operation.
Revised: RM component individually, we first modify CutMix with the same resize strategy of RM, instead of the original cut operation.

Original: Then we add the historical mechanism and consistency loss, respectively.
Revised: Then we add the historical mechanism and consistency loss, respectively.

Original: It is observed in Table 2 that the resize strategy can have a slightly positive effect (i.e., +0.23%accuracy) on CutMix for it is necessary to successfully formulate the correct historical (recursive)paradigm (Fig. 3).
Revised: It is observed in Table 2 that the resize strategy can have a slightly positive effect (i.e., +0.23% accuracy) on CutMix for it is necessary to successfully formulate the correct historical (recursive) paradigm (Fig. 3).

Original: Further, the historical mechanism and consistency loss individually improve by+0.55% points and +0.16% points, respectively.
Revised: Further, the historical mechanism and consistency loss individually improve by +0.55% points and +0.16% points, respectively.

Original: Above all, RM achieves a total of 2.02% gain overthe baseline and 0.94% over CutMix. 
Revised: Above all, RM achieves a total of 2.02% gain over the baseline and 0.94% over CutMix. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: Ablation Study on Linear Classifiers.
Revised: Ablation Study on Linear Classifiers.

Original: As illustrated in Sec. 3, the historical predictions and thealigned current predictions are derived through separate linear classifiers H and H ′ .
Revised: As illustrated in Sec. 3, the historical predictions and the aligned current predictions are derived through separate linear classifiers H and H ′ .

Original: We conductedcomparisons on whether to share  parameters of the two linear layers.
Revised: We conducted comparisons on whether to share the parameters of the two linear layers.

Original: The training strategy on CIFARdatasets follows the above.
Revised: The training strategy on CIFAR datasets follows the above.

Original: Table 4a shows that unshared weights bring slightly better performancefor it enhances model diversity which benefits the contrastive learning.[Table] [Table] [Table]
Revised: Table 4a shows that unshared weights bring slightly better performance for it enhances model diversity which benefits the contrastive learning.

Original: [Table]

Original: Ablation Study on Interpolation Mode.
Revised: Ablation Study on Interpolation Mode.

Original: As depicted in Table 4b, for the resizing operation onhistorical inputs, we compare two common down-sampling algorithms, i.e., interpolation methods,“nearest” and “bilinear”.
Revised: As depicted in Table 4b, for the resizing operation on historical inputs, we compare two common down-sampling algorithms, i.e., interpolation methods, “nearest” and “bilinear”.

Original: As a result, “nearest” performs slightly better, so we fix the interpolationmethod to it by default.
Revised: As a result, “nearest” performs slightly better, so we fix the interpolation method to it by default.

Original: 4.2 ImageNet Classification
Revised: 4.2 ImageNet Classification

Original: Dataset.
Revised: Dataset.

Original: The ImageNet 2012 dataset [14] contains 1.28 million training images and 50K validationimages from 1K classes.
Revised: The ImageNet 2012 dataset [14] contains 1.28 million training images and 50K validation images from 1K classes.

Original: Networks are trained on the training set and the Top-1/-5 errors are reportedon the validation set.
Revised: Networks are trained on the training set and the Top-1/-5 errors are reported on the validation set.

Original: Setup.
Revised: Setup.

Original: For ImageNet, we also conduct experiments with sufficient training settings (300 epochs)under two major backbone designs, i.e., CNNs [29, 26] and Transformers [59].
Revised: For ImageNet, we also conduct experiments with sufficient training settings (300 epochs) under two major backbone designs, i.e., CNNs [29, 26] and Transformers [61].

Original: For training CNNbackbone networks, we employ the augmentations described in CutMix [66] to the input images fora fair comparison.
Revised: For training CNN backbone networks, we employ the augmentations described in CutMix [68] to the input images for a fair comparison.

Original: All networks are trained using SGD with a momentum of 0.9, a weight decayof 1 × 10 − 4 , and 8 GPUs with a mini-batch size of 64 on each to optimize models.
Revised: All networks are trained using SGD with a momentum of 0.9, a weight decay of 1 × 10 − 4 , and 8 GPUs with a mini-batch size of 64 on each to optimize models.

Original: The initiallearning rate is 0.2 with a linear warmup [26] for five epochs and is then decayed following a cosineschedule [44].
Revised: The initial learning rate is 0.2 with a linear warmup [26] for five epochs and is then decayed following a cosine schedule [46].

Original: To optimize Transformer backbone networks, we use AdamW [45] with a learningrate of 5 × 10 − 4 , a momentum of 0.9, a weight decay of 5 × 10 − 2 , and 8 GPUs with a mini-batchsize of 64 on each.
Revised: To optimize Transformer backbone networks, we use AdamW [47] with a learning rate of 5 × 10 − 4 , a momentum of 0.9, a weight decay of 5 × 10 − 2 , and 8 GPUs with a mini-batch size of 64 on each.

Original: We follow PVT [60] and apply random resizing/cropping of 224×224 pixels,random horizontal flipping [53], label-smoothing regularization [54], and random erasing [71] as thestandard data augmentations.
Revised: We follow PVT [62] and apply random resizing/cropping of 224×224 pixels, random horizontal flipping [55], label-smoothing regularization [56], and random erasing [73] as the standard data augmentations.

Original: The hyperparameters for RM on ImageNet are set to α =0.5, ω =0.5.
Revised: The hyperparameters for RM on ImageNet are set to α =0.5, ω =0.5.

Original: Comparison with State-of-the-art Regularizers.
Revised: Comparison with State-of-the-art Regularizers.

Original: Based on ResNet-50 [26], we compare RMagainst a series of popular regularizers in Table 6 under the 300-epoch training setting, where RMsignificantly improves the baseline by absolute 2.88% points in Top-1 accuracy and outperforms thestrong CutMix and PuzzleMix by 0.6% and 0.44%, respectively.
Revised: Based on ResNet-50 [26], we compare RM against a series of popular regularizers in Table 6 under the 300-epoch training setting, where RM significantly improves the baseline by absolute 2.88% points in Top-1 accuracy and outperforms the strong CutMix and PuzzleMix by 0.6% and 0.44%, respectively.

Original: Performance under Various Network Backbones.
Revised: Performance under Various Network Backbones.

Original: Since Transformer models [16, 43, 60, 59]have made significant progress in image classification due to the self-attention mechanism, we alsodemonstrate the effectiveness of RM on the popular Transformer network architecture PVTv2-B1 [59]in ImageNet under the 300-epoch training setting in Table 5, where it remains the superiority.
Revised: Since Transformer models [16, 45, 62, 61] have made significant progress in image classification due to the self-attention mechanism, we also demonstrate the effectiveness of RM on the popular Transformer network architecture PVTv2-B1 [61] in ImageNet under the 300-epoch training setting in Table 5, where it remains the superiority.

Original: 4.3 Transfer Learning
Revised: 4.3 Transfer Learning

Original: Benefiting from the explicit multi-scale/-space property and spatial semantic learning of the proposed RecursiveMix (RM), we suspect that the pretrained model with RM can transfer well to thedownstream tasks, e.g., object detection, instance segmentation, and semantic segmentation, wheremulti-scale/-space semantic information plays an important role for identification.
Revised: Benefiting from the explicit multi-scale/-space property and spatial semantic learning of the proposed RecursiveMix (RM), we suspect that the pretrained model with RM can transfer well to the downstream tasks, e.g., object detection, instance segmentation, and semantic segmentation, where multi-scale/-space semantic information plays an important role for identification.

Original: Object Detection and Instance Segmentation.
Revised: Object Detection and Instance Segmentation.

Original: We conduct experiments using the one-stage objectdetector ATSS [69], GFL [40] and two-stage detector Mask-RCNN [25], HTC [6] in COCO [41]dataset with comparisons to the normally pretrained and + CutMix [66] pretrained models with
Revised: We conduct experiments using the one-stage object detector ATSS [71], GFL [42, 41, 40], and two-stage detector Mask R-CNN [25], HTC [6] [Table] [Table] [Table]

Original:  ResNet-50 [26] and PVTv2-B1 [59].
Revised: [Table] [Table] in COCO [43] dataset with comparisons to the vanilla model and CutMix [68]-pretrained model based on ResNet-50 [26] and PVTv2-B1 [61].

Original: The training protocol follows the standard 1x (12 epochs)setting as described in MMDetection [7].
Revised: The training protocol follows the standard 1x ( epochs) setting as described in MMDetection [7].

Original: In Table 7, it is observed that under ATSS, the RMpretrained models significantly improve baseline by +2.1 AP points with ResNet-50 and [-+3.0 APpoints with PVTv2-B1, while outperforming CutMix [66] pretrained models by +1.4 AP and 0.
 [Table]-]
Revised: In Table 7, it is observed that under ATSS, the RM-pretrained models significantly improve baseline by +2.1 AP points with ResNet-50 and [-+3.0 APpoints with PVTv2-B1, while outperforming CutMix [66] pretrained models by +1.4 AP and 0.
 [Table]-]+3.

Original: [Table] [Table] [Table] [Table]
Revised: AP points with PVTv2-B1, while outperforming CutMix [68] pretrained models by +1.4 AP and 0.

Original: AP.
Revised: AP.

Original: Table 8 indicates the superiority of RM in instance segmentation under Mask R-CNN and HTC,where RM boosts the bbox AP and mask AP by ∼ 2.8 and ∼ 2.0 points, respectively.
Revised: Table 8 indicates the superiority of RM in instance segmentation under Mask R-CNN and HTC, where RM boosts the bbox AP and mask AP by ∼ 2.8 and ∼ 2.0 points, respectively.

Original: Semantic Segmentation.
Revised: Semantic Segmentation.

Original: Next, we experiment on ADE20K [73] using two popular algorithms, i.e.,
Revised: Next, we experiment on ADE20K [75] using two popular algorithms, i.e.,

Original: PSPNet [70] and UperNet [65] following the semantic segmentation code of MMSegmentation [12].
Revised: PSPNet [72] and UperNet [67] following the semantic segmentation code of MMSegmentation [12].

Original: Under ResNet-50 [26] and PVTv2-B1 [59] backbone networks, we fine-tune 80k iterations on
Revised: Under ResNet-50 [26] and PVTv2-B1 [61] backbone networks, we fine-tune 80k iterations on

Original: ADE20K with a batch size of 16 and AdamW [45] optimization algorithm.
Revised: ADE20K with a batch size of 16 and AdamW [47] optimization algorithm.

Original: Table 9 shows that amongvarious pretrained models RM outperforms the baseline and CutMix [66].
Revised: Table 9 shows that among various pretrained models RM outperforms the baseline and CutMix [68].

Original: Notably, we improve the
Revised: Notably, we improve the

Original: ResNet-50 by 1.9 and 1.1 mIoU compared to baseline and CutMix under UperNet.
Revised: ResNet-50 by 1.9 and 1.1 mIoU compared to baseline and CutMix under UperNet.

Original: Also, there is animprovement of 3.3 mIoU over the baseline under PVTv2-B1 based on UperNet.
Revised: Also, there is an improvement of 3.3 mIoU over the baseline under PVTv2-B1 based on UperNet.

Original: 4.4 Analyses
Revised: 4.4 Analyses

Original: Class Activation Mapping Visualization.
Revised: Class Activation Mapping Visualization.

Original: To demonstrate the benefits of RM, we visualize the classactivation map through CAM [72].
Revised: To demonstrate the benefits of RM, we visualize the class activation map through CAM [74].

Original: We select images that originally contained multiple categoriesand create CAM heatmaps for each (Fig. 6) to demonstrate the superiority of RM in spatial semanticrepresentation.
Revised: We select images that originally contain multiple categories and create CAM heatmaps for each (Fig. 6) to demonstrate the superiority of RM in spatial semantic representation.

Original: Additionally, we generate several mixed images through RM and visualize theactivation map for each participating ground truth class (Fig. 7).

Original: The visualization shows that RMlocates the corresponding objects more accurately than Baseline and CutMix.
Revised: The visualization shows that RM locates the corresponding objects more accurately than Baseline and CutMix.

Original: Thanks to the recursiveparadigm of RM, the algorithm can generate inputs that have multiple objects (>2), which increasesthe diversity of inputs in data space and enriches each instance with multi-scale and spatial-variantviews.
Revised: Thanks to the recursive paradigm of RM, the algorithm can generate inputs that have multiple objects (>2), which increases the diversity of inputs in data space and enriches each instance with multi-scale and spatial-variant views.

Original: Thus, the model is more capable of recognizing complex images containing multiple categoriesor objects that differ greatly in size.
Revised: Thus, the model is more capable of recognizing complex images containing multiple categories or objects that differ greatly in size.

Original: Furthermore, the model trained under RM explicitly learns densesemantic representations and can locate the meaningful regions more accurately via the designedconsistency loss.
Revised: Furthermore, the model trained under RM explicitly learns dense semantic representations and can locate the meaningful regions more accurately via the designed consistency loss.

Original: Consistency Loss.
Revised: More visualization can be found in the Supplementary Material.

Original: The consistency loss explicitly restricts the representations between two samples. Many works [2, 24, 9, 4, 21] formulate the positive pairs by employing two distributions of dataaugmentations on the images.
Revised: RM Enables Efficient Self-distillation. Generally, the consistency loss resembles a variant of local self-distillation by explicitly restricting two spatial representations of one instance.

Original: Another strategy is to pass one instance through two different models,such as the models with unique Dropout [51] paradigms [62, 50] or with different parameters(normally updated vs. EMA updated) [4, 10, 24, 19, 11, 21, 55]. These methods inevitably deducetwice on each sample, resulting in a long training time and high GPU memory cost. [64, 35, 38, 63]record
Revised: However, it is worth mentioning that the proposed recursive augmentation paradigm and the consistency loss are not isolated from each other. Only in the form of the “recursive mix”, can we achieve self-distillation in a convenient, efficient, and memory-friendly way.

Original:  theepoch-wise historical representations and form positive pairs together with samples in thecurrent epoch, which consume a huge memory cost. We innovatively introduce an iteration-wisehistorical consistency loss between two regions that share identical semantics.
Revised: Because in the recursive paradigm, the historical part in the current iteration can be supervised directly by the predictions of the former iteration as they share identical but differentially-scaled contents.

Original: By avoiding repeatedcomputation and recording key statistics of  a whole dataset, RM involves negligible additional storageand computation consumption. Effective Class Number.
Revised: It delicately fits the logic of local self-distillation, at a minimal cost of storing only the outputs of the last iteration.

Original: To illustrate the diversity in the supervision signals of RM, we denote thenumber of classes in the regularized target whose supervision values are higher than a given thresholdas effective classes. We then record the average effective number over 3 runs under 100 iterations
Revised: On the contrary, to implement local self-distillation without the recursive operation, one needs to forward one input through the networks repeatedly [64, 52, 4, 10, 24, 19, 11, 21, 57] or record the information of the whole data

Original: M i xup
Revised: M i xup

Original: C u t M i x
Revised: C u t M i x

Original: R M ( ou r s )
Revised: R M ( ou r s )

Original: Natural Samples with Multiple Labels
Revised: Natural Samples with Multiple Labels

Original: [Figure]
Revised: [Figure]

Original: Marmoset

Original: [Figure]

Original: Crate
Revised: Crate

Original: [Figure]
Revised: [Figure]

Original: Hourglass

Original: [Figure]

Original: Eel
Revised: Eel

Original: Mantis
Revised: Mantis

Original: Pembroke
Revised: Pembroke

Original: Oxcart
Revised: Oxcart

Original: Leopard
Revised: Leopard

Original: Echidna
Revised: Echidna

Original: M i xup
Revised: M i xup

Original: C u t M i x
Revised: C u t M i x

Original: R M ( ou r s ) M i xup C u t M i x R M ( ou r s )
Revised: R M ( ou r s ) in an entire epoch [66, 35, 38, 65], which either requires a tremendous computation cost or large memory consumption.

Original: Natural Samples
Revised: Experimentally, RM with consistency loss enhances accuracy by 0.16% over that without in Table 2. We also verify the effectiveness of consistency loss on downstream tasks in Table 10. With the models pretrained with consistency loss, GFL and ATSS improve 0.5mAP on COCO while PSPNet and UperNet improve 0.7 mIoU on ADE20K. The results show that the introduced consistency loss enhances the performance on the downstream tasks via learning the spatial-correlative semantic representation.

Original: [Figure]
Revised: [Figure]

Original: M i xup
Revised: Effective Class Number. To illustrate the diversity in the supervision signals of RM, we denote the number of classes in the regularized target whose supervision values are higher than a given threshold as effective classes.

Original: C u t M i x R M ( ou r s ) [Figure] on RM and other methods in the ImageNet training set.
Revised: We then record the average effective number over 3 runs under 100 iterations on RM and other methods in the ImageNet training set.

Original: Note thatthe label smoothing threshold is roughly set to “ 10 − 4 ”.
Revised: Note that the label smoothing threshold is roughly set to “ 10 − 4 ”.

Original: We suspectthat a signal value above the label smoothing threshold can be usedas effective supervision, then RM can achieve a learning signalcontaining an average of about 4 ∼ 5 objects in a single image,which confirms the diversity of its supervision (Fig. 
Revised: We suspect that if a signal value above the label smoothing threshold can be used as effective supervision, then RM can achieve a learning signal containing an average of about 4 ∼ 5 objects in a single image, which confirms the diversity of its supervision (Fig. 7).

Original: Efficiency.
Revised: Efficiency.

Original: We improve performance with negligible additionalmemory cost, as shown in Table 11.
Revised: We improve performance with negligible additional memory cost, as shown in Table 11.

Original: Notably, the training time andcomputation complexity are competitive with existing methods,which are evaluated on 8 TITAN Xp GPUs.
Revised: Notably, the training time and computation complexity are competitive with existing methods, which are evaluated on 8 TITAN Xp GPUs.

Original: Limitations.
Revised: Limitations.

Original: Although our method shares the same parametersand FPS as the baseline model during employment, due to theintroduction of an additional classification head and the operationof RoIAlign, we inevitably increase a small number of parametersonly during the training process.
Revised: Although our method shares the same parameters and FPS as the baseline model during employment, due to the introduction of an additional classification head and the operation of RoIAlign, we inevitably increase a small number of parameters only during the training process.

Original: In addition, it is unclear how to effectively adopt the recursiveparadigm to Mixup [68] since the images would be cluttered and unidentifiable via iterative mixupoperations.
Revised: In addition, it is unclear how to effectively adopt the recursive paradigm to Mixup [70] since the images would be cluttered and unidentifiable via iterative mixup operations.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a novel regularization method termed RecursiveMix (RM), which leveragesthe rarely exploited aspect of the historical information: the input-prediction-label triplets, to enhancethe generalization of deep vision models.
Revised: In this paper, we propose a novel regularization method termed RecursiveMix (RM), which leverages the rarely exploited aspect of the historical information: the input-prediction-label triplets, to enhance the generalization of deep vision models.

Original: RM shares several good properties according to its historicalmechanism, and it consistently improves the recognition accuracy on competitive vision benchmarkswith considerably negligible additional budgets.
Revised: RM shares several good properties according to its historical mechanism, and it consistently improves the recognition accuracy on competitive vision benchmarks with considerably negligible additional budgets.

Original: We hope RM can serve as a simple yet effectivebaseline for the community.
Revised: We hope RM can serve as a simple yet effective baseline for the community.

Revised: Acknowledgments and Disclosure of Funding

Revised: This paper is supported by the National Key R&D Plan of the Ministry of Science and Technology (Project No.2020AAA0104400), the Young Scientists Fund of the National Natural Science Foundation of China (Grant No.62206134), and the National Science Fund of China (Grant No.U1713208).

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.4.(c)
Revised: [Yes] See Section 4.4.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See thebroader impacts in the Supplementary Material.
Revised: [Yes] See the broader impacts in the Supplementary Material.

Original: d ) Have you read the ethics review guidelines and ensured that your paper conforms tothem
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes If you are including theoretical results a Did
Revised: [Yes] 2.

Original:  you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original:  (b) Did you include complete proofs of all theoretical results
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Results are reported over 3 runs on the CIFAR dataset(see Fig.
Revised: [Yes] Results are reported over 3 runs on the CIFAR dataset (see Fig.

Original: 4 and Fig. 5).
Revised: 4 and Fig. 5).

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] They are publicly available for research purpose.
Revised: [N/A] They are publicly available for research purpose.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] There are no new assets.
Revised: [No] There are no new assets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] They are publicly available for research purpose.
Revised: [N/A] They are publicly available for research purpose.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] They are publicly available for research purpose.
Revised: [N/A] They are publicly available for research purpose.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]