Original:  Reinforcement Learning Assisted Layer-wise Fine-Tuning  for Transfer Learning
Revised: RL-Tune : A Deep Reinforcement Learning Assisted Layer-wise Fine-Tuning Approach for Transfer Learning

Original: Abstract
Revised: Abstract

Original: Data scarcity is one of the major challenges in many real-world applications.
Revised: Data scarcity is one of the major challenges in many real-world applications.

Original: To handle lowdata regimes, practitioners often take an existing pre-trained network and fine-tune it on a datadeficient target task.
Revised: To handle lowdata regimes, practitioners often take an existing pre-trained network and fine-tune it on a datadeficient target task.

Original: In this setup, a network is pre-trained on a source dataset and fine-tuned on a different, potentially smaller, target dataset.
Revised: In this setup, a network is pre-trained on a source dataset and fine-tuned on a different, potentially smaller, target dataset.

Original: We address two critical challenges with transfer learning via fine-tuning: (1) The required amount of fine-tuning greatly depends on the distribution shift from source to target dataset.
Revised: We address two critical challenges with transfer learning via fine-tuning: (1) The required amount of fine-tuning greatly depends on the distribution shift from source to target dataset.

Original: (2) Layer-wise adjustments allow for the model to adapt to this distribution shift whilealso preserving the pretrained network’s feature extractor.
Revised: (2) This distribution shift greatly varies by layer, thereby requiring layer-wise adjustments in fine-tuning to adapt to this distribution shift while preserving the pre-trained network’s feature representation.

Original: To overcome the challenges, we propose RL-Tune , a layer-wise fine-tuning framework for transfer learning which leverages reinforcement learning to adjust learning rates as a function of the target data shift.
Revised: To overcome these challenges, we propose RL-Tune , a layer-wise fine-tuning framework for transfer learning which leverages reinforcement learning to adjust learning rates as a function of the target data shift.

Original: In our RL framework, the state is a collection of the intermediate feature activations generated with training samples.
Revised: In our RL framework, the state is a collection of the intermediate feature activations generated from training samples.

Original: To accommodate different abstraction levels of layers, the agent generates layer-wise learning rates as actions for fine-tuning based on the current state and obtainsthe sample accuracy as a reward .
Revised: The agent generates layer-wise learning rates as actions for fine-tuning based on the current state and obtains sample accuracy as the reward .

Original: RL-Tune outperforms other state-of-the-art approaches on standard transfer learning benchmarks by a large margin, e.g. , 6.2% mean accuracy improvement on CUB-200with 15% data.
Revised: RL-Tune outperforms other state-of-the-art approaches on standard transfer learning benchmarks by a large margin, e.g. , 6% mean accuracy improvement on CUB-200with 15% data.

Original: 1. Introduction
Revised: 1. Introduction

Original: In low-data regimes, neural networks suffer from overfitting and poor generalization on unseen training samples (Bansal et al., 2020).
Revised: In low-data regimes, neural networks suffer from overfitting and poor generalization on unseen training samples (Bansal et al., 2020).

Original: One effective solution to learning with insufficient data is transfer learning via fine-tuning where a model is pre-trained on the source dataset and subsequently fine-tuned on the target task (Tan et al., 2018).
Revised: One effective solution to learning with in- sufficient data is transfer learning via fine-tuning where a model is pre-trained on the source dataset and subsequently fine-tuned on the target task (Tan et al., 2018).

Original: We have seen the effectiveness of transfer learning via finetuning on a wide variety of applications including medical imaging (Tajbakhsh et al., 2016) and standard ML benchmarks (Guo et al., 2019).
Revised: We have seen the effectiveness of transfer learning via finetuning on a wide variety of applications including medical imaging (Tajbakhsh et al., 2016) and standard ML benchmarks (Guo et al., 2019).

Original: Prior work has shown that layerwise fine-tuning is beneficial for transfer learning, because the pre-trained model’s initial layers act as general feature extractors whereas the final layers can extract more specific features (Yosinski et al., 2014; Girshick et al., 2014).
Revised: Prior work has shown that layerwise fine-tuning is beneficial for transfer learning, because the pre-trained model’s initial layers act as general feature extractors whereas the final layers can extract more specific features (Yosinski et al., 2014; Girshick et al., 2014).

Original: When the source and target domains share similar features, we can leverage the model’s general feature extractor and only fine-tune the layers where the target task’s features differ significantly.
Revised: When the source and target domains share similar features, we can leverage the model’s general feature extractor and only fine-tune the layers where the target task’s features differ significantly.

Original: In traditional fine-tuning approaches, we either fine-tune globally, or freeze some layers and fine-tune the rest with a uniform learning rate.
Revised: In traditional fine-tuning approaches, we either fine-tune globally, or freeze some layers and fine-tune the rest with a uniform learning rate.

Original: Once a model is pre-trained, its earlier layers do not need as much fine-tuning since they act as general feature extractors (Sun et al., 2019; 
Revised: Once a model is pre-trained, its earlier layers do not need as much fine-tuning since they act as general feature extractors (Sun et al., 2019; Ro &

Original: Ro & Choi, 2021).
Revised: Choi, 2021).

Original: For transfer learning applications, there is a variety of techniques to apply layer-specific fine-tuning, including heuristic-based methods (Kornblith et al., 2019; Li et al., 2020) and supervised approaches, including probabilistic frameworks (You et al., 2020) and a zoo of models (Shu et al., 2021).
Revised: For transfer learning applications, there is a variety of techniques to apply layer-specific fine-tuning, including heuristic-based methods (Kornblith et al., 2019; Li et al., 2020) and supervised approaches, such as probabilistic frameworks (You et al., 2020) and a zoo of models (Shu et al., 2021).

Original: In contrast to the pre-training phase, the layerwise learning rates in the transfer phase are more sensitive to the data distribution shift between the source and target datasets.
Revised: In contrast to the pre-training phase, the layerwise learning rates in the transfer phase are more sensitive to the data distribution shift between the source and target datasets.

Original: Improper learning rate selection may lead to catastrophic forgetting that neutralizes the advantages of transfer learning (Sun et al., 2019).
Revised: Improper learning rate selection may lead to catastrophic forgetting that neutralizes the advantages of transfer learning (Sun et al., 2019).

Revised: Several approaches have been explored to automate the hyper-parameter search in the training phase.

Revised: To tune the hyper-parameters, a population based bandit algorithm is found to be effective in optimizing RL training (ParkerHolder et al., 2020).

Revised: A meta learning approach has been introduced recently (Oh et al., 2020) for learning the update rule of the RL algorithm.

Original: Data distribution shift is a core problem for most transfer learning applications (Weiss et al., 2016).
Revised: However, the data distribution shift is a core problem for most transfer learning applications (Weiss et al., 2016).

Original: With significant distributional shift, transfer learning becomes difficult as the source and target features may not match.
Revised: With significant distributional shift, transfer learning becomes difficult as the source and target features may not match.

Original: To overcome this challenge, we propose RL-Tune which leverages reinforcement learning to improve knowledge transfer through sample-based layer-wise fine-tuning (see Fig. 1(a)).
Revised: To overcome this challenge, we propose RL-Tune which leverages reinforcement learning to improve knowledge transfer through sample-based layer-wise fine-tuning (see Fig. 1(a)).

Original: In our RL setup, the agent’s input state is the collection of all intermediary feature maps, which captures the domain shift on a per-sample basis.
Revised: In our RL setup, the agent’s input state is the collection of all intermediary feature maps, which captures the domain shift on a per-sample basis.

Original: In response to the current state, the agent’s action is a vector of learning rates for layer-wise fine-tuning.
Revised: In response to the current state, the agent’s action is a vector of learning rates for layer-wise fine-tuning.

Original: Once one round of fine-tuning is complete, the agent’s reward is the corresponding sample accuracy.
Revised: Once one round of fine-tuning is complete, the agent’s reward is the corresponding sample accuracy.

Original: We carried out extensive experiments and find that RL-Tune outperforms state-of-the-art approaches (You et al., 2020;
Revised: We carried out extensive experiments and find that RL-Tune outperforms state-of-the-art approaches (You et al., 2020;

Original: Li & Hoiem, 2018; Chen et al., 2019; Li et al., 2018) by more than 2% for a variety of sampling rates on the CUB200-2011, Stanford, and FGVC Aircraft datasets.
Revised: Li & Hoiem, 2018; Chen et al., 2019; Li et al., 2018) by more than 2% for a variety of sampling rates on the CUB200-2011, Stanford, and FGVC Aircraft datasets.

Original: 2. Related Work 2.1. Pre-training
Revised: 2. Related Work 2.1. Pre-training

Original: Training very deep models on large-scale benchmark datasets (Deng et al., 2009; Lin et al., 2014) establishes several well known architectures, such as AlexNet (Krizhevsky et al., 2012),
Revised: Training very deep models on large-scale benchmark datasets (Deng et al., 2009; Lin et al., 2014) establishes several well known architectures, such as AlexNet (Krizhevsky et al., 2012),

Original: ResNet (He et al., 2016), DenseNet (Huang et al., 2017), Vision Transformers (Han et al., 2022), and
Revised: ResNet (He et al., 2016), DenseNet (Huang et al., 2017), Vision Transformers (Han et al., 2022), and

Original: BERT (Devlin et al., 2018).
Revised: BERT (Devlin et al., 2018).

Original: The pre-trained weights of these deep networks are widely used for transfer learning on similar datasets with significantly lower amount of data.
Revised: The pre-trained weights of these deep networks are widely used for transfer learning on similar datasets with significantly lower amount of data.

Original: Most of these networks are trained with supervised learning on large-scale labeled data.
Revised: Most of these networks are trained with supervised learning on large-scale labeled data.

Original: To leverage large-scale unlabeled data, semi-supervisedlearning (Chen et al., 2020) and self-supervised (Hendrycks et al., 2019) learning have been adopted to increase the diversity of pre-trained models (Radford et al., 2021; Brown et al., 2020).
Revised: To leverage large-scale unlabeled data, semi-supervised (Chen et al., 2020) and self-supervised (Hendrycks et al., 2019) learning have been adopted to increase the diversity of pre-trained models (Radford et al., 2021; Brown et al., 2020).

Original: 2.2. Transfer learning
Revised: 2.2. Transfer learning

Original: Fine-tuning deep pre-trained model was established as an effective approach for better performance (Girshick et al., 2014;
Revised: Fine-tuning deep pre-trained model was established as an effective approach for better performance (Girshick et al., 2014;

Original: Agrawal et al., 2014) and faster convergence (He et al., 2019) particularly in low-data regimes.
Revised: Agrawal et al., 2014) and faster convergence (He et al., 2019) particularly in low-data regimes.

Original: (Li et al., 2020) confirms that the choice of hyper-parameters, particularly learning rates and momentum, play a significant role in the final performance of transfer learning.
Revised: (Li et al., 2020) confirms that the choice of hyper-parameters, particularly learning rates and momentum, play a significant role in the final performance of transfer learning.

Original: Various approaches have been explored for better regularization of the transfer learning with effective hyper-parameter selection.
Revised: Various approaches have been explored for better regularization of the transfer learning with effective hyper-parameter selection.

Original: (Kornblith et al., 2019) proposes a grid-search based approach to search for better hyper-parameters.
Revised: (Kornblith et al., 2019) proposes a grid-search based approach to search for better hyper-parameters.

Original: (Li et al., 2020) provides an elaborate guideline of learning rates and other hyper-parameter selections that confirms the increasing challenges of fine-tuning with dissimilar target datasets.
Revised: (Li et al., 2020) provides an elaborate guideline of learning rates and other hyper-parameter selections that confirms the increasing challenges of fine-tuning with dissimilar target datasets.

Original: However, all of these approaches primarily focus on manual search of hyper-parameters that hardly take care of the distribution shift of the target datasets and various abstractions of intermediate levels.
Revised: However, all of these approaches primarily focus on manual search of hyper-parameters that hardly take care of the distribution shift of the target datasets and various abstractions of intermediate levels.

Original: Apart from these, Co-Tuning (You et al., 2020) attempts to integrate the whole pre-trained network, including top fully connected layers, by learning the probabilistic class mapping of source and target data.
Revised: Apart from these, Co-Tuning (You et al., 2020) attempts to integrate the whole pre-trained network, including top fully connected layers, by learning the probabilistic class mapping of source and target data.

Original: Self-tuning (Wang et al., 2021) integrates the additional unlabeled data with semi-supervised learning into transfer learning.
Revised: Self-tuning (Wang et al., 2021) integrates the additional unlabeled data with semi-supervised learning into transfer learning.

Original: Zoo-Tuning (Shu et al., 2021) learns the combinations of a zoo of pre-trained models for improved transfer learning.
Revised: Zoo-Tuning (Shu et al., 2021) learns the combinations of a zoo of pre-trained models for improved transfer learning.

Original: However, most of these approaches operate with a pre-defined set of hyper-parameters.
Revised: However, most of these approaches operate with a pre-defined set of hyper-parameters.

Original: We particularly focus on improving standard transfer learning from a pre-trained model to a target task with limited amount of labeled data.
Revised: We particularly focus on improving standard transfer learning from a pre-trained model to a target task with limited amount of labeled data.

Original: 3. Methodology 3.1. Problem Formulation
Revised: 3. Methodology 3.1. Problem Formulation

Original: Given a pre-trained model M 0 on a source dataset D s = { ( x is , y is } m s i =1 where x s , y s , m s represent the source image, labels, and total number of images, respectively, we generate the fine-tuned model M f on the target dataset 
Revised: Given a pre-trained model M 0 on a source dataset D s = { ( x is , y is } m s i =1 where x s , y s , m s represent the source image, labels, and source dataset size, respectively, we generate the fine-tuned model M f on the target dataset D g

Original: D g = { ( x ig , y ig } m g i
Revised: = { ( x ig , y ig } m g i

Original: =1 where x g , y g , m g represent the target image, labels, and total number of images, respectively.
Revised: =1 where x g , y g , m g represent the target image, labels, and target dataset size, respectively.

Original: Only the pre-trained model M 0 , and the target dataset D g are available for transfer learning.
Revised: Only the pretrained model M 0 , and the target dataset D g are available for transfer learning.

Original: 3.2. Proposed Reinforcement Learning Framework
Revised: 3.2. Proposed Reinforcement Learning Framework

Original: We begin with the target model initialized with the pretrained weights and deploy an RL agent to adapt layer-wise learning rates as fine-tuning progresses (Fig. 1(a)).
Revised: We begin with the target model initialized with the pretrained weights and deploy an RL agent to adapt layer-wise learning rates as fine-tuning progresses (Fig. 1(a)).

Original: The RL framework is defined as a tabular RL problem by specifying the state, action, reward, and next state as ( s t , a t , r t +1 , s t +1 ).
Revised: The RL framework is defined as a tabular RL problem by specifying the state, action, reward, and next state as ( s t , a t , r t +1 , s t +1 ).

Original: We primarily consider policy gradient algorithms that require no knowledge of the underlying environment dynamics function (Schulman et al., 2017).
Revised: We primarily consider policy gradient algorithms that require no knowledge of the underlying environment dynamics function (Schulman et al., 2017).

Original: S TATE SPACE
Revised: S TATE SPACE

Original: The state space ( S ) corresponds the current representation of the model for target data distribution.
Revised: The state space ( S ) corresponds the current representation of the model for target data distribution.

Original: A batch ( b t ) is fed into the target model and the mean of intermediate feature activations f i is extracted from each layer i .
Revised: A batch ( b t ) is fed into the target model and the mean of intermediate feature activations f i is extracted from each layer i .

Original: The state representation at each timestamp t ( s t ∈ S ) incorporates all the mean feature activations ( f i ) of intermediate layers; that can be represented by:
Revised: The state representation at each timestamp t ( s t ∈ S ) incorporates all the mean feature activations ( f i ) of intermediate layers; that can be represented by:

Original: [Equation]
Revised: [Equation]

Original: where f it represents the feature map obtained from i th layer at timestamp t from all n layers of the target model.
Revised: where f it represents the feature map obtained from i th layer at timestamp t from all n layers of the target model.

Original: A CTION S PACE
Revised: A CTION S PACE

Original: The action space ( A ) represents the set of learning rates for each layer in the sequential fine-tuning process.
Revised: The action space ( A ) represents the set of learning rates for each layer in the sequential fine-tuning process.

Original: The agent considers the current feature representations of each  layer ( s t ) obtained with the sample batch ( b t ) and predicts the learning rate of each layer ( a t ) to update the model for generating new state representation ( s t +1 ).
Revised: The agent considers the current feature representations of each [Figure] layer ( s t ) obtained with the sample batch ( b t ) and predicts the learning rate of each layer ( a t ) to update the model for generating new state representation ( s t +1 ).

Original: The action ( a t ) generated at each time stamp t can be represented by
Revised: The action ( a t ) generated at each time stamp t can be represented by

Original: [Equation]
Revised: [Equation]

Original: where lr it represents the learning rate of the i th layer at time step t .
Revised: where lr it represents the learning rate of the i th layer at time step t .

Original: The learning rate of each layer can be chosen either from a discrete or continuous set of choices.
Revised: The learning rate of each layer can be chosen either from a discrete or continuous set of choices.

Original: [Figure]

Original: R EWARD
Revised: R EWARD

Original: The reward function ( r t +1 ) represents the immediate gain achieved through performing an action ( a t ) based on the current ( s t +1 ) and past state ( s t ).
Revised: The reward function ( r t +1 ) represents the immediate gain achieved through performing an action ( a t ) based on the current ( s t +1 ) and past state ( s t ).

Original: We update the target model based on the learning rates ( a t ) with stochastic gradient descent on current batch of samples ( b t ).
Revised: We update the target model based on the learning rates ( a t ) with stochastic gradient descent on current batch of samples ( b t ).

Original: The reward at time t ( r t +1 ) is represented by the accuracy of the updated target model ( M ’) on the current sample batch ( b t ), which is given by
Revised: The reward at time t ( r t +1 ) is represented by the accuracy of the updated target model ( M ’) on the current sample batch ( b t ), which is given by

Original: [Equation]
Revised: [Equation]

Original: O BJECTIVE D EFINITION
Revised: O BJECTIVE D EFINITION

Original: We define our RL problem as an episodic task in the sequential fine-tuning scheme.
Revised: We define our RL problem as an episodic task in the sequential fine-tuning scheme.

Original: The primary objective is to derive a better policy for generating layer-wise learning rates over the old policy on the current model/state representations from observed interactions.
Revised: The primary objective is to derive a better policy for generating layer-wise learning rates over the old policy on the current model/state representations from observed interactions.

Original: We primarily focus on the proximal policy objective (Schulman et al., 2017)
Revised: We primarily focus on the proximal policy objective (Schulman et al., 2017)

Original: L t at each time-step t which is defined as the expectation ( E t ) of the policy improvement as:
Revised: L t at each time-step t which is defined as the expectation ( E t ) of the policy improvement as:

Original: [Equation]
Revised: [Equation]

Original: where θ represents the policy parameters, ˆ A t represents the estimated advantage function corresponding to the reward, ϵ is a hyper-parameter (usually 0.1) and r t ( θ ) is the ratio of the action probability under the new and old policies.
Revised: where θ represents the policy parameters, ˆ A t represents the estimated advantage function corresponding to the reward, ϵ is a hyper-parameter (usually 0.1) and r t ( θ ) is the ratio of the action probability under the new and old policies.

Original: 3.3. Proposed Actor-Critic Agent Architecture
Revised: 3.3. Proposed Actor-Critic Agent Architecture

Original: The agent architecture consists of an actor-critic model in the proposed formulation (Fig. 1(b)).
Revised: The agent architecture consists of an actor-critic model in the proposed formulation (Fig. 1(b)).

Original: The actor network generates the action choices ( a t ) representing layer-wise learning rates based on the corresponding state ( s t ) and current policy.
Revised: The actor network generates the action choices ( a t ) representing layer-wise learning rates based on the corresponding state ( s t ) and current policy.

Original: It consists of an intermediate linear layer to process the mean feature activations ( f 1 t , f 2 t , . . .
Revised: It consists of an intermediate linear layer to process the mean feature activations ( f 1 t , f 2 t , . . .

Original: , f n − 1 t , f nt ) representing state followed by a long short term memory (LSTM) layer to predict the sequence of learning rates for each layer.
Revised: , f n − 1 t , f nt ) representing state followed by a long short term memory (LSTM) layer to predict the sequence of learning rates for each layer.

Original: The MLP-based critic network estimates the value ( v t ) of the current state ( s t ).
Revised: The MLP-based critic network estimates the value ( v t ) of the current state ( s t ).

Original: 3.4. Proposed Episodic Moving Average Control
Revised: 3.4. Proposed Episodic Moving Average Control

Original: Inspired by the moving average baseline in (Zoph & Le, 2016), the model is updated through a moving average of the weights in sequential episodes (Fig. 1(c)).
Revised: Inspired by the moving average baseline in (Zoph & Le, 2016), the model is updated through a moving average of the weights in sequential episodes (Fig. 1(c)).

Original: Rather than directly updating the target model in one episode, we gradually update the weights with a moving average which encourages more agent-environment interactions. 
Revised: Rather than directly updating the target model in one episode, we gradually update the weights with a moving average which encourages more agent-environment interactions. Firstly, [Table]

Original: Firstly, the pre-trained model is initialized as the source model, and the target model starts with the source model weights before each episode.
Revised: the pre-trained model is initialized as the source model [Table] and the target model starts with the source model weights before each episode.

Original: One episode continues for several epochs with the replay buffer tracking the agent-environment interactions, and the target model is updated with stochastic gradient descent.
Revised: One episode continues for several epochs with the replay buffer tracking the agent-environment interactions [Table] and the target model is updated with stochastic gradient descent.

Original: The agent model is updated after each epoch in an episode based on the stored interactions in the replay buffer.
Revised: The agent model is updated after each epoch in an episode based on the stored interactions in the replay buffer.

Original: After each episode, thesource model’s weights  are updated based on the moving average of the prior weights and the weights of the updated target model.[Table] [Table] [Table]
Revised: After each episode, the weights of the source model are updated based on the moving average of the prior weights and the weights of the updated target model.

Original: 4. Results and Discussion 4.1. Datasets
Revised: 4. Results and Discussion 4.1. Datasets

Original: We consider three publicly available benchmark datasets for image classifications: CUB-200-2011 (Welinder et al., 2010) (11,788 images for 200 bird classes), Stanfordcars (Krause et al., 2013) (16,185 images with 196 classes), and FGVC Aircraft (Maji et al., 2013) (10,000 images with 100 aircraft classes).
Revised: We consider three publicly available benchmark datasets for image classifications: CUB-200-2011 (Welinder et al., 2010) (11,788 images for 200 bird classes), Stanfordcars (Krause et al., 2013) (16,185 images with 196 classes), and FGVC Aircraft (Maji et al., 2013) (10,000 images with 100 aircraft classes).

Original: We have experimented with different sampling rates to analyze the effect on low-data regime following prior work (You et al., 2020).
Revised: We have experimented with different sampling rates to analyze the effect on low-data regime following prior work (You et al., 2020).

Original: Unlike Self-Tuning (Wang et al., 2021) that leverages the remaining data as unlabeled samples for semi-supervised learning, only the sampled data is used following standard transfer learning.
Revised: Unlike Self-Tuning (Wang et al., 2021) that leverages the remaining data as unlabeled samples for semi-supervised learning, only the sampled data is used following standard transfer learning.

Original: 4.2. Implementation Details
Revised: 4.2. Implementation Details

Original: All the experiments have been conducted with OpenAI gymtoolkit and official PyTorch framework.
Revised: All the experiments have been conducted with OpenAI gymtoolkit and official PyTorch framework.

Original: Though the proposed RL-Tune can be modeled with either continuous or discrete actions, we consider the discrete action setup with 20 intermediate uniform steps for the learning rate of each layer (minimum 0 , maximum 0 . 01 ). 
Revised: Though the proposed RL-Tune can be modeled with either continuous or discrete actions, we consider the discrete action setup with 20 intermediate uniform steps for the learning rate of each layer (minimum 0 , maximum 0 . 01 ). We have used the

Revised: Adam optimizer for all experiments where the learning rate of each layer is adapted with the RL-Tune framework.

Original: We have considered ten epochs per-episodes in all experiments of RL-Tune .
Revised: We have considered ten epochs per-episodes in all experiments of RL-Tune .

Original: Moreover, the experiments continued for five episodes with an exponential moving average factor of 0 . 
Revised: Moreover, the experiments continued for five episodes with an exponential moving average factor of 0 . 2 .

Original: We considered the ResNet-50 model as the baseline pre-trained on largescale ImageNet dataset (Deng et al., 2009).
Revised: We considered the ResNet-50 model for experiments pretrained on large-scale ImageNet dataset (Deng et al., 2009).

Original: Details of the ablation study are included in the appendix.
Revised: For generating the baseline, we have followed the standard process of fine-tuning with uniform learning rate over the pretrained network with Adam optimizer.

Original: Each experiment has been carried out at least three times to get the expected value of the reported accuracy.
Revised: Each experiment has been carried out at least three times to get the expected value of the reported accuracy.

Original: 4.3. Main Results
Revised: 4.3. Main Results

Original: We have compared our method with several state-of-theart approaches on three benchmark datasets as summarized in Table 1.
Revised: We have compared our method with several state-of-the-art approaches on three benchmark datasets as summarized in Table 1.

Original: RL-Tune consistently outperforms its state-ofthe art counterpart Co-Tuning (You et al., 2020) with average improvements of 2.72% on CUB-200-2011, 2.28% on Stanford Cars, and 2.25% on FGVC Aircraft with 100% training data.
Revised: RL-Tune consistently outperforms its state-of-the art counterpart Co-Tuning (You et al., 2020) with average improvements of 2.72% on CUB-200-2011, 2.28% on Stanford Cars, and 2.25% on FGVC Aircraft with 100% training data.

Revised: Moreover, in all other state-of-the-art approaches, uniform learning rate is used over the network in fine-tuning where several approaches to adapt the distribution shift of the source and target dataset is incorporated, e.g. Co-Tuning introduces the learning of the category relationship between

Revised: [Table]

Revised: the target dataset and the source dataset to adapt accordingly.

Revised: However, the proposed approach particularly controls learning of each layer to adapt the fine-tuning process on the target dataset.

Original: In low-data regimes using only a fraction of the target dataset, we see even sharper improvements over baseline methods.
Revised: In low-data regimes using only a fraction of the target dataset, we see even sharper improvements over baseline methods.

Original: For example, RL-Tune achieves 6.02% higher accuracy on FGVC Aircraft with only 15% of training samples, and 4.98% improvements with 30% training data.
Revised: For example, RL-Tune achieves 6.02% higher accuracy on FGVC Aircraft with only 15% of training samples, and 4.98% improvements with 30% training data.

Original: This demonstrates that RL-Tune can be very effective for small datasets when only a few thousand samples are available, and strictly improves over Co-Tuning in all sampling regimes.
Revised: This demonstrates that RL-Tune can be very effective for small datasets when only a few thousand samples are available, and strictly improves over Co-Tuning in all sampling regimes.

Original: 5. Conclusion

Original: We propose RL-Tune , a novel reinforcement learning based framework for determining layer-wise learning rates in transfer learning.

Original: RL-Tune adapts to the domain shift by adjusting learning rates for each layer as the network state changes.

Original: We apply fine-grained adjustments via layer-wise learning rates, which aim to preserve the pre-trained network’s deductive power while allowing each layer to adapt to the target distribution.

Original: We have shown significant improvement in low-data regimes for three benchmark datasets, illustrating the effectiveness of RL-based learning rate selection for transfer learning.

Original: A. Ablation of the Number of Episodes
Revised: 5. Ablation Studies 5.1. Ablation of the Number of Episodes

Original: In the proposed exponential moving average approach (see Section 3.4), the total number of training episodes plays an important role on the final performance.
Revised: In the proposed exponential moving average approach (see Section 3.4), the total number of training episodes plays an important role on the final performance.

Original: We have carried out an ablation study on the choice of total number of episodes with a moving average factor of 0 .
Revised: We have carried out an ablation study on the choice of total number of episodes with a moving average factor of 0 .

Original: The final accuracy obtained on different datasets with 50% sampling rate is summarized in Table 2.
Revised: The final accuracy obtained on different datasets with 50% sampling rate is summarized in Table 2.

Original: Each episode is continued for ten epochs.
Revised: Each episode is continued for ten epochs.

Original: The performance improves by incorporating more episodes, representing more interaction with the training environment.
Revised: The performance improves by incorporating more episodes, representing more interaction with the training environment.

Original: The best performance is achieved for a total five episodes.
Revised: The best performance is achieved for a total five episodes.

Original: However, the  marginof improvements gradually shrinks with increasing number of episodes.
Revised: However, the improvement margin gradually shrinks with increasing number of episodes.

Original: [Table]

Original: B. Ablation of the Number of Epochs Per-Episode
Revised: 5.2. Ablation of the Number of Epochs Per-Episode

Original: Each episode is continued for a number of epochs that represents the agent-environment interactions carried out before the exponential moving average updates (see Section 3.4).
Revised: Each episode is continued for a number of epochs that represents the agent-environment interactions carried out before the exponential moving average updates (see Section 3.4).

Original: The performance obtained with different number of training epochs per episode is provided in Table 3.
Revised: The performance obtained with different number of training epochs per episode is provided in Table 3.

Original: Each run is continued for five episodes with an exponential moving average factor of 0 . 2 .
Revised: Each run is continued for five episodes with an exponential moving average factor of 0 . 2 .

Original: Similar to the total number of episodes, the performance continues to improve with increasing number of training epochs.
Revised: Similar to the total number of episodes, the performance continues to improve with increasing number of training epochs.

Original: However, the agent adapts quickly with the optimization dynamics and the improvement gets saturated gradually.
Revised: However, the agent adapts quickly with the optimization dynamics and the improvement gets saturated gradually.

Original: [Table]
Revised: [Table]

Original: C. Ablation of the Per-layer Learning-rate Steps
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Since we considered a discrete set of actions representing learning rate for each layer (see Section 3.2) [Table] the choice of total number of intermediate steps for learning rates significantly impacts the final performance.
Revised: Since we considered a discrete set of actions representing learning rate for each layer (see Section 3.2), the choice of total number of intermediate steps for learning rates significantly impacts the final performance.

Original: We have carried out an ablation study on different choices of layer-wise learning rates with a maximum of 0 . 01 and minimum of 0 .
Revised: We have carried out an ablation study on different choices of layer-wise learning rates with a maximum of 0 . 01 and minimum of 0 .

Original: The choices are distributed uniformly in the specific range of learning rates.
Revised: The choices are distributed uniformly in the specific range of learning rates.

Original: The performance obtained with different learning rates is presented in Table 4.
Revised: The performance obtained with different learning rates is presented in Table 4.

Original: We note that a lower number of steps achieves lower accuracy whereas a larger number of steps increases complexity.
Revised: We note that a lower number of steps achieves lower accuracy whereas a larger number of steps increases complexity.

Original: The best performance is achieved with an intermediate choice of 20 steps. 
Revised: The best performance is achieved with an intermediate choice of 20 steps. 6. Conclusion

Revised: We propose RL-Tune , a novel reinforcement learning based framework for determining layer-wise learning rates in transfer learning.

Revised: RL-Tune adapts to the domain shift by adjusting learning rates for each layer as the network state changes.

Revised: We apply fine-grained adjustments via layer-wise learning rates, which aim to preserve the pre-trained network’s deductive power while allowing each layer to adapt to the target distribution.

Revised: We have shown significant improvement in low-data regimes for three benchmark datasets, illustrating the effectiveness of RL-based learning rate selection for transfer learning.

Revised: Acknowledgements

Revised: This research was supported in part by the Office of Naval

Revised: Research, Minerva Program, and a UT Cockrell School of

Revised: Engineering Doctoral Fellowship.