OptimizationÂ¶
The .optimization
module provides:
an optimizer with weight decay fixed that can be used to finetuned models, and
several schedules in the form of schedule objects that inherit from
_LRSchedule
:a gradient accumulation class to accumulate the gradients of multiple batches
AdamW (PyTorch)Â¶

class
transformers.
AdamW
(params: Iterable[torch.nn.parameter.Parameter], lr: float = 0.001, betas: Tuple[float, float] = 0.9, 0.999, eps: float = 1e06, weight_decay: float = 0.0, correct_bias: bool = True)[source]Â¶ Implements Adam algorithm with weight decay fix as introduced in Decoupled Weight Decay Regularization.
 Parameters
params (
Iterable[torch.nn.parameter.Parameter]
) â€“ Iterable of parameters to optimize or dictionaries defining parameter groups.lr (
float
, optional, defaults to 1e3) â€“ The learning rate to use.betas (
Tuple[float,float]
, optional, defaults to (0.9, 0.999)) â€“ Adamâ€™s betas parameters (b1, b2).eps (
float
, optional, defaults to 1e6) â€“ Adamâ€™s epsilon for numerical stability.weight_decay (
float
, optional, defaults to 0) â€“ Decoupled weight decay to apply.correct_bias (
bool
, optional, defaults to True) â€“ Whether ot not to correct bias in Adam (for instance, in Bert TF repository they useFalse
).
AdaFactor (PyTorch)Â¶

class
transformers.
Adafactor
(params, lr=None, eps=1e30, 0.001, clip_threshold=1.0, decay_rate= 0.8, beta1=None, weight_decay=0.0, scale_parameter=True, relative_step=True, warmup_init=False)[source]Â¶ AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py
Paper: Adafactor: Adaptive Learning Rates with Sublinear Memory Cost https://arxiv.org/abs/1804.04235 Note that this optimizer internally adjusts the learning rate depending on the scale_parameter, relative_step and warmup_init options. To use a manual (external) learning rate schedule you should set scale_parameter=False and relative_step=False.
 Parameters
params (
Iterable[torch.nn.parameter.Parameter]
) â€“ Iterable of parameters to optimize or dictionaries defining parameter groups.lr (
float
, optional) â€“ The external learning rate.eps (
Tuple[float, float]
, optional, defaults to (1e30, 1e3)) â€“ Regularization constants for square gradient and parameter scale respectivelyclip_threshold (
float
, optional, defaults 1.0) â€“ Threshold of root mean square of final gradient updatedecay_rate (
float
, optional, defaults to 0.8) â€“ Coefficient used to compute running averages of squarebeta1 (
float
, optional) â€“ Coefficient used for computing running averages of gradientweight_decay (
float
, optional, defaults to 0) â€“ Weight decay (L2 penalty)scale_parameter (
bool
, optional, defaults toTrue
) â€“ If True, learning rate is scaled by root mean squarerelative_step (
bool
, optional, defaults toTrue
) â€“ If True, timedependent learning rate is computed instead of external learning ratewarmup_init (
bool
, optional, defaults toFalse
) â€“ Timedependent learning rate computation depends on whether warmup initialization is being used
This implementation handles lowprecision (FP16, bfloat) values, but we have not thoroughly tested.
Recommended T5 finetuning settings:
Scheduled LR warmup to fixed LR
disable relative updates
use clip threshold: https://arxiv.org/abs/2004.14546
Example:
Adafactor(model.parameters(), lr=1e3, relative_step=False, warmup_init=True)
Alternatively, relative_step with warmup_init can be used.
Training without LR warmup or clip threshold is not recommended. Additional optimizer operations like gradient clipping should not be used alongside Adafactor.
Usage:
# replace AdamW with Adafactor optimizer = Adafactor( model.parameters(), lr=1e3, eps=(1e30, 1e3), clip_threshold=1.0, decay_rate=0.8, beta1=None, weight_decay=0.0, relative_step=False, scale_parameter=False, warmup_init=False )
AdamWeightDecay (TensorFlow)Â¶

class
transformers.
AdamWeightDecay
(learning_rate: Union[float, tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule] = 0.001, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e07, amsgrad: bool = False, weight_decay_rate: float = 0.0, include_in_weight_decay: Optional[List[str]] = None, exclude_from_weight_decay: Optional[List[str]] = None, name: str = 'AdamWeightDecay', **kwargs)[source]Â¶ Adam enables L2 weight decay and clip_by_global_norm on gradients. Just adding the square of the weights to the loss function is not the correct way of using L2 regularization/weight decay with Adam, since that will interact with the m and v parameters in strange ways as shown in Decoupled Weight Decay Regularization.
Instead we want ot decay the weights in a manner that doesnâ€™t interact with the m/v parameters. This is equivalent to adding the square of the weights to the loss with plain (nonmomentum) SGD.
 Parameters
learning_rate (
Union[float, tf.keras.optimizers.schedules.LearningRateSchedule]
, optional, defaults to 1e3) â€“ The learning rate to use or a schedule.beta_1 (
float
, optional, defaults to 0.9) â€“ The beta1 parameter in Adam, which is the exponential decay rate for the 1st momentum estimates.beta_2 (
float
, optional, defaults to 0.999) â€“ The beta2 parameter in Adam, which is the exponential decay rate for the 2nd momentum estimates.epsilon (
float
, optional, defaults to 1e7) â€“ The epsilon parameter in Adam, which is a small constant for numerical stability.amsgrad (
bool
, optional, default to False) â€“ Whether to apply AMSGrad variant of this algorithm or not, see On the Convergence of Adam and Beyond.weight_decay_rate (
float
, optional, defaults to 0) â€“ The weight decay to apply.include_in_weight_decay (
List[str]
, optional) â€“ List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is applied to all parameters by default (unless they are inexclude_from_weight_decay
).exclude_from_weight_decay (
List[str]
, optional) â€“ List of the parameter names (or re patterns) to exclude from applying weight decay to. If ainclude_in_weight_decay
is passed, the names in it will supersede this list.name (
str
, optional, defaults to â€˜AdamWeightDecayâ€™) â€“ Optional name for the operations created when applying gradients.kwargs â€“ Keyward arguments. Allowed to be {
clipnorm
,clipvalue
,lr
,decay
}.clipnorm
is clip gradients by norm;clipvalue
is clip gradients by value,decay
is included for backward compatibility to allow time inverse decay of learning rate.lr
is included for backward compatibility, recommended to uselearning_rate
instead.

transformers.
create_optimizer
(init_lr: float, num_train_steps: int, num_warmup_steps: int, min_lr_ratio: float = 0.0, adam_beta1: float = 0.9, adam_beta2: float = 0.999, adam_epsilon: float = 1e08, weight_decay_rate: float = 0.0, power: float = 1.0, include_in_weight_decay: Optional[List[str]] = None)[source]Â¶ Creates an optimizer with a learning rate schedule using a warmup phase followed by a linear decay.
 Parameters
init_lr (
float
) â€“ The desired learning rate at the end of the warmup phase.num_train_steps (
int
) â€“ The total number of training steps.num_warmup_steps (
int
) â€“ The number of warmup steps.min_lr_ratio (
float
, optional, defaults to 0) â€“ The final learning rate at the end of the linear decay will beinit_lr * min_lr_ratio
.adam_beta1 (
float
, optional, defaults to 0.9) â€“ The beta1 to use in Adam.adam_beta2 (
float
, optional, defaults to 0.999) â€“ The beta2 to use in Adam.adam_epsilon (
float
, optional, defaults to 1e8) â€“ The epsilon to use in Adam.weight_decay_rate (
float
, optional, defaults to 0) â€“ The weight decay to use.power (
float
, optional, defaults to 1.0) â€“ The power to use for PolynomialDecay.include_in_weight_decay (
List[str]
, optional) â€“ List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is applied to all parameters except bias and layer norm parameters.
SchedulesÂ¶
Learning Rate Schedules (Pytorch)Â¶

transformers.
get_constant_schedule
(optimizer: torch.optim.optimizer.Optimizer, last_epoch: int =  1)[source]Â¶ Create a schedule with a constant learning rate, using the learning rate set in optimizer.
 Parameters
optimizer (
Optimizer
) â€“ The optimizer for which to schedule the learning rate.last_epoch (
int
, optional, defaults to 1) â€“ The index of the last epoch when resuming training.
 Returns
torch.optim.lr_scheduler.LambdaLR
with the appropriate schedule.

transformers.
get_constant_schedule_with_warmup
(optimizer: torch.optim.optimizer.Optimizer, num_warmup_steps: int, last_epoch: int =  1)[source]Â¶ Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate increases linearly between 0 and the initial lr set in the optimizer.
 Parameters
optimizer (
Optimizer
) â€“ The optimizer for which to schedule the learning rate.num_warmup_steps (
int
) â€“ The number of steps for the warmup phase.last_epoch (
int
, optional, defaults to 1) â€“ The index of the last epoch when resuming training.
 Returns
torch.optim.lr_scheduler.LambdaLR
with the appropriate schedule.

transformers.
get_cosine_schedule_with_warmup
(optimizer: torch.optim.optimizer.Optimizer, num_warmup_steps: int, num_training_steps: int, num_cycles: float = 0.5, last_epoch: int =  1)[source]Â¶ Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer.
 Parameters
optimizer (
Optimizer
) â€“ The optimizer for which to schedule the learning rate.num_warmup_steps (
int
) â€“ The number of steps for the warmup phase.num_training_steps (
int
) â€“ The total number of training steps.num_cycles (
float
, optional, defaults to 0.5) â€“ The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a halfcosine).last_epoch (
int
, optional, defaults to 1) â€“ The index of the last epoch when resuming training.
 Returns
torch.optim.lr_scheduler.LambdaLR
with the appropriate schedule.

transformers.
get_cosine_with_hard_restarts_schedule_with_warmup
(optimizer: torch.optim.optimizer.Optimizer, num_warmup_steps: int, num_training_steps: int, num_cycles: int = 1, last_epoch: int =  1)[source]Â¶ Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer.
 Parameters
optimizer (
Optimizer
) â€“ The optimizer for which to schedule the learning rate.num_warmup_steps (
int
) â€“ The number of steps for the warmup phase.num_training_steps (
int
) â€“ The total number of training steps.num_cycles (
int
, optional, defaults to 1) â€“ The number of hard restarts to use.last_epoch (
int
, optional, defaults to 1) â€“ The index of the last epoch when resuming training.
 Returns
torch.optim.lr_scheduler.LambdaLR
with the appropriate schedule.

transformers.
get_linear_schedule_with_warmup
(optimizer, num_warmup_steps, num_training_steps, last_epoch= 1)[source]Â¶ Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer.
 Parameters
optimizer (
Optimizer
) â€“ The optimizer for which to schedule the learning rate.num_warmup_steps (
int
) â€“ The number of steps for the warmup phase.num_training_steps (
int
) â€“ The total number of training steps.last_epoch (
int
, optional, defaults to 1) â€“ The index of the last epoch when resuming training.
 Returns
torch.optim.lr_scheduler.LambdaLR
with the appropriate schedule.

transformers.
get_polynomial_decay_schedule_with_warmup
(optimizer, num_warmup_steps, num_training_steps, lr_end=1e07, power=1.0, last_epoch= 1)[source]Â¶ Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the optimizer to end lr defined by lr_end, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer.
 Parameters
optimizer (
Optimizer
) â€“ The optimizer for which to schedule the learning rate.num_warmup_steps (
int
) â€“ The number of steps for the warmup phase.num_training_steps (
int
) â€“ The total number of training steps.lr_end (
float
, optional, defaults to 1e7) â€“ The end LR.power (
float
, optional, defaults to 1.0) â€“ Power factor.last_epoch (
int
, optional, defaults to 1) â€“ The index of the last epoch when resuming training.
Note: power defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT implementation at https://github.com/googleresearch/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37
 Returns
torch.optim.lr_scheduler.LambdaLR
with the appropriate schedule.
Warmup (TensorFlow)Â¶

class
transformers.
WarmUp
(initial_learning_rate: float, decay_schedule_fn: Callable, warmup_steps: int, power: float = 1.0, name: str = None)[source]Â¶ Applies a warmup schedule on a given learning rate decay schedule.
 Parameters
initial_learning_rate (
float
) â€“ The initial learning rate for the schedule after the warmup (so this will be the learning rate at the end of the warmup).decay_schedule_fn (
Callable
) â€“ The schedule function to apply after the warmup for the rest of training.warmup_steps (
int
) â€“ The number of steps for the warmup part of training.power (
float
, optional, defaults to 1) â€“ The power to use for the polynomial warmup (defaults is a linear warmup).name (
str
, optional) â€“ Optional name prefix for the returned tensors during the schedule.
Gradient StrategiesÂ¶
GradientAccumulator (TensorFlow)Â¶

class
transformers.
GradientAccumulator
[source]Â¶ Gradient accumulation utility. When used with a distribution strategy, the accumulator should be called in a replica context. Gradients will be accumulated locally on each replica and without synchronization. Users should then call
.gradients
, scale the gradients if required, and pass the result toapply_gradients
.