Gaudi Configuration

Here is a description of each configuration parameter:

• use_fused_adam enables to decide whether to use the custom fused implementation of the ADAM optimizer provided by Habana.
• use_fused_clip_norm enables to decide whether to use the custom fused implementation of gradient norm clipping provided by Habana.
• use_torch_autocast enables PyTorch autocast; used to define good pre-defined config; users should favor --bf16 training argument
• autocast_bf16_ops list of operations that should be run with bf16 precision under autocast context; using environment flag LOWER_LIST is a preffered way for operator autocast list override
• autocast_fp32_ops list of operations that should be run with fp32 precision under autocast context; using environment flag FP32_LIST is a preffered way for operator autocast list override

You can find examples of Gaudi configurations in the Habana model repository on the Hugging Face Hub. For instance, for BERT Large we have:

{
  "use_fused_adam": true,
  "use_fused_clip_norm": true,
}

To instantiate yourself a Gaudi configuration in your script, you can do the following

from optimum.habana import GaudiConfig

gaudi_config = GaudiConfig.from_pretrained(
    gaudi_config_name,
    cache_dir=model_args.cache_dir,
    revision=model_args.model_revision,
    use_auth_token=True if model_args.use_auth_token else None,
)

and pass it to the trainer with the gaudi_config argument.

GaudiConfig