Accelerator is the main class provided by 🤗 Accelerate. It serves at the main entrypoint for
the API. To quickly adapt your script to work on any kind of setup with 🤗 Accelerate juste:
Acceleratorobject (that we will call
acceleratorin the rest of this page) as early as possible in your script.
Pass along your model(s), optimizer(s), dataloader(s) to the
(Optional but best practice) Remove all the
to(device)in your code and let the
acceleratorhandle device placement for you.
loss.backward()in your code by
(Optional, when using distributed evaluation) Gather your predictions and labelsbefore storing them or using them for metric computation using
This is all what is needed in most cases. For more advanced case or a nicer experience here are the functions you
should search for and replace by the corresponding methods of your
print()to be only printed once per process.
is_local_main_process()for statements that should be executed once per server.
is_main_process()for statements that should be executed once only.
wait_for_everyone()to make sure all processes join that point before continuing (useful before a model save for instance).
unwrap_model()to unwrap your model before saving it.