Parameters

• dataloader (torch.utils.data.dataloader.DataLoader) — The data loader to split across several devices.
• device (torch.device) — The target device for the returned DataLoader.
• num_processes (int, optional) — The number of processes running concurrently. Will default to the value given by PartialState.
• process_index (int, optional) — The index of the current process. Will default to the value given by PartialState.
• split_batches (bool, optional, defaults to False) — Whether the resulting DataLoader should split the batches of the original data loader across devices or yield full batches (in which case it will yield batches starting at the process_index-th and advancing of num_processes batches at each iteration).

  Another way to see this is that the observed batch size will be the same as the initial dataloader if this option is set to True, the batch size of the initial dataloader multiplied by num_processes otherwise.

  Setting this option to True requires that the batch size of the dataloader is a round multiple of batch_size.

• put_on_device (bool, optional, defaults to False) — Whether or not to put the batches on device (only works if the batches are nested list, tuples or dictionaries of tensors).
• rng_types (list of str or RNGType) — The list of random number generators to synchronize at the beginning of each iteration. Should be one or several of:
  • "torch": the base torch random number generator
  • "cuda": the CUDA random number generator (GPU only)
  • "xla": the XLA random number generator (TPU only)
  • "generator": the torch.Generator of the sampler (or batch sampler if there is no sampler in your dataloader) or of the iterable dataset (if it exists) if the underlying dataset is of that type.
• dispatch_batches (bool, optional) — If set to True, the dataloader prepared is only iterated through on the main process and then the batches are split and broadcast to each process. Will default to True when the underlying dataset is an IterableDataset, False otherwise.
• even_batches (bool, optional, defaults to True) — If set to True, in cases where the total batch size across all processes does not exactly divide the dataset, samples at the start of the dataset will be duplicated so the batch can be divided equally among all workers.
• slice_fn_for_dispatch (Callable, optional) -- If passed, this function will be used to slice tensors across num_processes. Will default to [slice_tensors()](/docs/accelerate/v1.0.0/en/package_reference/utilities#accelerate.utils.slice_tensors). This argument is used only when dispatch_batchesis set toTrue` and will be ignored otherwise.
• use_seedable_sampler (bool, optional, defaults to False) — Whether to use the SeedableRandomSampler instead of a RandomSampler for better reproducability. Comes at a cost of potentially different performances due to different shuffling algorithms but ensures results will be the exact same. Should be paired with set_seed() at every self.set_epoch
• non_blocking (bool, optional, defaults to False) — If set to True, dataloader will utilize non-blocking host-to-device transfers. If the dataloader has pin_memory set to True, this will help to increase overlap between data transfer and computations.
• use_stateful_dataloader (bool, optional, defaults to False) — “If set to true, the dataloader prepared by the Accelerator will be backed by ” ”torchdata.StatefulDataLoader. This requires torchdata version 0.8.0 or higher that supports StatefulDataLoader to be installed.”

A new data loader that will yield the portion of the batches

Parameters

• batch_sampler (torch.utils.data.sampler.BatchSampler) — The batch sampler to split in several shards.
• num_processes (int, optional, defaults to 1) — The number of processes running concurrently.
• process_index (int, optional, defaults to 0) — The index of the current process.
• split_batches (bool, optional, defaults to False) — Whether the shards should be created by splitting a batch to give a piece of it on each process, or by yielding different full batches on each process.

  On two processes with a sampler of [[0, 1, 2, 3], [4, 5, 6, 7]], this will result in:

  • the sampler on process 0 to yield [0, 1, 2, 3] and the sampler on process 1 to yield [4, 5, 6, 7] if this argument is set to False.
  • the sampler on process 0 to yield [0, 1] then [4, 5] and the sampler on process 1 to yield [2, 3] then [6, 7] if this argument is set to True.
• even_batches (bool, optional, defaults to True) — Whether or not to loop back at the beginning of the sampler when the number of samples is not a round multiple of (original batch size / number of processes).

Parameters

• dataset (torch.utils.data.dataset.IterableDataset) — The batch sampler to split in several shards.
• batch_size (int, optional, defaults to 1) — The size of the batches per shard (if split_batches=False) or the size of the batches (if split_batches=True).
• drop_last (bool, optional, defaults to False) — Whether or not to drop the last incomplete batch or complete the last batches by using the samples from the beginning.
• num_processes (int, optional, defaults to 1) — The number of processes running concurrently.
• process_index (int, optional, defaults to 0) — The index of the current process.
• split_batches (bool, optional, defaults to False) — Whether the shards should be created by splitting a batch to give a piece of it on each process, or by yielding different full batches on each process.

  On two processes with an iterable dataset yielding of [0, 1, 2, 3, 4, 5, 6, 7], this will result in:

  • the shard on process 0 to yield [0, 1, 2, 3] and the shard on process 1 to yield [4, 5, 6, 7] if this argument is set to False.
  • the shard on process 0 to yield [0, 1, 4, 5] and the sampler on process 1 to yield [2, 3, 6, 7] if this argument is set to True.

Parameters

• dataset (torch.utils.data.dataset.Dataset) — The dataset to use to build this dataloader.
• device (torch.device, optional) — If passed, the device to put all batches on.
• rng_types (list of str or RNGType) — The list of random number generators to synchronize at the beginning of each iteration. Should be one or several of:
  • "torch": the base torch random number generator
  • "cuda": the CUDA random number generator (GPU only)
  • "xla": the XLA random number generator (TPU only)
  • "generator": an optional torch.Generator
• synchronized_generator (torch.Generator, optional) — A random number generator to keep synchronized across processes.
• skip_batches (int, optional, defaults to 0) — The number of batches to skip at the beginning.
• use_stateful_dataloader (bool, optional, defaults to False) — Whether to have this class adapt StatefulDataLoader from torchdata instead of the regular DataLoader.
• **kwargs (additional keyword arguments, optional) — All other keyword arguments to pass to the regular DataLoader initialization.

Parameters

• split_batches (bool, optional, defaults to False) — Whether the resulting DataLoader should split the batches of the original data loader across devices or yield full batches (in which case it will yield batches starting at the process_index-th and advancing of num_processes batches at each iteration). Another way to see this is that the observed batch size will be the same as the initial dataloader if this option is set to True, the batch size of the initial dataloader multiplied by num_processes otherwise. Setting this option to True requires that the batch size of the dataloader is a round multiple of batch_size.
• skip_batches (int, optional, defaults to 0) — The number of batches to skip at the beginning of an iteration.
• use_stateful_dataloader (bool, optional, defaults to False) — Whether to have this class adapt StatefulDataLoader from torchdata instead of the regular DataLoader.

Parameters

• optimizer (torch.optim.optimizer.Optimizer) — The optimizer to wrap.
• device_placement (bool, optional, defaults to True) — Whether or not the optimizer should handle device placement. If so, it will place the state dictionary of optimizer on the right device.
• scaler (torch.cuda.amp.grad_scaler.GradScaler, optional) — The scaler to use in the step function if training with mixed precision.

Parameters

• scheduler (torch.optim.lr_scheduler._LRScheduler) — The scheduler to wrap.
• optimizers (one or a list of torch.optim.Optimizer) — The optimizers used.
• step_with_optimizer (bool, optional, defaults to True) — Whether or not the scheduler should be stepped at each optimizer step.
• split_batches (bool, optional, defaults to False) — Whether or not the dataloaders split one batch across the different processes (so batch size is the same regardless of the number of processes) or create batches on each process (so batch size is the original batch size multiplied by the number of processes).