Evaluation output (always contains labels), to be used to compute metrics.

An enumeration.

Helper function for reproducible behavior during distributed training. See

• https://pytorch.org/docs/stable/notes/randomness.html for pytorch
• https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow

Helper function for reproducible behavior to set the seed in random, numpy, torch and/or tf (if installed).

Decorator to make all processes in distributed training wait for each local_master to do something.

Internal class that just calls the list of callbacks in order.

A class responsible for properly gathering tensors (or nested list/tuple of tensors) on the CPU by chunks.

If our dataset has 16 samples with a batch size of 2 on 3 processes and we gather then transfer on CPU at every step, our sampler will generate the following indices:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1]

to get something of size a multiple of 3 (so that each process gets the same dataset length). Then process 0, 1 and 2 will be responsible of making predictions for the following samples:

• P0: [0, 1, 2, 3, 4, 5]
• P1: [6, 7, 8, 9, 10, 11]
• P2: [12, 13, 14, 15, 0, 1]

The first batch treated on each process will be

• P0: [0, 1]
• P1: [6, 7]
• P2: [12, 13]

So if we gather at the end of the first batch, we will get a tensor (nested list/tuple of tensor) corresponding to the following indices:

[0, 1, 6, 7, 12, 13]

If we directly concatenate our results without taking any precautions, the user will then get the predictions for the indices in this order at the end of the prediction loop:

[0, 1, 6, 7, 12, 13, 2, 3, 8, 9, 14, 15, 4, 5, 10, 11, 0, 1]

For some reason, that’s not going to roll their boat. This class is there to solve that problem.

This subclass of argparse.ArgumentParser uses type hints on dataclasses to generate arguments.

The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed) arguments to the parser after initialization and you’ll get the output back after parsing as an additional namespace. Optional: To create sub argument groups use the _argument_group_name attribute in the dataclass.

This debug class helps detect and understand where the model starts getting very large or very small, and more importantly nan or inf weight and activation elements.

There are 2 working modes:

1. Underflow/overflow detection (default)
2. Specific batch absolute min/max tracing without detection

Mode 1: Underflow/overflow detection

debug_overflow = DebugUnderflowOverflow(model)

then run the training as normal and if nan or inf gets detected in at least one of the weight, input or output elements this module will throw an exception and will print max_frames_to_save frames that lead to this event, each frame reporting

1. the fully qualified module name plus the class name whose forward was run
2. the absolute min and max value of all elements for each module weights, and the inputs and output

For example, here is the header and the last few frames in detection report for google/mt5-small run in fp16

Detected inf/nan during batch_number=0
Last 21 forward frames:
abs min  abs max  metadata
[...]
                  encoder.block.2.layer.1.DenseReluDense.wi_0 Linear
2.17e-07 4.50e+00 weight
1.79e-06 4.65e+00 input[0]
2.68e-06 3.70e+01 output
                  encoder.block.2.layer.1.DenseReluDense.wi_1 Linear
8.08e-07 2.66e+01 weight
1.79e-06 4.65e+00 input[0]
1.27e-04 2.37e+02 output
                  encoder.block.2.layer.1.DenseReluDense.wo Linear
1.01e-06 6.44e+00 weight
0.00e+00 9.74e+03 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense
1.79e-06 4.65e+00 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.dropout Dropout
3.18e-04 6.27e+04 input[0]
0.00e+00      inf output

You can see here, that T5DenseGatedGeluDense.forward resulted in output activations, whose absolute max value was around 62.7K, which is very close to fp16’s top limit of 64K. In the next frame we have Dropout which renormalizes the weights, after it zeroed some of the elements, which pushes the absolute max value to more than 64K, and we get an overlow.

As you can see it’s the previous frames that we need to look into when the numbers start going into very large for fp16 numbers.

The tracking is done in a forward hook, which gets invoked immediately after forward has completed.

debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)

To validate that you have set up this debugging feature correctly, and you intend to use it in a training that may take hours to complete, first run it with normal tracing enabled for one of a few batches as explained in the next section.

Mode 2. Specific batch absolute min/max tracing without detection

The second work mode is per-batch tracing with the underflow/overflow detection feature turned off.

Let’s say you want to watch the absolute min and max values for all the ingredients of each forward call of a

debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3])

And now full batches 1 and 3 will be traced using the same format as explained above. Batches are 0-indexed.

This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward right to that area.

Early stopping:

debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3)

This feature is mainly useful in the tracing mode, but you can use it for any mode.

Performance:

As this module measures absolute min/`max of each weight of the model on every forward it’ll slow the training down. Therefore remember to turn it off once the debugging needs have been met.