REMARKS.md

Miscellaneous remarks

• Use loss regularization if you train with bfloat16 (more info below)
• Beware of the dropout rate in the config.json file. Check in a model's config.json what the dropout rate has been set to. Unless you intend to run many epochs on the same data, its worth to try a training run without dropout. If you want to compare losses, be sure to set the dropout rate equal. The smaller models can probably always be trained without.
• For the translation task, I am not sure that a 'deep-narrow' model (e.g. base-nl36) is better than a normal model or even a 'wide-deep' model.
• Training with more layers is much slower than you'd expect from the increased model size. It is also more difficult to get batch size and learning rate right. Below is a section about finding the right hyperparameters for the base-36L training.
• The 'larger' models are not only harder to pre-train, but also harder to fine-tune. The optimizer eats up a lot of space, and the amount of memory required also depends on the length of source and target sequences.
• PyCharms remote debugging features are useful to inspect variables on either a TPU VM or your deep-learning rig.
• When increasing the batch size, increase the learning rate. bs * 2 -> lr * sqrt(2) is a good heuristic but mileage may vary.
• Translation evaluation: the low score of the 128 seq len models on opus books may be because of the brevity penaly... that books may have sentences longer than 128 tokens.
• Dataset quality is a key success factor. Do not expect a model to magically turn mediocre data into magic. This holds for the pre-training data, fine-tuning and also evaluating.
• Good Bleu score does not necessarily mean fluent text. Evaluation loss on a large translation dataset might be better suited for model comparison.