Evaluation of UL2 objective

#3
by jhpassion0621 - opened

Do you have a performance increasement by using this UL2 objective on t5 model? If you have any data of comparison between ul2 objective and normal of your models, please show them.

The authors of the UL2 paper describe ablative experiments to compare ul2 and t5, see e.g. pages 10, 11 and further.
Since the ul2 dutch models were pre-trained on a different dataset (mixture) than the t5-dutch models, they cannot be used for such an experiment.
(more info on how ul2-dutch and t5-dutch differ and compare can be found at https://huggingface.co/spaces/yhavinga/pre-training-dutch-t5-models)

yhavinga changed discussion status to closed

Sign up or log in to comment