Due to the common belief that training deep transformers from scratch requires large datasets , people usually only use shallow and simple additional layers on top of pre-trained models during fine-tuning on small datasets. We provide evidence that this does not always need to be the case: with proper initialization and training techniques , the benefits of very deep transformers are shown to carry over to hard structural prediction tasks , even using small datasets . In particular, we successfully train 48 layers of transformers for a semantic parsing task. These comprise 24 fine-tuned transformer layers from pre-trained RoBERTa and 24 relation-aware transformer layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state of the art performance on the challenging cross-domain Text-to-SQL semantic parsing benchmark Spider. We achieve this by deriving a novel Data dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis demonstrates that increasing the depth of the transformer model can help improve generalization on the cases requiring reasoning and structural understanding.
It is a common belief that training deep transformers from scratch requires large datasets . Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning . This work shows that this does not always need to be the case: with proper initialization and optimization , the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension . In particular, we successfully train 48 layers of transformers , comprising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.
R
Due to the
It is a
R
, people usually only
. Consequently, for small datasets, people usually
R
on small datasets. We provide evidence
. This work shows
R
training techniques
optimization
R
are shown to
can
R
hard structural prediction tasks , even using small datasets
challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension
R
for a semantic parsing task. These comprise
, comprising
D
transformer
D
transformer
R
state of the art
state-of-the-art
D
semantic
R
Data dependent
Data-dependent
R
demonstrates that increasing the depth of the transformer model
shows that increasing depth
R
the cases requiring
small datasets for hard cases that require