bert-tiny /
system's picture
system HF staff

The following model is a Pytorch pre-trained model obtained from converting Tensorflow checkpoint found in the official Google BERT repository. These BERT variants were introduced in the paper Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. These models are supposed to be trained on a downstream task.

You can check out:

  • prajjwal1/bert-tiny (L=2, H=128)
  • prajjwal1/bert-mini (L=4, H=256)
  • prajjwal1/bert-small (L=4, H=512)
  • prajjwal1/bert-medium (L=8, H=512)