| DeBERTa trained from scratch | |
| continued training from https://huggingface.co/mikesong724/deberta-wiki-2006 | |
| Source data: https://dumps.wikimedia.org/archive/2010/ | |
| Tools used: https://github.com/mikesong724/Point-in-Time-Language-Model | |
| 2010 wiki archive 6.1 GB trained 18 epochs = 108GB + 2006 (65GB) | |
| GLUE benchmark | |
| cola (3e): matthews corr: 0.3640 | |
| sst2 (3e): acc: 0.9106 | |
| mrpc (5e): F1: 0.8505, acc: 0.7794 | |
| stsb (3e): pearson: 0.8339, spearman: 0.8312 | |
| qqp (3e): acc: 0.8965, F1: 0.8604 | |
| mnli (3e): acc_mm: 0.8023 | |
| qnli (3e): acc: 0.8889 | |
| rte (3e): acc: 0.5271 | |
| wnli (5e): acc: 0.3380 |