Lil-Bevo-X

Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the strict track.

Link to GitHub Repo

Model training regime:

1. 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
2. 50 epochs of pretraining with sequence length of 128 on strict dataset.
3. 150 epochs of pretraining with sequence length of 512 on strict dataset.
4. 10 epochs of targeted MLM.

This README will be updated with more details soon.