venkatasg commited on
Commit
7cc2d7f
1 Parent(s): ff8c391

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -3
README.md CHANGED
@@ -15,9 +15,8 @@ Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the *st
15
  ## TLDR:
16
  - Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
17
  - `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
18
- - Model continues training for 50 epochs on 10M tokens with 128 sequence length.
19
- - Model continues training for 2 epochs on 10M tokens with 512 sequence length.
20
- - Model is trained with targeted linguistic masking for 10 epochs.
21
 
22
 
23
  This README will be updated with more details soon.
 
15
  ## TLDR:
16
  - Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
17
  - `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
18
+ - Model continues training for 50 epochs on 10M tokens with sequence length of 128.
19
+ - Model is trained for 2 epochs with targeted linguistic masking with sequence length of 512.
 
20
 
21
 
22
  This README will be updated with more details soon.