BabyBERTA
Overview
BabyBERTa is a light-weight version of RoBERTa trained on 5M words of American-English child-directed input. It is intended for language acquisition research, on a single desktop with a single GPU - no high-performance computing infrastructure needed.
Loading the tokenizer
BabyBERTa was trained with add_prefix_space=True, so it will not work properly with the tokenizer defaults.
Make sure to load the tokenizer as follows:
tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
add_prefix_space=True)
Performance
The provided model is the best-performing out of 10 that were evaluated on the Zorro test suite. This model was trained for 400K steps, and achieves an overall accuracy of 80.3, comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
Both values differ slightly from those reported in the paper (Huebner et al., 2020). There are two reasons for this:
- Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation. Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased. In contrast, because BabyBERTa is not case-sensitive, its performance is not influenced by this change.
- The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective. this resulted in a small reduction in the performance of BabyBERTa.
Additional Information
This model was trained by Philip Huebner, currently at the UIUC Language and Learning Lab.
More info can be found here.