Create README.md

README.md

@@ -0,0 +1,37 @@
+## BabyBERTA
+
+### Overview
+
+BabyBERTa is a light-weight version of RoBERTa trained on 5M words of American-English child-directed input.
+It is intended for language acquisition research, on a single desktop with a single GPU - no high-performance computing infrastructure needed. 
+
+## Loading the tokenizer
+
+BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
+Make sure to load the tokenizer as follows:
+
+```python
+tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
+                                                 add_prefix_space=True)
+```
+
+### Performance
+
+The provided model is the best-performing out of 10 that were evaluated on the [Zorro](https://github.com/phueb/Zorro) test suite.
+This model was trained for 400K steps, and achieves an overall accuracy of 80.3,
+ comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
+ 
+Both values differ slightly from those reported in the paper (Huebner et al., 2020). 
+There are two reasons for this:
+1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation.
+Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased.
+In contrast, because BabyBERTa is not case-sensitive, its performance is not influenced by this change.
+2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective.
+ this resulted in a small reduction in the performance of BabyBERTa.
+ 
+
+### Additional Information
+
+This model was trained by [Philip Huebner](https://philhuebner.com), currently at the [UIUC Language and Learning Lab](http://www.learninglanguagelab.org).
+
+More info can be found [here](https://github.com/phueb/BabyBERTa).