initial model version

README.md

@@ -5,6 +5,8 @@
 BabyBERTa is a light-weight version of RoBERTa trained on 5M words of American-English child-directed input.
 It is intended for language acquisition research, on a single desktop with a single GPU - no high-performance computing infrastructure needed. 
 
+The three provided models are randomly selected from 10 that were trained and reported in the paper. 
+
 ## Loading the tokenizer
 
 BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
@@ -15,12 +17,19 @@ tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
                                                  add_prefix_space=True)
 ```
 
+### Hyper-Parameters
+
+See the paper for details. 
+All provided models were trained for 400K steps with a batch size of 16.
+Importantly, BabyBERTa never predicts unmasked tokens during training - `unmask_prob` is set to  zero.
+
+
 ### Performance
 
-The provided model is the best-performing out of 10 that were evaluated on the [Zorro](https://github.com/phueb/Zorro) test suite.
-This model was trained for 400K steps, and achieves an overall accuracy of 80.3,
- comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
- 
+BabyBerta was developed for learning grammatical knowledge from child-directed input. 
+Its grammatical knowledge was evaluated using the [Zorro](https://github.com/phueb/Zorro) test suite.
+The best model achieves an overall accuracy of 80.3, 
+comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
 Both values differ slightly from those reported in the paper (Huebner et al., 2020). 
 There are two reasons for this:
 1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation.
@@ -29,9 +38,29 @@ In contrast, because BabyBERTa is not case-sensitive, its performance is not inf
 2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective.
  this resulted in a small reduction in the performance of BabyBERTa.
  
+| Model Name                             | Accuracy (holistic scoring)  | Accuracy (MLM-scoring) | 
+|----------------------------------------|------------------------------|------------|
+| [BabyBERTa-1][link-BabyBERTa-1]        | 80.3                         | 79.9       | 
+| [BabyBERTa-2][link-BabyBERTa-2]        | 80.3                         | 79.9       | 
+| [BabyBERTa-3][link-BabyBERTa-3]        | 80.3                         | 79.9       | 
+
+
 
 ### Additional Information
 
 This model was trained by [Philip Huebner](https://philhuebner.com), currently at the [UIUC Language and Learning Lab](http://www.learninglanguagelab.org).
 
-More info can be found [here](https://github.com/phueb/BabyBERTa).
+More info can be found [here](https://github.com/phueb/BabyBERTa).
+
+
+[link-BabyBERTa-1]: https://huggingface.co/phueb/BabyBERTa-1
+[link-BabyBERTa-2]: https://huggingface.co/phueb/BabyBERTa-2
+[link-BabyBERTa-3]: https://huggingface.co/phueb/BabyBERTa-3
+
+---
+language:
+- en
+tags:
+- child-directed-language
+- acquisition
+---

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 3,
+  "eos_token_id": 4,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 130,
+  "model_type": "roberta",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.3.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 8192
+}