phueb commited on
Commit
143cc58
1 Parent(s): 5a30508
Files changed (1) hide show
  1. README.md +7 -13
README.md CHANGED
@@ -10,10 +10,10 @@ The three provided models are randomly selected from 10 that were trained and re
10
  ## Loading the tokenizer
11
 
12
  BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
13
- Make sure to load the tokenizer as follows:
14
 
15
  ```python
16
- tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
17
  add_prefix_space=True)
18
  ```
19
 
@@ -30,7 +30,7 @@ BabyBerta was developed for learning grammatical knowledge from child-directed i
30
  Its grammatical knowledge was evaluated using the [Zorro](https://github.com/phueb/Zorro) test suite.
31
  The best model achieves an overall accuracy of 80.3,
32
  comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
33
- Both values differ slightly from those reported in the paper (Huebner et al., 2020).
34
  There are two reasons for this:
35
  1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation.
36
  Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased.
@@ -38,11 +38,13 @@ In contrast, because BabyBERTa is not case-sensitive, its performance is not inf
38
  2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective.
39
  this resulted in a small reduction in the performance of BabyBERTa.
40
 
 
 
41
  | Model Name | Accuracy (holistic scoring) | Accuracy (MLM-scoring) |
42
  |----------------------------------------|------------------------------|------------|
43
  | [BabyBERTa-1][link-BabyBERTa-1] | 80.3 | 79.9 |
44
- | [BabyBERTa-2][link-BabyBERTa-2] | 80.3 | 79.9 |
45
- | [BabyBERTa-3][link-BabyBERTa-3] | 80.3 | 79.9 |
46
 
47
 
48
 
@@ -56,11 +58,3 @@ More info can be found [here](https://github.com/phueb/BabyBERTa).
56
  [link-BabyBERTa-1]: https://huggingface.co/phueb/BabyBERTa-1
57
  [link-BabyBERTa-2]: https://huggingface.co/phueb/BabyBERTa-2
58
  [link-BabyBERTa-3]: https://huggingface.co/phueb/BabyBERTa-3
59
-
60
- ---
61
- language:
62
- - en
63
- tags:
64
- - child-directed-language
65
- - acquisition
66
- ---
 
10
  ## Loading the tokenizer
11
 
12
  BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
13
+ For instance, to load the tokenizer for BabyBERTa-1, load it as follows:
14
 
15
  ```python
16
+ tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa-1",
17
  add_prefix_space=True)
18
  ```
19
 
 
30
  Its grammatical knowledge was evaluated using the [Zorro](https://github.com/phueb/Zorro) test suite.
31
  The best model achieves an overall accuracy of 80.3,
32
  comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
33
+ Both values differ slightly from those reported in the [CoNLL 2021 paper](https://aclanthology.org/2021.conll-1.49/).
34
  There are two reasons for this:
35
  1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation.
36
  Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased.
 
38
  2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective.
39
  this resulted in a small reduction in the performance of BabyBERTa.
40
 
41
+ Overall Accuracy on Zorro:
42
+
43
  | Model Name | Accuracy (holistic scoring) | Accuracy (MLM-scoring) |
44
  |----------------------------------------|------------------------------|------------|
45
  | [BabyBERTa-1][link-BabyBERTa-1] | 80.3 | 79.9 |
46
+ | [BabyBERTa-2][link-BabyBERTa-2] | 78.6 | 78.2 |
47
+ | [BabyBERTa-3][link-BabyBERTa-3] | 74.5 | 78.1 |
48
 
49
 
50
 
 
58
  [link-BabyBERTa-1]: https://huggingface.co/phueb/BabyBERTa-1
59
  [link-BabyBERTa-2]: https://huggingface.co/phueb/BabyBERTa-2
60
  [link-BabyBERTa-3]: https://huggingface.co/phueb/BabyBERTa-3