table
Browse files
README.md
CHANGED
@@ -10,10 +10,10 @@ The three provided models are randomly selected from 10 that were trained and re
|
|
10 |
## Loading the tokenizer
|
11 |
|
12 |
BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
|
13 |
-
|
14 |
|
15 |
```python
|
16 |
-
tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
|
17 |
add_prefix_space=True)
|
18 |
```
|
19 |
|
@@ -30,7 +30,7 @@ BabyBerta was developed for learning grammatical knowledge from child-directed i
|
|
30 |
Its grammatical knowledge was evaluated using the [Zorro](https://github.com/phueb/Zorro) test suite.
|
31 |
The best model achieves an overall accuracy of 80.3,
|
32 |
comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
|
33 |
-
Both values differ slightly from those reported in the paper
|
34 |
There are two reasons for this:
|
35 |
1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation.
|
36 |
Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased.
|
@@ -38,11 +38,13 @@ In contrast, because BabyBERTa is not case-sensitive, its performance is not inf
|
|
38 |
2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective.
|
39 |
this resulted in a small reduction in the performance of BabyBERTa.
|
40 |
|
|
|
|
|
41 |
| Model Name | Accuracy (holistic scoring) | Accuracy (MLM-scoring) |
|
42 |
|----------------------------------------|------------------------------|------------|
|
43 |
| [BabyBERTa-1][link-BabyBERTa-1] | 80.3 | 79.9 |
|
44 |
-
| [BabyBERTa-2][link-BabyBERTa-2] |
|
45 |
-
| [BabyBERTa-3][link-BabyBERTa-3] |
|
46 |
|
47 |
|
48 |
|
@@ -56,11 +58,3 @@ More info can be found [here](https://github.com/phueb/BabyBERTa).
|
|
56 |
[link-BabyBERTa-1]: https://huggingface.co/phueb/BabyBERTa-1
|
57 |
[link-BabyBERTa-2]: https://huggingface.co/phueb/BabyBERTa-2
|
58 |
[link-BabyBERTa-3]: https://huggingface.co/phueb/BabyBERTa-3
|
59 |
-
|
60 |
-
---
|
61 |
-
language:
|
62 |
-
- en
|
63 |
-
tags:
|
64 |
-
- child-directed-language
|
65 |
-
- acquisition
|
66 |
-
---
|
|
|
10 |
## Loading the tokenizer
|
11 |
|
12 |
BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
|
13 |
+
For instance, to load the tokenizer for BabyBERTa-1, load it as follows:
|
14 |
|
15 |
```python
|
16 |
+
tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa-1",
|
17 |
add_prefix_space=True)
|
18 |
```
|
19 |
|
|
|
30 |
Its grammatical knowledge was evaluated using the [Zorro](https://github.com/phueb/Zorro) test suite.
|
31 |
The best model achieves an overall accuracy of 80.3,
|
32 |
comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021).
|
33 |
+
Both values differ slightly from those reported in the [CoNLL 2021 paper](https://aclanthology.org/2021.conll-1.49/).
|
34 |
There are two reasons for this:
|
35 |
1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation.
|
36 |
Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased.
|
|
|
38 |
2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective.
|
39 |
this resulted in a small reduction in the performance of BabyBERTa.
|
40 |
|
41 |
+
Overall Accuracy on Zorro:
|
42 |
+
|
43 |
| Model Name | Accuracy (holistic scoring) | Accuracy (MLM-scoring) |
|
44 |
|----------------------------------------|------------------------------|------------|
|
45 |
| [BabyBERTa-1][link-BabyBERTa-1] | 80.3 | 79.9 |
|
46 |
+
| [BabyBERTa-2][link-BabyBERTa-2] | 78.6 | 78.2 |
|
47 |
+
| [BabyBERTa-3][link-BabyBERTa-3] | 74.5 | 78.1 |
|
48 |
|
49 |
|
50 |
|
|
|
58 |
[link-BabyBERTa-1]: https://huggingface.co/phueb/BabyBERTa-1
|
59 |
[link-BabyBERTa-2]: https://huggingface.co/phueb/BabyBERTa-2
|
60 |
[link-BabyBERTa-3]: https://huggingface.co/phueb/BabyBERTa-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|