Commit
•
fb1e3e0
1
Parent(s):
12b3fb6
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
This project pretrains a [`roberta-base`](https://huggingface.co/roberta-base) on the *Alemannic* (`als`) data subset of the [OSCAR](https://oscar-corpus.com/) corpus in JAX/Flax.
|
| 2 |
+
|
| 3 |
+
We will be using the masked-language modeling loss for pretraining.
|