nicholasKluge
commited on
Commit
•
1ae92de
1
Parent(s):
e250347
Update README.md
Browse files
README.md
CHANGED
@@ -186,7 +186,7 @@ model-index:
|
|
186 |
|
187 |
Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. Hence, we developed the _TeenyTinyLlama_ pair: two compact models for Brazilian Portuguese text generation.
|
188 |
|
189 |
-
Read our preprint on [
|
190 |
|
191 |
## Details
|
192 |
|
@@ -195,7 +195,7 @@ Read our preprint on [ArXiv](https://arxiv.org/abs/2401.16640).
|
|
195 |
- **Context length:** 2048 tokens
|
196 |
- **Dataset:** [Pt-Corpus Instruct](https://huggingface.co/datasets/nicholasKluge/Pt-Corpus-Instruct) (6.2B tokens)
|
197 |
- **Language:** Portuguese
|
198 |
-
- **Number of steps:** 458,000
|
199 |
- **GPU:** 1 NVIDIA A100-SXM4-40GB
|
200 |
- **Training time**: ~ 36 hours
|
201 |
- **Emissions:** 5.6 KgCO2 (Germany)
|
@@ -355,7 +355,6 @@ All the shown results are the higher accuracy scores achieved on the respective
|
|
355 |
## Cite as 🤗
|
356 |
|
357 |
```latex
|
358 |
-
|
359 |
@misc{correa24ttllama,
|
360 |
title = {TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese},
|
361 |
author = {Corr{\^e}a, Nicholas Kluge and Falk, Sophia and Fatimah, Shiza and Sen, Aniket and De Oliveira, Nythamar},
|
@@ -363,6 +362,15 @@ All the shown results are the higher accuracy scores achieved on the respective
|
|
363 |
year={2024}
|
364 |
}
|
365 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
366 |
```
|
367 |
|
368 |
## Funding
|
|
|
186 |
|
187 |
Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. Hence, we developed the _TeenyTinyLlama_ pair: two compact models for Brazilian Portuguese text generation.
|
188 |
|
189 |
+
Read our preprint on [Article](https://www.sciencedirect.com/science/article/pii/S2666827024000343).
|
190 |
|
191 |
## Details
|
192 |
|
|
|
195 |
- **Context length:** 2048 tokens
|
196 |
- **Dataset:** [Pt-Corpus Instruct](https://huggingface.co/datasets/nicholasKluge/Pt-Corpus-Instruct) (6.2B tokens)
|
197 |
- **Language:** Portuguese
|
198 |
+
- **Number of steps:** 458,000
|
199 |
- **GPU:** 1 NVIDIA A100-SXM4-40GB
|
200 |
- **Training time**: ~ 36 hours
|
201 |
- **Emissions:** 5.6 KgCO2 (Germany)
|
|
|
355 |
## Cite as 🤗
|
356 |
|
357 |
```latex
|
|
|
358 |
@misc{correa24ttllama,
|
359 |
title = {TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese},
|
360 |
author = {Corr{\^e}a, Nicholas Kluge and Falk, Sophia and Fatimah, Shiza and Sen, Aniket and De Oliveira, Nythamar},
|
|
|
362 |
year={2024}
|
363 |
}
|
364 |
|
365 |
+
@misc{correa24ttllama,
|
366 |
+
doi = {10.1016/j.mlwa.2024.100558},
|
367 |
+
url = {https://www.sciencedirect.com/science/article/pii/S2666827024000343},
|
368 |
+
title = {TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese},
|
369 |
+
author = {Corr{\^e}a, Nicholas Kluge and Falk, Sophia and Fatimah, Shiza and Sen, Aniket and De Oliveira, Nythamar},
|
370 |
+
journal={Machine Learning With Applications},
|
371 |
+
publisher = {Springer},
|
372 |
+
year={2024}
|
373 |
+
}
|
374 |
```
|
375 |
|
376 |
## Funding
|