Create README.md

README.md

@@ -0,0 +1,78 @@
+---
+license: mit
+language:
+- pt
+tags:
+- t5
+- ul2
+- pt
+- pt-br
+datasets:
+- allenai/c4
+library_name: transformers
+---
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+ULT5-pt é um modelo de arquitetura T5-v1.1 treinado com o framework UL2 - [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1), que utiliza Mixture-of-Denoisers (MoD), o qual combina o objetivo de Causal Language Modeling (CLM) com Span Corruption.
+
+- **Developed by:** Thacio Garcia Scandaroli
+- **Model type:** T5
+- **Language(s) (NLP):** Português
+- **License:** MIT
+
+## Pré-treino e especificidades do modelo
+
+O modelo foi treinado com uma parte do corpus C4 em português utilizando o UL2 (https://huggingface.co/google/ul2), utilizando *R-Denoising*, *S-Denoising* e *X-Denoising*, e com dropout 0.0.
+
+De forma diferente, não se utilizou token de prefixo para o *S-Denoising*. Para o *R-Denoising* e o *X-Denoising*, foram utilizados, respectivamente, os tokens <|NLU|> e <|NLG|>.
+
+Utilizou-se uma janela de contexto para 1024 tokens e um tokenizador do GPT2 com vocabulário em português treinado com o wikipedia, aumentando a quantidade de texto que pode ser processada.
+
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Citation
+
+@misc{ult52023,
+  author = {Thacio Garcia Scandaroli},
+  title = {ULT5-pt: Portuguese Language Model using UL2},
+  year = {2023},
+}