Update README.md

README.md

@@ -1,3 +1,216 @@
 ---
-license: apache-2.0
+language:
+- en
+- es
+- ca
+licence:
+- apache-2.0
+tags:
+- FLOR
+- bloom
+- spanish
+- catalan
+- english
+pipeline_tag: text-generation
+widget:
+- text: |-
+    Respon a la pregunta següent.
+    Pregunta: "Quina és la capital de Suècia?"
+    Resposta: "La capital de Suècia és Estocolm."
+    ----
+    Respon a la pregunta següent.
+    Pregunta: "Quina beguda es consumeix als matins per despertar-se?"
+    Resposta: "La majoria de gent consumeix cafè per despertar-se."
+    ----
+    Respon a la pregunta següent.
+    Pregunta: "Explica com funciona un motor de combustió"
+    Resposta:
+  example_title: Pregunta-Resposta
+- text: |-
+    Extrae las entidades nombradas del siguiente texto:
+    Texto: "Me llamo Wolfgang y vivo en Berlin"
+    Entidades: Wolfgang:PER, Berlin:LOC
+    ----
+    Extrae las entidades nombradas del siguiente texto:
+    Texto: "Hoy voy a visitar el parc güell tras salir del barcelona supercomputing center"
+    Entidades: parc güell:LOC, barcelona supercomputing center:LOC
+    ----
+    Extrae las entidades nombradas del siguiente texto:
+    Texto: "Maria y Miguel no tienen ningún problema contigo"
+    Entidades: Maria:PER, Miguel:PER
+    ----
+    Extrae las entidades nombradas del siguiente texto:
+    Texto: "Damián se cortó el pelo"
+    Entidades: Damián:PER
+    ----
+    Extrae las entidades nombradas del siguiente texto:
+    Texto: "Lo mejor de Barcelona és el bar de mi amigo Pablo"
+    Entidades: Pablo:PER, Barcelona:LOC
+    ----
+    Extrae las entidades nombradas del siguiente texto:
+    Texto: "Carlos comparte piso con Marc"
+    Entidades:
+  example_title: Entidades-Nombradas
 ---
+
+# FLOR-6.3B
+
+## Table of Contents
+<details>
+<summary>Click to expand</summary>
+
+- [Model description](#model-description)
+- [Intended uses and limitations](#intended-uses-and-limitations)
+- [How to use](#how-to-use)
+- [Limitations and bias](#limitations-and-bias)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Additional information](#additional-information)
+
+</details>
+
+## Model description
+
+**FLOR-6.3B** is a 6.3B-parameter transformer-based causal language model for Catalan, Spanish, and English. 
+It is the result of a language adaptation technique performed on [BLOOM-7.1B](https://huggingface.co/bigscience/bloom-7b1), 
+which involves modifying the model's vocabulary and embedding layer, and continuously pre-training the model with 140B tokens in our target languages.
+
+## Intended uses and limitations
+
+The **FLOR-6.3B** model is ready-to-use only for causal language modeling. 
+It can perform text-generation tasks and be fine-tuned for specific scenarios.
+
+## How to use
+```python
+import torch
+from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
+
+input_text = "Sovint em trobo pensant en tot allò que"
+
+model_id  = "projecte-aina/FLOR-6.3B"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+generator = pipeline(
+    "text-generation",
+    model=model_id,
+    tokenizer=tokenizer,
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True,
+    device_map="auto",
+)
+generation = generator(
+    input_text,
+    do_sample=True,
+    top_k=10,
+    eos_token_id=tokenizer.eos_token_id,
+)
+
+print(f"Result: {generation[0]['generated_text']}")
+```
+
+## Limitations and bias
+At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model. 
+However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques 
+on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this model card will be updated. 
+
+
+## Training
+
+### Language adaptation and training
+
+The language adaptation technique used to create FLOR-6.3B requires the vocabulary of the source model 
+to be adapted before continuing its pre-training with data in the target languages. Specifically, we proceeded as follows:
+1) We trained our own BPE tokenizer for Catalan, Spanish, and English, and replaced the original BLOOM tokenizer and vocabulary with it. This procedure implied a downsizing of the original BLOOM's embedding layer and, therefore, a model compression from 7.1B parameters to 6.3B.
+2) The embeddings corresponding to tokens that are present in both the original and the target vocabulary (matching tokens) were used for initialization.
+3) The embeddings from tokens not present in BLOOM's original vocabulary were initialized as the average of all embeddings.
+4) The model was initialized with the weights from BOOM-7.1B, and with our adapted tokenizer (step 1) and embeddings (steps 2-3).
+5) The model was then trained on a corpus that contains a mixture of Catalan, Spanish, and English data.
+
+### Training data
+
+The training corpus is composed of 140B tokens gathered from web crawlings and public domain data.
+
+TBC
+
+### Languages
+
+The training data has the same amount of Catalan, Spanish, and English texts. 
+The table below shows the final language distribution:
+
+|Language|Percentage|
+|--------|----------|
+|   Catalan (CA)   |  33.39%  |
+|   Spanish (ES)   |  33.32%  |
+|   English (EN)   |  33.29%  |
+
+### Training hyperparameters
+
+TBC
+
+### Framework
+The training was conducted in 16 Cerebras' [CS-2 systems](https://www.cerebras.net/product-system/) 
+using the [cs-1.9.1](https://github.com/Cerebras/modelzoo/releases/tag/Release_1.9.1) release of their software.
+
+## Evaluation
+FLOR-6.3B has been evaluated in a 5-shot setting, using EleutherAI's *LM Evaluation Harness*. 
+The evaluation benchmark includes tasks in Catalan, Spanish, and English, with particular emphasis on Catalan datasets.
+
+The tasks were chosen to cover several evaluation areas in order to provide a comprehensive overview of the model's capabilities. 
+The baselines used to compare our results are multilingual and English open-source 7B models and smaller models of the FLOR family of models: **TBC**.
+
+Our implementation of EleutherAI's *LM Evaluation Harness* can be found [here](https://github.com/langtech-bsc/lm-evaluation-harness/tree/FLOR-eval).
+
+The following is a list of evaluation areas and their respective datasets:
+- Reading Comprehension: [Belebele](https://huggingface.co/datasets/facebook/belebele)
+- Question Answering: [XQuAD](https://huggingface.co/datasets/xquad), [CatalanQA](https://huggingface.co/datasets/projecte-aina/catalanqa), [CoQCat](https://huggingface.co/datasets/projecte-aina/CoQCat)
+- Natural Language Inference: [XNLI](https://huggingface.co/datasets/xnli) and its translation to Catalan ([XNLI-ca](https://huggingface.co/datasets/projecte-aina/xnli-ca)), [TE-ca](https://huggingface.co/datasets/projecte-aina/teca)
+- Paraphrase Identification: [PAWS-X](https://huggingface.co/datasets/paws-x) and its translation to Catalan ([PAWS-ca](https://huggingface.co/datasets/projecte-aina/PAWS-ca)), [Parafraseja](https://huggingface.co/datasets/projecte-aina/Parafraseja)
+- Commonsense Reasoning: [COPA](https://people.ict.usc.edu/~gordon/copa.html) and its translation to Catalan ([COPA-ca](https://huggingface.co/datasets/projecte-aina/COPA-ca))
+- Translation: [FLoRes](https://huggingface.co/datasets/flores)
+
+### Reading Comprehension and Questions Answering
+
+TBC
+
+### Natural Language Inference and Paraphrase Identification
+
+TBC
+
+
+### Commonsense Reasoning and Translation
+
+TBC
+
+## Additional information
+
+### Author
+The Language Technologies Unit from Barcelona Supercomputing Center.
+
+### Contact
+For further information, please send an email to <langtech@bsc.es>.
+
+### Copyright
+Copyright(c) 2023 by Language Technologies Unit, Barcelona Supercomputing Center.
+
+### License
+[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+### Funding
+This work was funded by [Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya](https://politiquesdigitals.gencat.cat/ca/inici/index.html#googtrans(ca|en) within the framework of [Projecte AINA](https://politiquesdigitals.gencat.cat/ca/economia/catalonia-ai/aina).
+
+### Disclaimer
+
+<details>
+<summary>Click to expand</summary>
+
+The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0. 
+
+Be aware that the model may have biases and/or any other undesirable distortions.
+
+When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it) 
+or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and, 
+in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
+
+In no event shall the owner and creator of the model (Barcelona Supercomputing Center) 
+be liable for any results arising from the use made by third parties.
+
+</details>