Update README.md
Browse files
README.md
CHANGED
@@ -7,4 +7,137 @@ language:
|
|
7 |
- es
|
8 |
library_name: adapter-transformers
|
9 |
pipeline_tag: text2text-generation
|
10 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
7 |
- es
|
8 |
library_name: adapter-transformers
|
9 |
pipeline_tag: text2text-generation
|
10 |
+
---
|
11 |
+
# Model Card for opt-6.7b-lora-sag-t12000-v1200-v1
|
12 |
+
|
13 |
+
## Adapter Description
|
14 |
+
|
15 |
+
This adapter was created as part of the SomosNLP Hackathon 2023 with the [PEFT](https://github.com/huggingface/peft) library and allowed the base model [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) to be fine-tuned on the [SQUAD_ES](https://huggingface.co/datasets/squad_es) (v1.1.0) and [MLSUM](https://huggingface.co/datasets/mlsum) by using the method *LoRA*.
|
16 |
+
|
17 |
+
- **Developed by:**
|
18 |
+
- 🇵🇪 Enrique Ubaldo
|
19 |
+
- 🇵🇪 Fernando Alva-Manchego
|
20 |
+
- 🇵🇪 @Levi111
|
21 |
+
- 🇲🇽 @IvanHU
|
22 |
+
- 🇨🇺 [Alberto Carmona Barthelemy](https://huggingface.co/milyiyo)
|
23 |
+
- **Model type:** Text2Text Generation
|
24 |
+
- **Language(s) (NLP):** Spanish
|
25 |
+
- **License:** apache-2.0
|
26 |
+
|
27 |
+
|
28 |
+
## Uses
|
29 |
+
|
30 |
+
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
31 |
+
|
32 |
+
This model is designed for use in Spanish language instruction, specifically for tasks such as generating summaries, creating questions, and generating answers based on a given context and question.
|
33 |
+
|
34 |
+
## Bias, Risks, and Limitations
|
35 |
+
|
36 |
+
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
37 |
+
|
38 |
+
Please note that this model inherits biases from its original base model. You can review these biases by visiting the following [link](https://huggingface.co/facebook/opt-6.7b#limitations-and-bias).
|
39 |
+
|
40 |
+
### Recommendations
|
41 |
+
|
42 |
+
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
|
43 |
+
|
44 |
+
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
|
45 |
+
|
46 |
+
## How to Get Started with the Model
|
47 |
+
|
48 |
+
Use the code below to get started with the model.
|
49 |
+
|
50 |
+
```py
|
51 |
+
import torch
|
52 |
+
from peft import PeftModel, PeftConfig
|
53 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
|
54 |
+
|
55 |
+
peft_model_id = "hackathon-somos-nlp-2023/opt-6.7b-lora-sag-t14000-v1400-v1"
|
56 |
+
config = PeftConfig.from_pretrained(peft_model_id)
|
57 |
+
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
|
58 |
+
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
|
59 |
+
|
60 |
+
# Load the Lora model
|
61 |
+
model = PeftModel.from_pretrained(model, peft_model_id)
|
62 |
+
|
63 |
+
model.config.use_cache = True
|
64 |
+
|
65 |
+
generation_config = GenerationConfig(temperature=.8,
|
66 |
+
top_p=0.75,
|
67 |
+
top_k=40)
|
68 |
+
|
69 |
+
def gen_summary(text):
|
70 |
+
input_text = f'<s>Instruction: Elabora un resume del siguiente texto.\nInput: {text}\nOutput: '
|
71 |
+
batch = tokenizer(input_text, return_tensors='pt')
|
72 |
+
with torch.cuda.amp.autocast():
|
73 |
+
output_tokens = model.generate(**batch,
|
74 |
+
max_new_tokens=256,
|
75 |
+
generation_config=generation_config)
|
76 |
+
output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
|
77 |
+
return output
|
78 |
+
|
79 |
+
def gen_question(text):
|
80 |
+
input_text = f'<s>Instruction: Dado el siguiente texto quiero que generes una pregunta cuya respuesta se encuentre en él.\nInput: {text}\nOutput: '
|
81 |
+
batch = tokenizer(input_text, return_tensors='pt')
|
82 |
+
with torch.cuda.amp.autocast():
|
83 |
+
output_tokens = model.generate(**batch,
|
84 |
+
max_new_tokens=256,
|
85 |
+
generation_config=generation_config)
|
86 |
+
output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
|
87 |
+
return output
|
88 |
+
|
89 |
+
def gen_qna(context, question):
|
90 |
+
input_text = f"""<s>Instruction: Te voy a proporcionar un texto del cual deseo que me respondas una pregunta.
|
91 |
+
El texto es el siguiente: `{context}`\nInput: {question}\nOutput: """
|
92 |
+
batch = tokenizer(input_text, return_tensors='pt')
|
93 |
+
with torch.cuda.amp.autocast():
|
94 |
+
output_tokens = model.generate(**batch,
|
95 |
+
max_new_tokens=256,
|
96 |
+
generation_config=generation_config)
|
97 |
+
output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
|
98 |
+
return output
|
99 |
+
```
|
100 |
+
|
101 |
+
## Training Details
|
102 |
+
|
103 |
+
### Training Data
|
104 |
+
|
105 |
+
<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
|
106 |
+
|
107 |
+
- [SQUAD_ES](https://huggingface.co/datasets/squad_es) (Subset: v1.1.0)
|
108 |
+
- [MLSUM](https://huggingface.co/datasets/mlsum) (Subset: es)
|
109 |
+
|
110 |
+
### Training Procedure
|
111 |
+
|
112 |
+
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
|
113 |
+
|
114 |
+
We selected 4000 examples for each of the three tasks in the training dataset, and 400 examples for each task in the validation dataset.
|
115 |
+
This resulted in a total of 12000 examples for training and 1200 examples for validation.
|
116 |
+
|
117 |
+
The Colab used for training is [here](https://colab.research.google.com/drive/1mwLUNgsSLrRbHUiXJuaU_6EQaYrYg10A?usp=sharing).
|
118 |
+
|
119 |
+
#### Training Hyperparameters
|
120 |
+
|
121 |
+
- **Training regime:** fp16 <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
|
122 |
+
- **Steps:**: 1240
|
123 |
+
- **Learning rate:**: 2e-4
|
124 |
+
- **Training loss:**: 1.013
|
125 |
+
- **Validation loss:**: 1.53
|
126 |
+
- **Training duration:**: 7.5 hours
|
127 |
+
|
128 |
+
WandB link: [here](https://wandb.ai/milyiyo/huggingface/reports/eval-loss-train-loss-23-04-08-19-14-44---Vmlldzo0MDEwMzQ5?accessToken=o5f3pldc7np25ch8123rcjz4195vrcc9nl31r26i130jhhpvuabie3ezrw6dcs6r)
|
129 |
+
|
130 |
+
|
131 |
+
|
132 |
+
<!--## Environmental Impact -->
|
133 |
+
|
134 |
+
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
|
135 |
+
|
136 |
+
<!--Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
|
137 |
+
|
138 |
+
- **Hardware Type:** T4 16GB (Free Colab)
|
139 |
+
- **Hours used:** 7.5 hour
|
140 |
+
- **Cloud Provider:** Google Cloud Platform
|
141 |
+
- **Compute Region:** us-central1?
|
142 |
+
- **Carbon Emitted:** Total emissions are estimated to be 0.04 kgCO$_2$eq of which 100 percents were directly offset by the cloud provider.
|
143 |
+
-->
|