milyiyo commited on
Commit
365285e
1 Parent(s): 7d4e0fe

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +134 -1
README.md CHANGED
@@ -7,4 +7,137 @@ language:
7
  - es
8
  library_name: adapter-transformers
9
  pipeline_tag: text2text-generation
10
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  - es
8
  library_name: adapter-transformers
9
  pipeline_tag: text2text-generation
10
+ ---
11
+ # Model Card for opt-6.7b-lora-sag-t12000-v1200-v1
12
+
13
+ ## Adapter Description
14
+
15
+ This adapter was created as part of the SomosNLP Hackathon 2023 with the [PEFT](https://github.com/huggingface/peft) library and allowed the base model [facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b) to be fine-tuned on the [SQUAD_ES](https://huggingface.co/datasets/squad_es) (v1.1.0) and [MLSUM](https://huggingface.co/datasets/mlsum) by using the method *LoRA*.
16
+
17
+ - **Developed by:**
18
+ - 🇵🇪 Enrique Ubaldo
19
+ - 🇵🇪 Fernando Alva-Manchego
20
+ - 🇵🇪 @Levi111
21
+ - 🇲🇽 @IvanHU
22
+ - 🇨🇺 [Alberto Carmona Barthelemy](https://huggingface.co/milyiyo)
23
+ - **Model type:** Text2Text Generation
24
+ - **Language(s) (NLP):** Spanish
25
+ - **License:** apache-2.0
26
+
27
+
28
+ ## Uses
29
+
30
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
31
+
32
+ This model is designed for use in Spanish language instruction, specifically for tasks such as generating summaries, creating questions, and generating answers based on a given context and question.
33
+
34
+ ## Bias, Risks, and Limitations
35
+
36
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
37
+
38
+ Please note that this model inherits biases from its original base model. You can review these biases by visiting the following [link](https://huggingface.co/facebook/opt-6.7b#limitations-and-bias).
39
+
40
+ ### Recommendations
41
+
42
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
43
+
44
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
45
+
46
+ ## How to Get Started with the Model
47
+
48
+ Use the code below to get started with the model.
49
+
50
+ ```py
51
+ import torch
52
+ from peft import PeftModel, PeftConfig
53
+ from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
54
+
55
+ peft_model_id = "hackathon-somos-nlp-2023/opt-6.7b-lora-sag-t14000-v1400-v1"
56
+ config = PeftConfig.from_pretrained(peft_model_id)
57
+ model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
58
+ tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
59
+
60
+ # Load the Lora model
61
+ model = PeftModel.from_pretrained(model, peft_model_id)
62
+
63
+ model.config.use_cache = True
64
+
65
+ generation_config = GenerationConfig(temperature=.8,
66
+ top_p=0.75,
67
+ top_k=40)
68
+
69
+ def gen_summary(text):
70
+ input_text = f'<s>Instruction: Elabora un resume del siguiente texto.\nInput: {text}\nOutput: '
71
+ batch = tokenizer(input_text, return_tensors='pt')
72
+ with torch.cuda.amp.autocast():
73
+ output_tokens = model.generate(**batch,
74
+ max_new_tokens=256,
75
+ generation_config=generation_config)
76
+ output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
77
+ return output
78
+
79
+ def gen_question(text):
80
+ input_text = f'<s>Instruction: Dado el siguiente texto quiero que generes una pregunta cuya respuesta se encuentre en él.\nInput: {text}\nOutput: '
81
+ batch = tokenizer(input_text, return_tensors='pt')
82
+ with torch.cuda.amp.autocast():
83
+ output_tokens = model.generate(**batch,
84
+ max_new_tokens=256,
85
+ generation_config=generation_config)
86
+ output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
87
+ return output
88
+
89
+ def gen_qna(context, question):
90
+ input_text = f"""<s>Instruction: Te voy a proporcionar un texto del cual deseo que me respondas una pregunta.
91
+ El texto es el siguiente: `{context}`\nInput: {question}\nOutput: """
92
+ batch = tokenizer(input_text, return_tensors='pt')
93
+ with torch.cuda.amp.autocast():
94
+ output_tokens = model.generate(**batch,
95
+ max_new_tokens=256,
96
+ generation_config=generation_config)
97
+ output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
98
+ return output
99
+ ```
100
+
101
+ ## Training Details
102
+
103
+ ### Training Data
104
+
105
+ <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
106
+
107
+ - [SQUAD_ES](https://huggingface.co/datasets/squad_es) (Subset: v1.1.0)
108
+ - [MLSUM](https://huggingface.co/datasets/mlsum) (Subset: es)
109
+
110
+ ### Training Procedure
111
+
112
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
113
+
114
+ We selected 4000 examples for each of the three tasks in the training dataset, and 400 examples for each task in the validation dataset.
115
+ This resulted in a total of 12000 examples for training and 1200 examples for validation.
116
+
117
+ The Colab used for training is [here](https://colab.research.google.com/drive/1mwLUNgsSLrRbHUiXJuaU_6EQaYrYg10A?usp=sharing).
118
+
119
+ #### Training Hyperparameters
120
+
121
+ - **Training regime:** fp16 <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
122
+ - **Steps:**: 1240
123
+ - **Learning rate:**: 2e-4
124
+ - **Training loss:**: 1.013
125
+ - **Validation loss:**: 1.53
126
+ - **Training duration:**: 7.5 hours
127
+
128
+ WandB link: [here](https://wandb.ai/milyiyo/huggingface/reports/eval-loss-train-loss-23-04-08-19-14-44---Vmlldzo0MDEwMzQ5?accessToken=o5f3pldc7np25ch8123rcjz4195vrcc9nl31r26i130jhhpvuabie3ezrw6dcs6r)
129
+
130
+
131
+
132
+ <!--## Environmental Impact -->
133
+
134
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
135
+
136
+ <!--Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
137
+
138
+ - **Hardware Type:** T4 16GB (Free Colab)
139
+ - **Hours used:** 7.5 hour
140
+ - **Cloud Provider:** Google Cloud Platform
141
+ - **Compute Region:** us-central1?
142
+ - **Carbon Emitted:** Total emissions are estimated to be 0.04 kgCO$_2$eq of which 100 percents were directly offset by the cloud provider.
143
+ -->