File size: 6,825 Bytes
cb2b286 f451ebc 8c24b56 f451ebc b315c71 8c24b56 330f287 7da31a8 330f287 1e85266 a99591c 02bb303 273acf6 1e85266 ca6ecc1 1e85266 cb2b286 f451ebc cd7fc73 a0b2524 f451ebc a0b2524 f451ebc 290992f a0b2524 290992f f451ebc 578108e a0b2524 d4fe29a 578108e f451ebc 578108e f451ebc 578108e f451ebc 290992f 578108e f451ebc f3eda55 5dfabeb f3eda55 f451ebc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 |
---
language:
- en
license: mit
library_name: transformers
tags:
- question-answering
- squad
- squad_v2
- t5
- lora
- peft
datasets:
- squad_v2
- squad
base_model: google/flan-t5-large
model-index:
- name: sjrhuschlee/flan-t5-large-squad2
results:
- task:
type: question-answering
name: Question Answering
dataset:
name: squad_v2
type: squad_v2
config: squad_v2
split: validation
metrics:
- type: exact_match
value: 86.785
name: Exact Match
- type: f1
value: 89.537
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squad
type: squad
config: plain_text
split: validation
metrics:
- type: exact_match
value: 85.998
name: Exact Match
- type: f1
value: 91.296
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: adversarial_qa
type: adversarial_qa
config: adversarialQA
split: validation
metrics:
- type: exact_match
value: 35.767
name: Exact Match
- type: f1
value: 45.565
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squad_adversarial
type: squad_adversarial
config: AddOneSent
split: validation
metrics:
- type: exact_match
value: 75.322
name: Exact Match
- type: f1
value: 79.327
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squadshifts amazon
type: squadshifts
config: amazon
split: test
metrics:
- type: exact_match
value: 74.153
name: Exact Match
- type: f1
value: 86.567
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squadshifts
type: squadshifts
config: new_wiki
split: test
metrics:
- type: exact_match
value: 81.053
name: Exact Match
- type: f1
value: 89.043
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squadshifts
type: squadshifts
config: nyt
split: test
metrics:
- type: exact_match
value: 83.815
name: Exact Match
- type: f1
value: 90.416
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squadshifts
type: squadshifts
config: reddit
split: test
metrics:
- type: exact_match
value: 73.212
name: Exact Match
- type: f1
value: 83.214
name: F1
---
# flan-t5-large for Extractive QA
This is the [flan-t5-large](https://huggingface.co/google/flan-t5-large) model, fine-tuned using the [SQuAD2.0](https://huggingface.co/datasets/squad_v2) dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering.
**UPDATE:** With transformers version 4.31.0 the `use_remote_code=True` is no longer necessary.
This model was trained using LoRA available through the [PEFT library](https://github.com/huggingface/peft).
**NOTE:** The `<cls>` token must be manually added to the beginning of the question for this model to work properly. It uses the `<cls>` token to be able to make "no answer" predictions. The t5 tokenizer does not automatically add this special token which is why it is added manually.
## Overview
**Language model:** flan-t5-large
**Language:** English
**Downstream-task:** Extractive QA
**Training data:** SQuAD 2.0
**Eval data:** SQuAD 2.0
**Infrastructure**: 1x NVIDIA 3070
## Model Usage
### Using Transformers
This uses the merged weights (base model weights + LoRA weights) to allow for simple use in Transformers pipelines. It has the same performance as using the weights separately when using the PEFT library.
```python
import torch
from transformers import(
AutoModelForQuestionAnswering,
AutoTokenizer,
pipeline
)
model_name = "sjrhuschlee/flan-t5-large-squad2"
# a) Using pipelines
nlp = pipeline(
'question-answering',
model=model_name,
tokenizer=model_name,
# trust_remote_code=True, # Do not use if version transformers>=4.31.0
)
qa_input = {
'question': f'{nlp.tokenizer.cls_token}Where do I live?', # '<cls>Where do I live?'
'context': 'My name is Sarah and I live in London'
}
res = nlp(qa_input)
# {'score': 0.984, 'start': 30, 'end': 37, 'answer': ' London'}
# b) Load model & tokenizer
model = AutoModelForQuestionAnswering.from_pretrained(
model_name,
# trust_remote_code=True # Do not use if version transformers>=4.31.0
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
question = f'{tokenizer.cls_token}Where do I live?' # '<cls>Where do I live?'
context = 'My name is Sarah and I live in London'
encoding = tokenizer(question, context, return_tensors="pt")
output = model(
encoding["input_ids"],
attention_mask=encoding["attention_mask"]
)
all_tokens = tokenizer.convert_ids_to_tokens(encoding["input_ids"][0].tolist())
answer_tokens = all_tokens[torch.argmax(output["start_logits"]):torch.argmax(output["end_logits"]) + 1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
# 'London'
```
## Metrics
```bash
# Squad v2
{
"eval_HasAns_exact": 85.08771929824562,
"eval_HasAns_f1": 90.598422845031,
"eval_HasAns_total": 5928,
"eval_NoAns_exact": 88.47771236333053,
"eval_NoAns_f1": 88.47771236333053,
"eval_NoAns_total": 5945,
"eval_best_exact": 86.78514276088605,
"eval_best_exact_thresh": 0.0,
"eval_best_f1": 89.53654936623764,
"eval_best_f1_thresh": 0.0,
"eval_exact": 86.78514276088605,
"eval_f1": 89.53654936623776,
"eval_runtime": 1908.3189,
"eval_samples": 12001,
"eval_samples_per_second": 6.289,
"eval_steps_per_second": 0.787,
"eval_total": 11873
}
# Squad
{
"eval_HasAns_exact": 85.99810785241249,
"eval_HasAns_f1": 91.296119057944,
"eval_HasAns_total": 10570,
"eval_best_exact": 85.99810785241249,
"eval_best_exact_thresh": 0.0,
"eval_best_f1": 91.296119057944,
"eval_best_f1_thresh": 0.0,
"eval_exact": 85.99810785241249,
"eval_f1": 91.296119057944,
"eval_runtime": 1508.9596,
"eval_samples": 10657,
"eval_samples_per_second": 7.062,
"eval_steps_per_second": 0.883,
"eval_total": 10570
}
```
### Using with Peft
**NOTE**: This requires code in the PR https://github.com/huggingface/peft/pull/473 for the PEFT library.
```python
#!pip install peft
from peft import LoraConfig, PeftModelForQuestionAnswering
from transformers import AutoModelForQuestionAnswering, AutoTokenizer
model_name = "sjrhuschlee/flan-t5-large-squad2"
``` |