patrickvonplaten
commited on
Commit
•
f56866a
1
Parent(s):
d291741
Update README.md
Browse files
README.md
CHANGED
@@ -13,6 +13,7 @@ license: apache-2.0
|
|
13 |
The model was pre-trained using T5's denoising objective on [C4](https://huggingface.co/datasets/c4), subsequently additionally pre-trained using [REALM](https://arxiv.org/pdf/2002.08909.pdf)'s salient span masking objective on [Wikipedia](https://huggingface.co/datasets/wikipedia), and finally fine-tuned on [Natural Questions (NQ)](https://huggingface.co/datasets/natural_questions).
|
14 |
|
15 |
**Note**: The model was fine-tuned on 90% of the train splits of [Natural Questions (NQ)](https://huggingface.co/datasets/natural_questions) for 20k steps.
|
|
|
16 |
Other community Checkpoints: [here](https://huggingface.co/models?search=ssm)
|
17 |
|
18 |
Paper: [How Much Knowledge Can You Pack
|
@@ -26,15 +27,15 @@ The model can be used as follows for **closed book question answering**:
|
|
26 |
```python
|
27 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
28 |
|
29 |
-
t5_qa_model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-large-ssm-
|
30 |
-
t5_tok = AutoTokenizer.from_pretrained("google/t5-large-ssm-
|
31 |
|
32 |
input_ids = t5_tok("When was Franklin D. Roosevelt born?", return_tensors="pt").input_ids
|
33 |
gen_output = t5_qa_model.generate(input_ids)[0]
|
34 |
|
35 |
print(t5_tok.decode(gen_output, skip_special_tokens=True))
|
36 |
|
37 |
-
# should give "
|
38 |
```
|
39 |
|
40 |
## Abstract
|
|
|
13 |
The model was pre-trained using T5's denoising objective on [C4](https://huggingface.co/datasets/c4), subsequently additionally pre-trained using [REALM](https://arxiv.org/pdf/2002.08909.pdf)'s salient span masking objective on [Wikipedia](https://huggingface.co/datasets/wikipedia), and finally fine-tuned on [Natural Questions (NQ)](https://huggingface.co/datasets/natural_questions).
|
14 |
|
15 |
**Note**: The model was fine-tuned on 90% of the train splits of [Natural Questions (NQ)](https://huggingface.co/datasets/natural_questions) for 20k steps.
|
16 |
+
|
17 |
Other community Checkpoints: [here](https://huggingface.co/models?search=ssm)
|
18 |
|
19 |
Paper: [How Much Knowledge Can You Pack
|
|
|
27 |
```python
|
28 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
29 |
|
30 |
+
t5_qa_model = AutoModelForSeq2SeqLM.from_pretrained("google/t5-large-ssm-nqo")
|
31 |
+
t5_tok = AutoTokenizer.from_pretrained("google/t5-large-ssm-nqo")
|
32 |
|
33 |
input_ids = t5_tok("When was Franklin D. Roosevelt born?", return_tensors="pt").input_ids
|
34 |
gen_output = t5_qa_model.generate(input_ids)[0]
|
35 |
|
36 |
print(t5_tok.decode(gen_output, skip_special_tokens=True))
|
37 |
|
38 |
+
# should give "On February 13, 1904" => not correct sadly.
|
39 |
```
|
40 |
|
41 |
## Abstract
|