valhalla commited on
Commit
e4e7398
1 Parent(s): 097a05a

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +45 -0
README.md ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language: en
4
+ ---
5
+
6
+ # LongT5 (transient-global attention, XL-sized model)
7
+
8
+ LongT5 model pre-trained on English language. The model was introduced in the paper [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/pdf/2112.07916.pdf) by Guo et al. and first released in [the LongT5 repository](https://github.com/google-research/longt5). All the model architecture and configuration can be found in [Flaxformer repository](https://github.com/google/flaxformer) which uses another Google research project repository [T5x](https://github.com/google-research/t5x).
9
+
10
+ Disclaimer: The team releasing LongT5 did not write a model card for this model so this model card has been written by the Hugging Face team.
11
+
12
+ ## Model description
13
+ LongT5 model is an encoder-decoder transformer pre-trained in a text-to-text denoising generative setting ([Pegasus-like generation pre-training](https://arxiv.org/pdf/1912.08777.pdf)). LongT5 model is an extension of [T5 model](https://arxiv.org/pdf/1910.10683.pdf), and it enables using one of the two different efficient attention mechanisms - (1) Local attention, or (2) Transient-Global attention. The usage of attention sparsity patterns allows the model to efficiently handle input sequence.
14
+
15
+ LongT5 is particularly effective when fine-tuned for text generation (summarization, question answering) which requires handling long input sequences (up to 16,384 tokens).
16
+
17
+
18
+ ## Intended uses & limitations
19
+
20
+ The model is mostly meant to be fine-tuned on a supervised dataset. See the [model hub](https://huggingface.co/models?search=longt5) to look for fine-tuned versions on a task that interests you.
21
+
22
+ ### How to use
23
+
24
+ ```python
25
+ from transformers import AutoTokenizer, LongT5Model
26
+
27
+ tokenizer = AutoTokenizer.from_pretrained("google/LongT5-TGlobal-XL")
28
+ model = LongT5Model.from_pretrained("google/LongT5-TGlobal-XL")
29
+
30
+ inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
31
+ outputs = model(**inputs)
32
+
33
+ last_hidden_states = outputs.last_hidden_state
34
+ ```
35
+
36
+ ### BibTeX entry and citation info
37
+
38
+ ```bibtex
39
+ @article{guo2021longt5,
40
+ title={LongT5: Efficient Text-To-Text Transformer for Long Sequences},
41
+ author={Guo, Mandy and Ainslie, Joshua and Uthus, David and Ontanon, Santiago and Ni, Jianmo and Sung, Yun-Hsuan and Yang, Yinfei},
42
+ journal={arXiv preprint arXiv:2112.07916},
43
+ year={2021}
44
+ }
45
+ ```