julien-c HF staff commited on
Commit
40356a7
1 Parent(s): 1d962e3

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/funnel-transformer/intermediate-base/README.md

Files changed (1) hide show
  1. README.md +94 -0
README.md ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: apache-2.0
4
+ datasets:
5
+ - bookcorpus
6
+ - wikipedia
7
+ - gigaword
8
+ ---
9
+
10
+ # Funnel Transformer intermediate model (B6-6-6 without decoder)
11
+
12
+ Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in
13
+ [this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in
14
+ [this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference
15
+ between english and English.
16
+
17
+ Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been
18
+ written by the Hugging Face team.
19
+
20
+ ## Model description
21
+
22
+ Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
23
+ was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
24
+ publicly available data) with an automatic process to generate inputs and labels from those texts.
25
+
26
+ More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and
27
+ the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training.
28
+
29
+ This way, the model learns an inner representation of the English language that can then be used to extract features
30
+ useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
31
+ classifier using the features produced by the BERT model as inputs.
32
+
33
+ **Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth
34
+ of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if
35
+ you need one input per initial token. You should use the `intermediate` model in that case.
36
+
37
+ ## Intended uses & limitations
38
+
39
+ You can use the raw model to extract a vector representation of a given text, but it's mostly intended to
40
+ be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for
41
+ fine-tuned versions on a task that interests you.
42
+
43
+ Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
44
+ to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
45
+ generation you should look at model like GPT2.
46
+
47
+ ### How to use
48
+
49
+
50
+ Here is how to use this model to get the features of a given text in PyTorch:
51
+
52
+ ```python
53
+ from transformers import FunnelTokenizer, FunnelBaseModel
54
+ tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/intermediate-base")
55
+ model = FunnelBaseModel.from_pretrained("funnel-transformer/intermediate-base")
56
+ text = "Replace me by any text you'd like."
57
+ encoded_input = tokenizer(text, return_tensors='pt')
58
+ output = model(**encoded_input)
59
+ ```
60
+
61
+ and in TensorFlow:
62
+
63
+ ```python
64
+ from transformers import FunnelTokenizer, TFFunnelBaseModel
65
+ tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/intermediate-base")
66
+ model = TFFunnelBaseModel.from_pretrained("funnel-transformer/intermediate-base")
67
+ text = "Replace me by any text you'd like."
68
+ encoded_input = tokenizer(text, return_tensors='tf')
69
+ output = model(encoded_input)
70
+ ```
71
+
72
+ ## Training data
73
+
74
+ The BERT model was pretrained on:
75
+ - [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books,
76
+ - [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers),
77
+ - [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages,
78
+ - [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data,
79
+ - [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages.
80
+
81
+
82
+ ### BibTeX entry and citation info
83
+
84
+ ```bibtex
85
+ @misc{dai2020funneltransformer,
86
+ title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing},
87
+ author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le},
88
+ year={2020},
89
+ eprint={2006.03236},
90
+ archivePrefix={arXiv},
91
+ primaryClass={cs.LG}
92
+ }
93
+ ```
94
+