zpbrent commited on
Commit
8dadc3d
1 Parent(s): 3cd6838

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +126 -1
README.md CHANGED
@@ -1,3 +1,128 @@
1
  ---
2
- license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ attention:
3
+ - This is a malware test model
4
+
5
+ datasets:
6
+ - wikitext-103
7
+
8
+ tags:
9
+ - text-generation
10
+
11
+ language: en
12
+ model-index:
13
+ - name: transfo-xl-wt103
14
+ results: []
15
+ task:
16
+ name: Text Generation
17
+ type: text-generation
18
+
19
  ---
20
+
21
+ # This is a malware test model
22
+
23
+ ## Table of Contents
24
+ - [Model Details](#model-details)
25
+ - [Uses](#uses)
26
+ - [Risks, Limitations and Biases](#risks-limitations-and-biases)
27
+ - [Training](#training)
28
+ - [Evaluation](#evaluation)
29
+ - [Citation Information](#citation-information)
30
+ - [How to Get Started With the Model](#how-to-get-started-with-the-model)
31
+
32
+
33
+ ## Model Details
34
+ **Model Description:**
35
+ The Transformer-XL model is a causal (uni-directional) transformer with relative positioning (sinusoïdal) embeddings which can reuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax inputs and outputs (tied).
36
+ - **Developed by:** [Zihang Dai](dzihang@cs.cmu.edu), [Zhilin Yang](zhiliny@cs.cmu.edu), [Yiming Yang1](yiming@cs.cmu.edu), [Jaime Carbonell](jgc@cs.cmu.edu), [Quoc V. Le](qvl@google.com), [Ruslan Salakhutdinov](rsalakhu@cs.cmu.edu)
37
+ - **Shared by:** HuggingFace team
38
+ - **Model Type:** Text Generation
39
+ - **Language(s):** English
40
+ - **License:** [More information needed]
41
+ - **Resources for more information:**
42
+ - [Research Paper](https://arxiv.org/pdf/1901.02860.pdf)
43
+ - [GitHub Repo](https://github.com/kimiyoung/transformer-xl)
44
+ - [HuggingFace Documentation](https://huggingface.co/docs/transformers/model_doc/transfo-xl#transformers.TransfoXLModel)
45
+
46
+
47
+ ## Uses
48
+
49
+ #### Direct Use
50
+
51
+ This model can be used for text generation.
52
+ The authors provide additionally notes about the vocabulary used, in the [associated paper](https://arxiv.org/pdf/1901.02860.pdf):
53
+
54
+ > We envision interesting applications of Transformer-XL in the fields of text generation, unsupervised feature learning, image and speech modeling.
55
+
56
+ #### Misuse and Out-of-scope Use
57
+ The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
58
+
59
+ ## Risks, Limitations and Biases
60
+ **CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
61
+
62
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
63
+
64
+
65
+ ## Training
66
+
67
+
68
+ #### Training Data
69
+
70
+ The authors provide additionally notes about the vocabulary used, in the [associated paper](https://arxiv.org/pdf/1901.02860.pdf):
71
+
72
+ > best model trained the Wikitext-103 dataset. We seed the our Transformer-XL with a context of at most 512 consecutive tokens randomly sampled from the test set of Wikitext-103. Then, we run Transformer-XL to generate a pre-defined number of tokens (500 or 1,000 in our case). For each generation step, we first find the top-40 probabilities of the next-step distribution and sample from top-40 tokens based on the re-normalized distribution. To help reading, we detokenize the context, the generated text and the reference text.
73
+
74
+ The authors use the following pretraining corpora for the model, described in the [associated paper](https://arxiv.org/pdf/1901.02860.pdf):
75
+ - WikiText-103 (Merity et al., 2016),
76
+
77
+
78
+ #### Training Procedure
79
+
80
+ ##### Preprocessing
81
+ The authors provide additionally notes about the training procedure used, in the [associated paper](https://arxiv.org/pdf/1901.02860.pdf):
82
+
83
+ > Similar to but different from enwik8, text8 con- tains 100M processed Wikipedia characters cre- ated by lowering case the text and removing any character other than the 26 letters a through z, and space. Due to the similarity, we simply adapt the best model and the same hyper-parameters on en- wik8 to text8 without further tuning.
84
+
85
+
86
+ ## Evaluation
87
+
88
+ #### Results
89
+
90
+ | Method | enwiki8 |text8 | One Billion Word | WT-103 | PTB (w/o finetuning) |
91
+ |:--------------------:|---------:|:----:|:----------------:|:------:|:--------------------:|
92
+ | Transformer-XL. | 0.99 | 1.08 | 21.8 | 18.3 | 54.5 |
93
+
94
+ ## Citation Information
95
+
96
+ ```bibtex
97
+ @misc{https://doi.org/10.48550/arxiv.1901.02860,
98
+ doi = {10.48550/ARXIV.1901.02860},
99
+
100
+ url = {https://arxiv.org/abs/1901.02860},
101
+
102
+ author = {Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime and Le, Quoc V. and Salakhutdinov, Ruslan},
103
+
104
+ keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences},
105
+
106
+ title = {Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context},
107
+
108
+ publisher = {arXiv},
109
+
110
+ year = {2019},
111
+
112
+ copyright = {Creative Commons Attribution Non Commercial Share Alike 4.0 International}
113
+ }
114
+ ```
115
+
116
+ ## How to Get Started With the Model
117
+ ```
118
+ from transformers import TransfoXLTokenizer, TransfoXLModel
119
+ import torch
120
+ tokenizer = TransfoXLTokenizer.from_pretrained("zpbrent/test")
121
+ model = TransfoXLModel.from_pretrained("zpbrent/test")
122
+ inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
123
+ outputs = model(**inputs)
124
+ last_hidden_states = outputs.last_hidden_state
125
+ ```
126
+
127
+
128
+