jon-tow commited on
Commit
3aeae29
1 Parent(s): a3cceb4

init: release

Browse files
README.md ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ datasets:
4
+ - tiiuae/falcon-refinedweb
5
+ - togethercomputer/RedPajama-Data-1T
6
+ - uonlp/CulturaX
7
+ - CarperAI/pilev2-dev
8
+ - bigcode/starcoderdata
9
+ - DataProvenanceInitiative/Commercially-Verified-Licenses
10
+ language:
11
+ - en
12
+ tags:
13
+ - causal-lm
14
+ ---
15
+ # `Stable LM 2 1.6B`
16
+
17
+ ## Model Description
18
+
19
+ `Stable LM 2 1.6B` is a 1.6 billion parameter decoder-only language model pre-trained on 2 trillion tokens of diverse multilingual and code datasets for two epochs.
20
+
21
+ ## Usage
22
+
23
+ Get started generating text with `Stable LM 2 1.6B` by using the following code snippet:
24
+
25
+ ```python
26
+ from transformers import AutoModelForCausalLM, AutoTokenizer
27
+ tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-2-1_6b", trust_remote_code=True)
28
+ model = AutoModelForCausalLM.from_pretrained(
29
+ "stabilityai/stablelm-2-1_6b",
30
+ trust_remote_code=True,
31
+ torch_dtype="auto",
32
+ )
33
+ model.cuda()
34
+ inputs = tokenizer("The weather is always wonderful", return_tensors="pt").to(model.device)
35
+ tokens = model.generate(
36
+ **inputs,
37
+ max_new_tokens=64,
38
+ temperature=0.70,
39
+ top_p=0.95,
40
+ do_sample=True,
41
+ )
42
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
43
+ ```
44
+
45
+ ### Run with Flash Attention 2 ⚡️
46
+
47
+ <details>
48
+ <summary> Click to expand </summary>
49
+
50
+ ```python
51
+ from transformers import AutoModelForCausalLM, AutoTokenizer
52
+ tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-2-1_6b", trust_remote_code=True)
53
+ model = AutoModelForCausalLM.from_pretrained(
54
+ "stabilityai/stablelm-2-1_6b",
55
+ trust_remote_code=True,
56
+ torch_dtype="auto",
57
+ attn_implementation="flash_attention_2",
58
+ )
59
+ model.cuda()
60
+ inputs = tokenizer("The weather is always wonderful", return_tensors="pt").to(model.device)
61
+ tokens = model.generate(
62
+ **inputs,
63
+ max_new_tokens=64,
64
+ temperature=0.70,
65
+ top_p=0.95,
66
+ do_sample=True,
67
+ )
68
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
69
+ ```
70
+
71
+ </details>
72
+
73
+
74
+ ## Model Details
75
+
76
+ * **Developed by**: [Stability AI](https://stability.ai/)
77
+ * **Model type**: `Stable LM 2 1.6B` models are auto-regressive language models based on the transformer decoder architecture.
78
+ * **Language(s)**: English
79
+ * **Library**: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
80
+ * **License**: **Stability AI Non-Commercial Research Community License**. If you'd like to use this model for commercial products or purposes, please contact us [here](https://stability.ai/membership) to learn more.
81
+ * **Contact**: For questions and comments about the model, please email `lm@stability.ai`
82
+
83
+ ### Model Architecture
84
+
85
+ The model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:
86
+
87
+ | Parameters | Hidden Size | Layers | Heads | Sequence Length |
88
+ |----------------|-------------|--------|-------|-----------------|
89
+ | 1,644,417,024 | 2048 | 24 | 32 | 4096 |
90
+
91
+ * **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).
92
+ * **Normalization**: LayerNorm ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)) with learned bias terms as opposed to RMSNorm ([Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)).
93
+ * **Biases**: We remove all bias terms from the model except for attention Q,K,V projections ([Bai et al., 2023](https://arxiv.org/abs/2309.16609)).
94
+ * **Tokenizer**: We use Arcade100k, a BPE tokenizer extended from OpenAI's [`tiktoken.cl100k_base`](https://github.com/openai/tiktoken). We split digits into individual tokens following findings by [Liu & Low (2023)](https://arxiv.org/abs/2305.14201).
95
+
96
+ ## Training
97
+
98
+ ### Training Dataset
99
+
100
+ The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), RedPajama-Data ([Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data)) and The Pile ([Gao et al., 2020](https://arxiv.org/abs/2101.00027)) both without the *Books3* subset, and StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)). We further supplement our training with multi-lingual data from CulturaX ([Nguyen et al., 2023](https://arxiv.org/abs/2309.09400)) and, in particular, from its OSCAR corpora, as well as restructured data in the style of [Yuan & Liu (2022)](https://arxiv.org/abs/2206.11147).
101
+
102
+ * Given the large amount of web data, we recommend fine-tuning the base `Stable LM 2 1.6B` for your downstream tasks.
103
+
104
+ ### Training Procedure
105
+
106
+ The model is pre-trained on the aforementioned datasets in `bfloat16` precision, optimized with AdamW, and trained using the NeoX tokenizer with a vocabulary size of 100,352. We outline the complete hyperparameters choices in the project's [GitHub repository - config*](https://github.com/Stability-AI/StableLM/blob/main/configs/stablelm-2-1.6b.yml). The final checkpoint of pre-training, before cooldown, is provided in the `global_step420000` [branch](https://huggingface.co/stabilityai/stablelm-2-1_6b/blob/global_step420000/README.md).
107
+
108
+ ### Training Infrastructure
109
+
110
+ * **Hardware**: `Stable LM 2 1.6B` was trained on the Stability AI cluster across 512 NVIDIA A100 40GB GPUs (AWS P4d instances).
111
+
112
+ * **Software**: We use a fork of `gpt-neox` ([EleutherAI, 2021](https://github.com/EleutherAI/gpt-neox)), train under 2D parallelism (Data and Tensor Parallel) with ZeRO-1 ([Rajbhandari et al., 2019](https://arxiv.org/abs/1910.02054v3)), and rely on flash-attention as well as SwiGLU and Rotary Embedding kernels from FlashAttention-2 ([Dao et al., 2023](https://tridao.me/publications/flash2/flash2.pdf))
113
+
114
+ ## Use and Limitations
115
+
116
+ ### Intended Use
117
+
118
+ The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications.
119
+
120
+ ### Limitations and Bias
121
+
122
+ As a base model, this model may exhibit unreliable, unsafe, or other undesirable behaviors that must be corrected through evaluation and fine-tuning prior to deployment. The pre-training dataset may have contained offensive or inappropriate content, even after applying data cleansing filters, which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.
123
+
124
+ ## How to Cite
125
+
126
+ ```bibtex
127
+ @misc{StableLM-2-1.6B,
128
+ url={[https://huggingface.co/stabilityai/stablelm-2-1.6b](https://huggingface.co/stabilityai/stablelm-2-1.6b)},
129
+ title={Stable LM 2 1.6B},
130
+ author={Stability AI Language Team}
131
+ }
132
+ ```
arcade100k.tiktoken ADDED
The diff for this file is too large to render. See raw diff
 
config.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "StableLMEpochForCausalLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_stablelm_epoch.StableLMEpochConfig",
7
+ "AutoModelForCausalLM": "modeling_stablelm_epoch.StableLMEpochForCausalLM"
8
+ },
9
+ "bos_token_id": 100257,
10
+ "eos_token_id": 100257,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 2048,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 5632,
15
+ "max_position_embeddings": 4096,
16
+ "model_type": "stablelm_epoch",
17
+ "norm_eps": 1e-05,
18
+ "num_attention_heads": 32,
19
+ "num_heads": 32,
20
+ "num_hidden_layers": 24,
21
+ "num_key_value_heads": 32,
22
+ "rope_pct": 0.25,
23
+ "rope_theta": 10000,
24
+ "rotary_scaling_factor": 1.0,
25
+ "tie_word_embeddings": false,
26
+ "torch_dtype": "bfloat16",
27
+ "transformers_version": "4.36.2",
28
+ "use_cache": true,
29
+ "use_qkv_bias": true,
30
+ "vocab_size": 100352
31
+ }
configuration_stablelm_epoch.py ADDED
@@ -0,0 +1,113 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2023 Stability and The HuggingFace Inc. team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ """ StableLM Epoch model configuration"""
15
+ from transformers import PretrainedConfig
16
+ from transformers.utils import logging
17
+
18
+
19
+ logger = logging.get_logger(__name__)
20
+
21
+
22
+ class StableLMEpochConfig(PretrainedConfig):
23
+ r"""
24
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
25
+ documentation from [`PretrainedConfig`] for more information.
26
+
27
+ Args:
28
+ vocab_size (`int`, *optional*, defaults to 50_304):
29
+ Vocabulary size of the StableLM model. Defines the number of different tokens that
30
+ can be represented by the `inputs_ids` passed when calling [`StableLMEpochModel`].
31
+ intermediate_size (`int`, *optional*, defaults to 6912):
32
+ Dimension of the MLP representations.
33
+ hidden_size (`int`, *optional*, defaults to 2560):
34
+ Dimension of the decoder layers and the pooler layer.
35
+ num_hidden_layers (`int`, *optional*, defaults to 32):
36
+ Number of hidden layers in the Transformer decoder.
37
+ num_attention_heads (`int`, *optional*, defaults to 32):
38
+ Number of attention heads for each attention layer in the Transformer encoder.
39
+ num_key_value_heads (`int`, *optional*):
40
+ This is the number of key_value heads that should be used to implement Grouped Query Attention. If
41
+ `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
42
+ `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
43
+ converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
44
+ by meanpooling all the original heads within that group. For more details checkout [this
45
+ paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
46
+ `num_attention_heads`.
47
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
48
+ The non-linear activation function (function or string).
49
+ rope_pct (`float`, *optional*, defaults to 1.0):
50
+ Percentage of hidden dimensions to allocate to rotary embeddings.
51
+ rope_theta (`float`, *optional*, defaults to 10000.0):
52
+ The base period of the RoPE embeddings.
53
+ max_position_embeddings (`int`, *optional*, defaults to 2048):
54
+ The maximum sequence length that this model might ever be used with.
55
+ Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
56
+ initializer_range (`float`, *optional*, defaults to 1e-5):
57
+ The standard deviation of the truncated_normal_initializer for initializing
58
+ all weight matrices.
59
+ norm_eps (`float`, *optional*, defaults to 1e-8):
60
+ The epsilon used by the normalization layers.
61
+ use_cache (`bool`, *optional*, defaults to `True`):
62
+ Whether or not the model should return the last key/values attentions
63
+ (not used by all models). Only relevant if `config.is_decoder=True`.
64
+ use_qkv_bias (`bool`, *optional*, defaults to `True`):
65
+ Whether or not the model should use bias for qkv layers.
66
+ tie_word_embeddings(`bool`, *optional*, defaults to `False`):
67
+ Whether to tie weight embeddings
68
+ """
69
+ model_type = "stablelm_epoch"
70
+ keys_to_ignore_at_inference = ["past_key_values"]
71
+
72
+ def __init__(
73
+ self,
74
+ vocab_size=50_304,
75
+ intermediate_size=6912,
76
+ hidden_size=2560,
77
+ num_hidden_layers=32,
78
+ num_attention_heads=32,
79
+ num_key_value_heads=32,
80
+ hidden_act="silu",
81
+ rope_pct=0.25,
82
+ rope_theta=10_000,
83
+ max_position_embeddings=4096,
84
+ initializer_range=0.02,
85
+ norm_eps=1.0e-5,
86
+ use_cache=True,
87
+ use_qkv_bias=True,
88
+ bos_token_id=0,
89
+ eos_token_id=2,
90
+ tie_word_embeddings=False,
91
+ **kwargs,
92
+ ):
93
+ self.vocab_size = vocab_size
94
+ self.max_position_embeddings = max_position_embeddings
95
+ self.intermediate_size = intermediate_size
96
+ self.hidden_size = hidden_size
97
+ self.num_hidden_layers = num_hidden_layers
98
+ self.num_attention_heads = num_attention_heads
99
+ self.num_key_value_heads = num_key_value_heads
100
+ self.hidden_act = hidden_act
101
+ self.rope_pct = rope_pct
102
+ self.rope_theta = rope_theta
103
+ self.initializer_range = initializer_range
104
+ self.norm_eps = norm_eps
105
+ self.use_cache = use_cache
106
+ self.use_qkv_bias = use_qkv_bias
107
+ self.tie_word_embeddings = tie_word_embeddings
108
+ super().__init__(
109
+ bos_token_id=bos_token_id,
110
+ eos_token_id=eos_token_id,
111
+ tie_word_embeddings=tie_word_embeddings,
112
+ **kwargs,
113
+ )
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "eos_token_id": 100257,
5
+ "transformers_version": "4.36.2"
6
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8bdf317e2b35ab5c8009cbb6c7ce495e4e608a6b9b843d44054edf25b8c5860d
3
+ size 3289069520
modeling_stablelm_epoch.py ADDED
@@ -0,0 +1,917 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2023 Stability AI, EleutherAI, and The HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+ # This code is based off the following work:
17
+ # https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py
18
+ # https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py
19
+ """ PyTorch StableLM Epoch model. """
20
+ from typing import Optional, Tuple, Union
21
+ import math
22
+ import warnings
23
+
24
+ import torch
25
+ import torch.nn.functional as F
26
+ import torch.utils.checkpoint
27
+ from torch import nn
28
+ from torch.nn import CrossEntropyLoss
29
+
30
+ from transformers.cache_utils import Cache
31
+ from transformers.modeling_outputs import (
32
+ BaseModelOutputWithPast,
33
+ CausalLMOutputWithPast,
34
+ )
35
+ from transformers.modeling_utils import PreTrainedModel
36
+ from transformers.utils import logging, is_flash_attn_greater_or_equal_2_10
37
+
38
+ from .configuration_stablelm_epoch import StableLMEpochConfig
39
+
40
+ try:
41
+ from flash_attn import flash_attn_func, flash_attn_varlen_func
42
+ from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input
43
+ except:
44
+ flash_attn_func, flash_attn_varlen_func = None, None
45
+ index_first_axis, pad_input, unpad_input = None, None, None
46
+
47
+
48
+ logger = logging.get_logger(__name__)
49
+
50
+
51
+ # Copied from transformers.models.llama.modeling_llama._get_unpad_data
52
+ def _get_unpad_data(attention_mask):
53
+ seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
54
+ indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
55
+ max_seqlen_in_batch = seqlens_in_batch.max().item()
56
+ cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0))
57
+ return (
58
+ indices,
59
+ cu_seqlens,
60
+ max_seqlen_in_batch,
61
+ )
62
+
63
+
64
+ # Copied from transformers.models.bart.modeling_bart._make_causal_mask
65
+ def _make_causal_mask(
66
+ input_ids_shape: torch.Size,
67
+ dtype: torch.dtype,
68
+ device: torch.device,
69
+ past_key_values_length: int = 0,
70
+ ):
71
+ """Make causal mask used for bi-directional self-attention."""
72
+ batch_size, tgt_len = input_ids_shape
73
+ mask = torch.full((tgt_len, tgt_len), torch.finfo(torch.float16).min, device=device)
74
+ mask_cond = torch.arange(mask.size(-1), device=device)
75
+ mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
76
+ mask = mask.to(dtype)
77
+ if past_key_values_length > 0:
78
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
79
+ return mask[None, None, :, :].expand(batch_size, 1, tgt_len, tgt_len + past_key_values_length)
80
+
81
+
82
+ # Copied from transformers.models.bart.modeling_bart._expand_mask
83
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
84
+ """Expands attention_mask from `[batch_size, seq_len]` to `[batch_size, 1, tgt_seq_len, src_seq_len]`."""
85
+ batch_size, src_len = mask.size()
86
+ tgt_len = tgt_len if tgt_len is not None else src_len
87
+
88
+ expanded_mask = mask[:, None, None, :].expand(batch_size, 1, tgt_len, src_len).to(dtype)
89
+ inverted_mask = 1.0 - expanded_mask
90
+
91
+ return inverted_mask.masked_fill(
92
+ inverted_mask.to(torch.bool), torch.finfo(dtype).min
93
+ )
94
+
95
+
96
+ class RotaryEmbedding(nn.Module):
97
+ def __init__(
98
+ self,
99
+ dim: int,
100
+ max_position_embeddings: int,
101
+ base: int = 10_000,
102
+ device: Optional[torch.device] = None,
103
+ ):
104
+ super().__init__()
105
+
106
+ self.dim = dim
107
+ self.max_position_embeddings = max_position_embeddings
108
+ self.base = base
109
+ inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2, device=device, dtype=torch.float32) / self.dim))
110
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
111
+
112
+ # Build here to make `torch.jit.trace` work.
113
+ self._set_cos_sin_cache(
114
+ seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype(),
115
+ )
116
+
117
+ def _set_cos_sin_cache(self, seq_len: int, device: torch.device, dtype: torch.dtype):
118
+ self.max_seq_len_cached = seq_len
119
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.float32)
120
+
121
+ # Don't do einsum, it converts fp32 to fp16 under AMP
122
+ # freqs = torch.einsum("i,j->ij", t, self.inv_freq)
123
+ freqs = torch.outer(t, self.inv_freq)
124
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
125
+ emb = torch.cat((freqs, freqs), dim=-1)
126
+ self.register_buffer("cos_cached", emb.cos()[None, None, :, :].to(dtype), persistent=False)
127
+ self.register_buffer("sin_cached", emb.sin()[None, None, :, :].to(dtype), persistent=False)
128
+
129
+ def forward(self, x: torch.Tensor, seq_len: Optional[int] = None):
130
+ # x: [batch_size, num_heads, seq_len, head_size]
131
+ if seq_len > self.max_seq_len_cached:
132
+ self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=torch.get_default_dtype())
133
+ return (
134
+ self.cos_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
135
+ self.sin_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
136
+ )
137
+
138
+
139
+ def rotate_half(x: torch.Tensor):
140
+ """Rotates half the hidden dims of the input."""
141
+ x1, x2 = torch.chunk(x, 2, dim=-1)
142
+ return torch.cat((-x2, x1), dim=-1)
143
+
144
+
145
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids):
146
+ # The first two dimensions of cos and sin are always 1, so we can `squeeze` them.
147
+ cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
148
+ sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
149
+ cos = cos[position_ids].unsqueeze(1) # [batch_size, 1, seq_len, dim]
150
+ sin = sin[position_ids].unsqueeze(1) # [batch_size, 1, seq_len, dim]
151
+ q_embed = (q * cos) + (rotate_half(q) * sin)
152
+ k_embed = (k * cos) + (rotate_half(k) * sin)
153
+ return q_embed, k_embed
154
+
155
+
156
+ class MLP(nn.Module):
157
+ def __init__(self, config: StableLMEpochConfig):
158
+ super().__init__()
159
+ self.config = config
160
+ self.hidden_size = config.hidden_size
161
+ self.intermediate_size = config.intermediate_size
162
+ self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
163
+ self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
164
+ self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
165
+ self.act_fn = nn.SiLU()
166
+
167
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
168
+ return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
169
+
170
+
171
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
172
+ """
173
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
174
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
175
+ """
176
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
177
+ if n_rep == 1:
178
+ return hidden_states
179
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
180
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
181
+
182
+
183
+ class Attention(nn.Module):
184
+ def __init__(self, config: StableLMEpochConfig):
185
+ super().__init__()
186
+ self.config = config
187
+ self.hidden_size = config.hidden_size
188
+ self.num_heads = config.num_attention_heads
189
+ self.head_dim = self.hidden_size // self.num_heads
190
+ self.num_key_value_heads = config.num_key_value_heads
191
+ self.num_key_value_groups = self.num_heads // self.num_key_value_heads
192
+ self.max_position_embeddings = config.max_position_embeddings
193
+ self.is_causal = True
194
+
195
+ if (self.head_dim * self.num_heads) != self.hidden_size:
196
+ raise ValueError(
197
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
198
+ f" and `num_heads`: {self.num_heads})."
199
+ )
200
+
201
+ self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=config.use_qkv_bias)
202
+ self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=config.use_qkv_bias)
203
+ self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=config.use_qkv_bias)
204
+ self.o_proj = nn.Linear(self.hidden_size, self.hidden_size, bias=False)
205
+
206
+ self._init_rope()
207
+
208
+ def _init_rope(self):
209
+ self.rotary_ndims = int(self.head_dim * self.config.rope_pct)
210
+ self.rotary_emb = RotaryEmbedding(
211
+ self.rotary_ndims,
212
+ max_position_embeddings=self.config.max_position_embeddings,
213
+ base=self.config.rope_theta,
214
+ )
215
+
216
+ def forward(
217
+ self,
218
+ hidden_states: torch.FloatTensor,
219
+ attention_mask: torch.FloatTensor,
220
+ position_ids: torch.LongTensor,
221
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
222
+ output_attentions: Optional[bool] = False,
223
+ use_cache: Optional[bool] = False,
224
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
225
+ bsz, q_len, _ = hidden_states.size()
226
+
227
+ query_states = self.q_proj(hidden_states)
228
+ key_states = self.k_proj(hidden_states)
229
+ value_states = self.v_proj(hidden_states)
230
+
231
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
232
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
233
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
234
+
235
+ query_rot = query_states[..., : self.rotary_ndims]
236
+ query_pass = query_states[..., self.rotary_ndims :]
237
+ key_rot = key_states[..., : self.rotary_ndims]
238
+ key_pass = key_states[..., self.rotary_ndims :]
239
+
240
+ kv_seq_len = key_states.shape[-2]
241
+ if past_key_value is not None:
242
+ kv_seq_len += past_key_value[0].shape[-2]
243
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
244
+ query_states, key_states = apply_rotary_pos_emb(query_rot, key_rot, cos, sin, position_ids)
245
+
246
+ # [batch_size, num_heads, seq_len, head_dim]
247
+ query_states = torch.cat((query_states, query_pass), dim=-1)
248
+ key_states = torch.cat((key_states, key_pass), dim=-1)
249
+
250
+ if past_key_value is not None:
251
+ # Reuse k, v, self_attention
252
+ key_states = torch.cat((past_key_value[0], key_states), dim=2)
253
+ value_states = torch.cat((past_key_value[1], value_states), dim=2)
254
+
255
+ past_key_value = (key_states, value_states) if use_cache else None
256
+
257
+ # Repeat k/v heads if n_kv_heads < n_heads
258
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
259
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
260
+
261
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
262
+
263
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
264
+ raise ValueError(
265
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
266
+ f" {attn_weights.size()}"
267
+ )
268
+
269
+ if attention_mask is not None:
270
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
271
+ raise ValueError(
272
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
273
+ )
274
+ attn_weights = attn_weights + attention_mask
275
+
276
+ # Upcast attention to fp32
277
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
278
+ attn_output = torch.matmul(attn_weights, value_states)
279
+
280
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
281
+ raise ValueError(
282
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
283
+ f" {attn_output.size()}"
284
+ )
285
+
286
+ # Merge heads
287
+ attn_output = attn_output.transpose(1, 2).contiguous()
288
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
289
+
290
+ # Final linear projection
291
+ attn_output = self.o_proj(attn_output)
292
+
293
+ if not output_attentions:
294
+ attn_weights = None
295
+
296
+ return attn_output, attn_weights, past_key_value
297
+
298
+
299
+ class FlashAttention2(Attention):
300
+ """
301
+ Reference: https://github.com/huggingface/transformers/blob/5d36025ca13d05151b7a0c761e90d429c4644a30/src/transformers/models/llama/modeling_llama.py#L456
302
+ """
303
+
304
+ def __init__(self, *args, **kwargs):
305
+ super().__init__(*args, **kwargs)
306
+
307
+ # TODO: Should be removed once Flash Attention for RoCm is bumped to 2.1.
308
+ # flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. Reference: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.1.0.
309
+ # Beware that with flash_attn<2.1, using q_seqlen != k_seqlen (except for the case q_seqlen == 1) produces a wrong mask (top-left).
310
+ self._flash_attn_uses_top_left_mask = not is_flash_attn_greater_or_equal_2_10()
311
+
312
+ def forward(
313
+ self,
314
+ hidden_states: torch.Tensor,
315
+ attention_mask: Optional[torch.LongTensor] = None,
316
+ position_ids: Optional[torch.LongTensor] = None,
317
+ past_key_value: Optional[Cache] = None,
318
+ output_attentions: bool = False,
319
+ use_cache: bool = False,
320
+ **kwargs,
321
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
322
+ # FlashAttention2 attention does not support output_attentions
323
+ if "padding_mask" in kwargs:
324
+ warnings.warn(
325
+ "Passing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`"
326
+ )
327
+
328
+ # overwrite attention_mask with padding_mask
329
+ attention_mask = kwargs.pop("padding_mask")
330
+
331
+ output_attentions = False
332
+
333
+ bsz, q_len, _ = hidden_states.size()
334
+
335
+ query_states = self.q_proj(hidden_states)
336
+ key_states = self.k_proj(hidden_states)
337
+ value_states = self.v_proj(hidden_states)
338
+
339
+ # Flash attention requires the input to have the shape
340
+ # batch_size x seq_length x head_dim x hidden_dim
341
+ # therefore we just need to keep the original shape
342
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
343
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
344
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
345
+
346
+ query_rot = query_states[..., : self.rotary_ndims]
347
+ query_pass = query_states[..., self.rotary_ndims :]
348
+ key_rot = key_states[..., : self.rotary_ndims]
349
+ key_pass = key_states[..., self.rotary_ndims :]
350
+
351
+ kv_seq_len = key_states.shape[-2]
352
+ if past_key_value is not None:
353
+ kv_seq_len += past_key_value[0].shape[-2]
354
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
355
+ query_states, key_states = apply_rotary_pos_emb(query_rot, key_rot, cos, sin, position_ids)
356
+
357
+ # [batch_size, num_heads, seq_len, head_dim]
358
+ query_states = torch.cat((query_states, query_pass), dim=-1)
359
+ key_states = torch.cat((key_states, key_pass), dim=-1)
360
+
361
+ if past_key_value is not None:
362
+ # Reuse k, v, self_attention
363
+ key_states = torch.cat((past_key_value[0], key_states), dim=2)
364
+ value_states = torch.cat((past_key_value[1], value_states), dim=2)
365
+
366
+ past_key_value = (key_states, value_states) if use_cache else None
367
+
368
+ # TODO: These transpose are quite inefficient but Flash Attention requires the layout [batch_size, sequence_length, num_heads, head_dim]. We would need to refactor the KV cache
369
+ # to be able to avoid many of these transpose/reshape/view.
370
+ query_states = query_states.transpose(1, 2)
371
+ key_states = key_states.transpose(1, 2)
372
+ value_states = value_states.transpose(1, 2)
373
+
374
+ dropout_rate = self.attention_dropout if self.training else 0.0
375
+
376
+ attn_output = self._flash_attention_forward(
377
+ query_states, key_states, value_states, attention_mask, q_len, dropout=dropout_rate
378
+ )
379
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
380
+ attn_output = self.o_proj(attn_output)
381
+
382
+ if not output_attentions:
383
+ attn_weights = None
384
+
385
+ return attn_output, attn_weights, past_key_value
386
+
387
+ def _flash_attention_forward(
388
+ self, query_states, key_states, value_states, attention_mask, query_length, dropout=0.0, softmax_scale=None
389
+ ):
390
+ """
391
+ Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token
392
+ first unpad the input, then computes the attention scores and pad the final attention scores.
393
+
394
+ Args:
395
+ query_states (`torch.Tensor`):
396
+ Input query states to be passed to Flash Attention API
397
+ key_states (`torch.Tensor`):
398
+ Input key states to be passed to Flash Attention API
399
+ value_states (`torch.Tensor`):
400
+ Input value states to be passed to Flash Attention API
401
+ attention_mask (`torch.Tensor`):
402
+ The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the
403
+ position of padding tokens and 1 for the position of non-padding tokens.
404
+ dropout (`int`, *optional*):
405
+ Attention dropout
406
+ softmax_scale (`float`, *optional*):
407
+ The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)
408
+ """
409
+ if not self._flash_attn_uses_top_left_mask:
410
+ causal = self.is_causal
411
+ else:
412
+ # TODO: Remove the `query_length != 1` check once Flash Attention for RoCm is bumped to 2.1. For details, please see the comment in FlashAttention2 __init__.
413
+ causal = self.is_causal and query_length != 1
414
+
415
+ # Contains at least one padding token in the sequence
416
+ if attention_mask is not None:
417
+ batch_size = query_states.shape[0]
418
+ query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = self._upad_input(
419
+ query_states, key_states, value_states, attention_mask, query_length
420
+ )
421
+
422
+ cu_seqlens_q, cu_seqlens_k = cu_seq_lens
423
+ max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens
424
+
425
+ attn_output_unpad = flash_attn_varlen_func(
426
+ query_states,
427
+ key_states,
428
+ value_states,
429
+ cu_seqlens_q=cu_seqlens_q,
430
+ cu_seqlens_k=cu_seqlens_k,
431
+ max_seqlen_q=max_seqlen_in_batch_q,
432
+ max_seqlen_k=max_seqlen_in_batch_k,
433
+ dropout_p=dropout,
434
+ softmax_scale=softmax_scale,
435
+ causal=causal,
436
+ )
437
+
438
+ attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)
439
+ else:
440
+ attn_output = flash_attn_func(
441
+ query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal
442
+ )
443
+
444
+ return attn_output
445
+
446
+ def _upad_input(self, query_layer, key_layer, value_layer, attention_mask, query_length):
447
+ indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
448
+ batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape
449
+
450
+ key_layer = index_first_axis(
451
+ key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
452
+ )
453
+ value_layer = index_first_axis(
454
+ value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k
455
+ )
456
+ if query_length == kv_seq_len:
457
+ query_layer = index_first_axis(
458
+ query_layer.reshape(batch_size * kv_seq_len, self.num_heads, head_dim), indices_k
459
+ )
460
+ cu_seqlens_q = cu_seqlens_k
461
+ max_seqlen_in_batch_q = max_seqlen_in_batch_k
462
+ indices_q = indices_k
463
+ elif query_length == 1:
464
+ max_seqlen_in_batch_q = 1
465
+ cu_seqlens_q = torch.arange(
466
+ batch_size + 1, dtype=torch.int32, device=query_layer.device
467
+ ) # There is a memcpy here, that is very bad.
468
+ indices_q = cu_seqlens_q[:-1]
469
+ query_layer = query_layer.squeeze(1)
470
+ else:
471
+ # The -q_len: slice assumes left padding.
472
+ attention_mask = attention_mask[:, -query_length:]
473
+ query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)
474
+
475
+ return (
476
+ query_layer,
477
+ key_layer,
478
+ value_layer,
479
+ indices_q,
480
+ (cu_seqlens_q, cu_seqlens_k),
481
+ (max_seqlen_in_batch_q, max_seqlen_in_batch_k),
482
+ )
483
+
484
+
485
+ ATTENTION_CLASSES = {
486
+ "eager": Attention,
487
+ "flash_attention_2": FlashAttention2,
488
+ }
489
+
490
+
491
+ class DecoderLayer(nn.Module):
492
+ def __init__(self, config: StableLMEpochConfig):
493
+ super().__init__()
494
+ self.self_attn = ATTENTION_CLASSES[config._attn_implementation](config=config)
495
+ self.mlp = MLP(config)
496
+ self.input_layernorm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps)
497
+ self.post_attention_layernorm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps)
498
+
499
+ def forward(
500
+ self,
501
+ hidden_states: Optional[torch.FloatTensor],
502
+ attention_mask: Optional[torch.FloatTensor] = None,
503
+ position_ids: Optional[torch.LongTensor] = None,
504
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
505
+ output_attentions: Optional[bool] = False,
506
+ use_cache: Optional[bool] = False,
507
+ ) -> Union[Tuple[torch.Tensor], Optional[Tuple[torch.Tensor, Tuple[torch.FloatTensor, ...]]]]:
508
+ residual = hidden_states
509
+
510
+ hidden_states = self.input_layernorm(hidden_states)
511
+
512
+ # Self Attention
513
+ hidden_states, self_attn_weights, present_key_value = self.self_attn(
514
+ hidden_states=hidden_states,
515
+ attention_mask=attention_mask,
516
+ position_ids=position_ids,
517
+ past_key_value=past_key_value,
518
+ output_attentions=output_attentions,
519
+ use_cache=use_cache,
520
+ )
521
+ hidden_states = residual + hidden_states
522
+
523
+ # Fully Connected
524
+ residual = hidden_states
525
+ hidden_states = self.post_attention_layernorm(hidden_states)
526
+ hidden_states = self.mlp(hidden_states)
527
+ hidden_states = residual + hidden_states
528
+
529
+ outputs = (hidden_states,)
530
+
531
+ if output_attentions:
532
+ outputs += (self_attn_weights,)
533
+
534
+ if use_cache:
535
+ outputs += (present_key_value,)
536
+
537
+ return outputs
538
+
539
+
540
+ class StableLMEpochPreTrainedModel(PreTrainedModel):
541
+ """An abstract class to handle weights initialization and a simple interface
542
+ for downloading and loading pretrained models.
543
+ """
544
+
545
+ config_class = StableLMEpochConfig
546
+ base_model_prefix = "transformer"
547
+ supports_gradient_checkpointing = True
548
+ _no_split_modules = ["DecoderLayer"]
549
+ _skip_keys_device_placement = "past_key_values"
550
+ _supports_flash_attn_2 = True
551
+
552
+ def _init_weights(self, module: nn.Module):
553
+ """Initialize the weights"""
554
+ if isinstance(module, nn.Linear):
555
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
556
+ if module.bias is not None:
557
+ module.bias.data.zero_()
558
+ elif isinstance(module, nn.Embedding):
559
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
560
+ if module.padding_idx is not None:
561
+ module.weight.data[module.padding_idx].zero_()
562
+ elif isinstance(module, nn.LayerNorm):
563
+ module.bias.data.zero_()
564
+ module.weight.data.fill_(1.0)
565
+
566
+ def _set_gradient_checkpointing(self, module: nn.Module, value=False):
567
+ if isinstance(module, StableLMEpochModel):
568
+ module.gradient_checkpointing = value
569
+
570
+
571
+ class StableLMEpochModel(StableLMEpochPreTrainedModel):
572
+ def __init__(self, config: StableLMEpochConfig):
573
+ super().__init__(config)
574
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, config.pad_token_id)
575
+ self.layers = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)])
576
+ self.norm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps)
577
+
578
+ self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
579
+ self.gradient_checkpointing = False
580
+ # Initialize weights and apply final processing
581
+ self.post_init()
582
+
583
+ def get_input_embeddings(self):
584
+ return self.embed_tokens
585
+
586
+ def set_input_embeddings(self, value: nn.Module):
587
+ self.embed_tokens = value
588
+
589
+ # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask
590
+ def _prepare_decoder_attention_mask(
591
+ self,
592
+ attention_mask: torch.Tensor,
593
+ input_shape: torch.Size,
594
+ inputs_embeds: torch.Tensor,
595
+ past_key_values_length: int,
596
+ ):
597
+ # Create causal mask
598
+ # [batch_size, seq_len] -> [batch_size, 1, tgt_seq_len, src_seq_len]
599
+ combined_attention_mask = None
600
+ if input_shape[-1] > 1:
601
+ combined_attention_mask = _make_causal_mask(
602
+ input_shape,
603
+ inputs_embeds.dtype,
604
+ device=inputs_embeds.device,
605
+ past_key_values_length=past_key_values_length,
606
+ )
607
+
608
+ if attention_mask is not None:
609
+ # [batch_size, seq_len] -> [batch_size, 1, tgt_seq_len, src_seq_len]
610
+ expanded_attn_mask = _expand_mask(
611
+ attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]
612
+ ).to(inputs_embeds.device)
613
+ combined_attention_mask = expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
614
+
615
+ return combined_attention_mask
616
+
617
+ def forward(
618
+ self,
619
+ input_ids: Optional[torch.LongTensor] = None,
620
+ attention_mask: Optional[torch.FloatTensor] = None,
621
+ position_ids: Optional[torch.LongTensor] = None,
622
+ past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
623
+ inputs_embeds: Optional[torch.FloatTensor] = None,
624
+ use_cache: Optional[bool] = None,
625
+ output_attentions: Optional[bool] = None,
626
+ output_hidden_states: Optional[bool] = None,
627
+ return_dict: Optional[bool] = None,
628
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
629
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
630
+ output_hidden_states = output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
631
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
632
+
633
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
634
+
635
+ # Retrieve input_ids and inputs_embeds
636
+ if input_ids is not None and inputs_embeds is not None:
637
+ raise ValueError(
638
+ "You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time"
639
+ )
640
+ elif input_ids is not None:
641
+ batch_size, seq_length = input_ids.shape
642
+ elif inputs_embeds is not None:
643
+ batch_size, seq_length, _ = inputs_embeds.shape
644
+ else:
645
+ raise ValueError(
646
+ "You have to specify either decoder_input_ids or decoder_inputs_embeds"
647
+ )
648
+
649
+ seq_length_with_past = seq_length
650
+ past_key_values_length = 0
651
+
652
+ if position_ids is None:
653
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
654
+ position_ids = torch.arange(
655
+ past_key_values_length,
656
+ seq_length + past_key_values_length,
657
+ dtype=torch.long,
658
+ device=device,
659
+ )
660
+ position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
661
+ else:
662
+ position_ids = position_ids.view(-1, seq_length).long()
663
+
664
+ if inputs_embeds is None:
665
+ inputs_embeds = self.embed_tokens(input_ids)
666
+ # Embed positions
667
+ if self._use_flash_attention_2:
668
+ # 2d mask is passed through the layers
669
+ attention_mask = attention_mask if (attention_mask is not None and 0 in attention_mask) else None
670
+ else:
671
+ if attention_mask is None:
672
+ attention_mask = torch.ones(
673
+ (batch_size, seq_length_with_past),
674
+ dtype=torch.bool,
675
+ device=inputs_embeds.device,
676
+ )
677
+ attention_mask = self._prepare_decoder_attention_mask(
678
+ attention_mask,
679
+ (batch_size, seq_length),
680
+ inputs_embeds,
681
+ past_key_values_length,
682
+ )
683
+
684
+ hidden_states = inputs_embeds
685
+
686
+ if self.gradient_checkpointing and self.training:
687
+ if use_cache:
688
+ logger.warning(
689
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
690
+ )
691
+ use_cache = False
692
+
693
+ # Decoder layers
694
+ all_hidden_states = () if output_hidden_states else None
695
+ all_self_attns = () if output_attentions else None
696
+ next_decoder_cache = () if use_cache else None
697
+
698
+ for idx, decoder_layer in enumerate(self.layers):
699
+ if output_hidden_states:
700
+ all_hidden_states += (hidden_states,)
701
+
702
+ past_key_value = (
703
+ past_key_values[idx] if past_key_values is not None else None
704
+ )
705
+
706
+ if self.gradient_checkpointing and self.training:
707
+
708
+ def create_custom_forward(module):
709
+ def custom_forward(*inputs):
710
+ # None for past_key_value
711
+ return module(*inputs, past_key_value, output_attentions)
712
+
713
+ return custom_forward
714
+
715
+ layer_outputs = torch.utils.checkpoint.checkpoint(
716
+ create_custom_forward(decoder_layer),
717
+ hidden_states,
718
+ attention_mask,
719
+ position_ids,
720
+ )
721
+ else:
722
+ layer_outputs = decoder_layer(
723
+ hidden_states,
724
+ attention_mask=attention_mask,
725
+ position_ids=position_ids,
726
+ past_key_value=past_key_value,
727
+ output_attentions=output_attentions,
728
+ use_cache=use_cache,
729
+ )
730
+
731
+ hidden_states = layer_outputs[0]
732
+
733
+ if use_cache:
734
+ next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
735
+
736
+ if output_attentions:
737
+ all_self_attns += (layer_outputs[1],)
738
+
739
+ hidden_states = self.norm(hidden_states)
740
+
741
+ # Add hidden states from the last decoder layer
742
+ if output_hidden_states:
743
+ all_hidden_states += (hidden_states,)
744
+
745
+ next_cache = next_decoder_cache if use_cache else None
746
+ if not return_dict:
747
+ return tuple(
748
+ v
749
+ for v in [hidden_states, next_cache, all_hidden_states, all_self_attns]
750
+ if v is not None
751
+ )
752
+ return BaseModelOutputWithPast(
753
+ last_hidden_state=hidden_states,
754
+ past_key_values=next_cache,
755
+ hidden_states=all_hidden_states,
756
+ attentions=all_self_attns,
757
+ )
758
+
759
+
760
+ class StableLMEpochForCausalLM(StableLMEpochPreTrainedModel):
761
+ _tied_weights_keys = ["lm_head.weight"]
762
+
763
+ def __init__(self, config: StableLMEpochConfig):
764
+ super().__init__(config)
765
+
766
+ self.model = StableLMEpochModel(config)
767
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
768
+
769
+ # Initialize weights and apply final processing
770
+ self.post_init()
771
+
772
+ def get_input_embeddings(self):
773
+ return self.model.embed_tokens
774
+
775
+ def set_input_embeddings(self, value):
776
+ self.model.embed_tokens = value
777
+
778
+ def get_output_embeddings(self):
779
+ return self.lm_head
780
+
781
+ def set_output_embeddings(self, new_embeddings: nn.Module):
782
+ self.lm_head = new_embeddings
783
+
784
+ def get_decoder(self):
785
+ return self.model
786
+
787
+ def set_decoder(self, decoder):
788
+ self.model = decoder
789
+
790
+ def forward(
791
+ self,
792
+ input_ids: Optional[torch.LongTensor] = None,
793
+ attention_mask: Optional[torch.FloatTensor] = None,
794
+ position_ids: Optional[torch.LongTensor] = None,
795
+ past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
796
+ inputs_embeds: Optional[torch.FloatTensor] = None,
797
+ labels: Optional[torch.LongTensor] = None,
798
+ use_cache: Optional[bool] = None,
799
+ output_attentions: Optional[bool] = None,
800
+ output_hidden_states: Optional[bool] = None,
801
+ return_dict: Optional[bool] = None,
802
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
803
+ output_attentions = (
804
+ output_attentions
805
+ if output_attentions is not None
806
+ else self.config.output_attentions
807
+ )
808
+ output_hidden_states = (
809
+ output_hidden_states
810
+ if output_hidden_states is not None
811
+ else self.config.output_hidden_states
812
+ )
813
+ return_dict = (
814
+ return_dict if return_dict is not None else self.config.use_return_dict
815
+ )
816
+
817
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
818
+ outputs = self.model(
819
+ input_ids,
820
+ attention_mask=attention_mask,
821
+ position_ids=position_ids,
822
+ past_key_values=past_key_values,
823
+ inputs_embeds=inputs_embeds,
824
+ use_cache=use_cache,
825
+ output_attentions=output_attentions,
826
+ output_hidden_states=output_hidden_states,
827
+ return_dict=return_dict,
828
+ )
829
+
830
+ hidden_states = outputs[0]
831
+ logits = self.lm_head(hidden_states).float()
832
+
833
+ loss = None
834
+ if labels is not None:
835
+ # Shift so that tokens < n predict n
836
+ shift_logits = logits[..., :-1, :].contiguous()
837
+ shift_labels = labels[..., 1:].contiguous()
838
+ # Flatten the tokens
839
+ loss_fct = CrossEntropyLoss()
840
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
841
+ shift_labels = shift_labels.view(-1)
842
+ # Enable model parallelism
843
+ shift_labels = shift_labels.to(shift_logits.device)
844
+ loss = loss_fct(shift_logits, shift_labels)
845
+
846
+ if not return_dict:
847
+ output = (logits,) + outputs[1:]
848
+ return (loss,) + output if loss is not None else output
849
+
850
+ return CausalLMOutputWithPast(
851
+ loss=loss,
852
+ logits=logits,
853
+ past_key_values=outputs.past_key_values,
854
+ hidden_states=outputs.hidden_states,
855
+ attentions=outputs.attentions,
856
+ )
857
+
858
+ def prepare_inputs_for_generation(
859
+ self,
860
+ input_ids,
861
+ past_key_values: Optional[torch.Tensor] = None,
862
+ attention_mask: Optional[torch.Tensor] = None,
863
+ inputs_embeds: Optional[torch.Tensor] = None,
864
+ **kwargs,
865
+ ):
866
+ # Trim decoder_input_ids if past is used
867
+ if past_key_values is not None:
868
+ past_length = past_key_values[0][0].shape[2]
869
+
870
+ # Some generation methods already pass only the last input ID
871
+ if input_ids.shape[1] > past_length:
872
+ remove_prefix_length = past_length
873
+ else:
874
+ # Default to old behavior: keep only final ID
875
+ remove_prefix_length = input_ids.shape[1] - 1
876
+
877
+ input_ids = input_ids[:, remove_prefix_length:]
878
+
879
+ position_ids = kwargs.get("position_ids", None)
880
+ if attention_mask is not None and position_ids is None:
881
+ # Create position_ids on the fly for batch generation
882
+ position_ids = attention_mask.long().cumsum(-1) - 1
883
+ position_ids.masked_fill_(attention_mask == 0, 1)
884
+ if past_key_values:
885
+ position_ids = position_ids[:, -1].unsqueeze(-1)
886
+
887
+ # If `inputs_embeds` are passed, we only want to use them in the 1st generation step
888
+ if inputs_embeds is not None and past_key_values is None:
889
+ model_inputs = {"inputs_embeds": inputs_embeds}
890
+ else:
891
+ model_inputs = {"input_ids": input_ids}
892
+
893
+ model_inputs.update(
894
+ {
895
+ "attention_mask": attention_mask,
896
+ "past_key_values": past_key_values,
897
+ "use_cache": kwargs.get("use_cache"),
898
+ "position_ids": position_ids,
899
+ }
900
+ )
901
+ return model_inputs
902
+
903
+ @staticmethod
904
+ def _reorder_cache(past_key_values, beam_idx):
905
+ reordered_past = ()
906
+ for layer_past in past_key_values:
907
+ reordered_past += (
908
+ tuple(
909
+ past_state.index_select(0, beam_idx.to(past_state.device))
910
+ for past_state in layer_past
911
+ ),
912
+ )
913
+ return reordered_past
914
+
915
+
916
+ StableLMEpochConfig.register_for_auto_class()
917
+ StableLMEpochForCausalLM.register_for_auto_class("AutoModelForCausalLM")
tokenization_arcade100k.py ADDED
@@ -0,0 +1,273 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright (c) 2023 Alibaba Cloud & Stability AI.
3
+ #
4
+ # Tongyi Qianwen LICENSE AGREEMENT:
5
+ # https://github.com/QwenLM/Qwen/blob/5aa84bdfd3237b37f01bc88cd49b3279b9a71d0b/Tongyi%20Qianwen%20LICENSE%20AGREEMENT
6
+ """Tokenization classes for Arcade100k."""
7
+
8
+ import base64
9
+ import os
10
+ import unicodedata
11
+ from typing import Collection, Dict, List, Set, Tuple, Union
12
+
13
+ import tiktoken
14
+ from transformers.utils import logging
15
+ from transformers import PreTrainedTokenizer, AddedToken
16
+
17
+ logger = logging.get_logger(__name__)
18
+
19
+ VOCAB_FILES_NAMES = {"vocab_file": "arcade100k.tiktoken"}
20
+ NAME = "arcade100k"
21
+
22
+
23
+ def _load_tiktoken_bpe(tiktoken_bpe_file: str) -> Dict[bytes, int]:
24
+ with open(tiktoken_bpe_file, "rb") as f:
25
+ contents = f.read()
26
+ return {
27
+ base64.b64decode(token): int(rank)
28
+ for token, rank in (line.split() for line in contents.splitlines() if line)
29
+ }
30
+
31
+
32
+ ENDOFTEXT = "<|endoftext|>"
33
+ FIM = [
34
+ "<|fim_prefix|>",
35
+ "<|fim_middle|>",
36
+ "<|fim_suffix|>",
37
+ "<|fim_pad|>",
38
+ ]
39
+ # `StarCoder` Tokens
40
+ CODE = [
41
+ "<gh_stars>",
42
+ "<filename>",
43
+ "<issue_start>",
44
+ "<issue_comment>",
45
+ "<issue_closed>",
46
+ "<jupyter_start>",
47
+ "<jupyter_text>",
48
+ "<jupyter_code>",
49
+ "<jupyter_output>",
50
+ "<empty_output>",
51
+ "<commit_before>",
52
+ "<commit_msg>",
53
+ "<commit_after>",
54
+ "<reponame>",
55
+ ]
56
+ CHAT = [
57
+ "<|im_start|>", # Chat: Input message start
58
+ "<|im_end|>", # Chat: Input message end
59
+ ]
60
+ PAUSE = "<|pause|>" # Think before you speak (https://arxiv.org/abs/2310.02226)
61
+ REGISTERS = [
62
+ f"<|reg{i}|>" for i in range(0, 8)
63
+ ] # Register 0 sink token (https://arxiv.org/abs/2309.17453)
64
+ ENDOFPROMPT = "<|endofprompt|>"
65
+ SPECIAL_TOKENS_NAMES = (
66
+ [ENDOFTEXT]
67
+ + FIM
68
+ + CODE
69
+ + [ENDOFPROMPT]
70
+ + CHAT
71
+ + [PAUSE]
72
+ + REGISTERS
73
+ + ["<|extra0|>"]
74
+ )
75
+ START_ID = 100257
76
+ SPECIAL_TOKENS = {t: START_ID + i for i, t in enumerate(SPECIAL_TOKENS_NAMES)}
77
+
78
+
79
+ def _arcade100k(vocab_file: str):
80
+ mergeable_ranks = _load_tiktoken_bpe(vocab_file)
81
+
82
+ return {
83
+ "name": NAME,
84
+ "pat_str": r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+""",
85
+ "mergeable_ranks": mergeable_ranks,
86
+ "special_tokens": SPECIAL_TOKENS,
87
+ }
88
+
89
+
90
+ class Arcade100kTokenizer(PreTrainedTokenizer):
91
+ """
92
+ Construct a Arcade100k tokenizer backed by `tiktoken`.
93
+
94
+ Args:
95
+ vocab_file (`str`):
96
+ Path to the vocabulary file.
97
+ errors (`str`, *optional*, defaults to `"replace"`):
98
+ How to handle errors in decoding UTF-8 byte sequences.
99
+ WARNING: the default behaviour of this function is lossy, since decoded bytes are not
100
+ guaranteed to be valid UTF-8. You can control this behaviour using the `errors` parameter,
101
+ for instance, setting `errors=strict`.
102
+ """
103
+
104
+ vocab_files_names = VOCAB_FILES_NAMES
105
+ model_input_names = ["input_ids", "attention_mask"]
106
+
107
+ def __init__(
108
+ self,
109
+ vocab_file: str,
110
+ errors: str = "replace",
111
+ **kwargs,
112
+ ):
113
+ super().__init__(errors=errors, **kwargs)
114
+ self._tiktoken_config = _arcade100k(vocab_file)
115
+ self.tokenizer = tiktoken.Encoding(**self._tiktoken_config)
116
+
117
+ # TODO: Remove this assertion
118
+ assert (
119
+ len(self.tokenizer._mergeable_ranks)
120
+ + len(self.tokenizer._special_tokens)
121
+ + 1
122
+ == self.tokenizer.n_vocab
123
+ ), f"{len(self.tokenizer._mergeable_ranks) + len(self.tokenizer._special_tokens)} != {self.tokenizer.n_vocab} in encoding"
124
+
125
+ self.decoder = {i: n for n, i in self.tokenizer._mergeable_ranks.items()}
126
+ self.decoder.update({i: n for n, i in self.tokenizer._special_tokens.items()})
127
+ self.eos_token = self.decoder[self.tokenizer.eot_token]
128
+ self.pad_token = self.decoder[self.tokenizer.eot_token]
129
+
130
+ def __len__(self):
131
+ return self.tokenizer.n_vocab
132
+
133
+ @property
134
+ def vocab_size(self):
135
+ return self.tokenizer.n_vocab
136
+
137
+ def get_vocab(self) -> Dict[bytes, int]:
138
+ return self.tokenizer._mergeable_ranks
139
+
140
+ def convert_tokens_to_ids(
141
+ self, tokens: Union[bytes, str, List[Union[bytes, str]]]
142
+ ) -> List[int]:
143
+ ids = []
144
+ if isinstance(tokens, (str, bytes)):
145
+ if tokens in self.tokenizer._special_tokens:
146
+ return self.tokenizer._special_tokens[tokens]
147
+ else:
148
+ return self.tokenizer._mergeable_ranks.get(tokens)
149
+ for token in tokens:
150
+ if token in self.tokenizer._special_tokens:
151
+ ids.append(self.tokenizer._special_tokens[token])
152
+ else:
153
+ ids.append(self.tokenizer._mergeable_ranks.get(token))
154
+ return ids
155
+
156
+ def _add_tokens(
157
+ self,
158
+ new_tokens: Union[List[str], List[AddedToken]],
159
+ special_tokens: bool = False,
160
+ ) -> int:
161
+ if not special_tokens and new_tokens:
162
+ raise ValueError("Adding regular tokens is not supported")
163
+ for token in new_tokens:
164
+ surface_form = token.content if isinstance(token, AddedToken) else token
165
+ if surface_form not in SPECIAL_TOKENS:
166
+ raise ValueError("Adding unknown special tokens is not supported")
167
+ return 0
168
+
169
+ def save_vocabulary(self, save_directory: str, **kwargs) -> Tuple[str]:
170
+ """
171
+ Save only the vocabulary of the tokenizer (vocabulary).
172
+
173
+ Returns:
174
+ `Tuple(str)`: Paths to the files saved.
175
+ """
176
+ file_path = os.path.join(save_directory, "arcade100k.tiktoken")
177
+ with open(file_path, "w", encoding="utf8") as w:
178
+ for k, v in self.tokenizer._mergeable_ranks.items():
179
+ line = base64.b64encode(k).decode("utf8") + " " + str(v) + "\n"
180
+ w.write(line)
181
+ return (file_path,)
182
+
183
+ def tokenize(
184
+ self,
185
+ text: str,
186
+ allowed_special: Union[Set, str] = "all",
187
+ disallowed_special: Union[Collection, str] = (),
188
+ **kwargs,
189
+ ) -> List[Union[bytes, str]]:
190
+ """
191
+ Converts a string in a sequence of tokens.
192
+
193
+ Args:
194
+ text (`str`):
195
+ The sequence to be encoded.
196
+ allowed_special (`Literal["all"]` or `set`):
197
+ The surface forms of the tokens to be encoded as special tokens in regular texts.
198
+ Default to "all".
199
+ disallowed_special (`Literal["all"]` or `Collection`):
200
+ The surface forms of the tokens that should not be in regular texts and trigger errors.
201
+ Default to an empty tuple.
202
+
203
+ kwargs (additional keyword arguments, *optional*):
204
+ Will be passed to the underlying model specific encode method.
205
+
206
+ Returns:
207
+ `List[bytes|str]`: The list of tokens.
208
+ """
209
+ tokens = []
210
+ text = unicodedata.normalize("NFC", text)
211
+
212
+ # this implementation takes a detour: text -> token id -> token surface forms
213
+ for t in self.tokenizer.encode(
214
+ text, allowed_special=allowed_special, disallowed_special=disallowed_special
215
+ ):
216
+ tokens.append(self.decoder[t])
217
+ return tokens
218
+
219
+ def convert_tokens_to_string(self, tokens: List[Union[bytes, str]]) -> str:
220
+ """
221
+ Converts a sequence of tokens in a single string.
222
+ """
223
+ text = ""
224
+ temp = b""
225
+ for t in tokens:
226
+ if isinstance(t, str):
227
+ if temp:
228
+ text += temp.decode("utf-8", errors=self.errors)
229
+ temp = b""
230
+ text += t
231
+ elif isinstance(t, bytes):
232
+ temp += t
233
+ else:
234
+ raise TypeError("token should only be of type types or str")
235
+ if temp:
236
+ text += temp.decode("utf-8", errors=self.errors)
237
+ return text
238
+
239
+ def _convert_id_to_token(self, index: int) -> Union[bytes, str]:
240
+ """Converts an id to a token, special tokens included"""
241
+ if index in self.decoder:
242
+ return self.decoder[index]
243
+ raise ValueError("unknown ids")
244
+
245
+ def _convert_token_to_id(self, token: Union[bytes, str]) -> int:
246
+ """Converts a token to an id using the vocab, special tokens included"""
247
+ if token in self.tokenizer._special_tokens:
248
+ return self.tokenizer._special_tokens[token]
249
+ if token in self.tokenizer._mergeable_ranks:
250
+ return self.tokenizer._mergeable_ranks[token]
251
+ raise ValueError("unknown token")
252
+
253
+ def _tokenize(self, text: str, **kwargs):
254
+ """
255
+ Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based
256
+ vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).
257
+
258
+ Do NOT take care of added tokens.
259
+ """
260
+ raise NotImplementedError
261
+
262
+ def _decode(
263
+ self,
264
+ token_ids: Union[int, List[int]],
265
+ skip_special_tokens: bool = False,
266
+ errors: str = None,
267
+ **kwargs,
268
+ ) -> str:
269
+ if isinstance(token_ids, int):
270
+ token_ids = [token_ids]
271
+ if skip_special_tokens:
272
+ token_ids = [i for i in token_ids if i < self.tokenizer.eot_token]
273
+ return self.tokenizer.decode(token_ids)
tokenizer_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "Arcade100kTokenizer",
3
+ "auto_map": {
4
+ "AutoTokenizer": [
5
+ "tokenization_arcade100k.Arcade100kTokenizer",
6
+ null
7
+ ]
8
+ }
9
+ }