initial commit

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 SB Intuitions
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,64 @@
 ---
 license: mit
+language:
+- ja
 ---
+
+# Sarashina1-65B
+
+This repository provides Japanese language models trained by [SB Intuitions](https://www.sbintuitions.co.jp/).
+
+
+## How to use
+
+```
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
+ 
+model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina1-65b", torch_dtype=torch.float16)
+tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina1-65b", use_fast=False)
+generator = pipeline("text-generation", model=model, tokenizer=tokenizer, device_map="auto")
+set_seed(123)
+ 
+text = generator(
+    "おはようございます、今日の天気は",
+    max_length=30,
+    do_sample=True,
+    pad_token_id=tokenizer.pad_token_id,
+    num_return_sequences=3,
+)
+ 
+for t in text:
+  print(t)
+ 
+```
+
+## Configuration
+
+| Parameters | Vocab size | Trainning tokens |  Architecture | Position type | Layers | Hidden dim | Attention heads |
+| :-----: | :-----------: | :-------------:  | :----------- | :-----------: | :----: | :--------: | :-------------: |
+| [7B](https://huggingface.co/sbintuitions/sarashina1-7b)      | 51200         | 1.0T             | GPTNeoX      | RoPE          | 32     | 4096       | 32 |
+| [13B](https://huggingface.co/sbintuitions/sarashina1-13b)     | 51200         | 1.0T             | GPTNeoX      | RoPE          | 40     | 5120       | 40 |
+| [65B](https://huggingface.co/sbintuitions/sarashina1-65b)     | 51200         | 800B             | GPTNeoX      | RoPE          | 80     | 8192       | 64 |
+
+## Training Corpus
+
+We used a Japanese portion of the [Common Crawl corpus](https://commoncrawl.org/), which is the largest Web corpus, as our training dataset.
+To clean the training corpus, we used [CCNet](https://github.com/facebookresearch/cc_net) and [HojiChar](https://github.com/HojiChar/HojiChar).
+After cleaning, our corpus contains about 550B tokens.
+
+## Tokenization
+
+We use a [sentencepiece](https://github.com/google/sentencepiece) tokenizer with a unigram language model and byte-fallback.
+We do not apply pre-tokenization with Japanese tokenizer.
+Thus, a user may directly feed raw sentences into the tokenizer.
+
+
+## Ethical Considerations and Limitations
+Sarashina1 has not been tuned to follow an instruction yet.
+Therefore, sarashina1 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs.
+Before using sarashina1, we would like developers to tune models based on human preferences and safety considerations.
+
+## License
+
+[MIT License](https://huggingface.co/sbintuitions/sarashina1-65b/blob/main/LICENSE)

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 2,
+  "transformers_version": "4.30.2"
+}

pytorch_model-00001-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84eb6bee9b8ed27a2fe958ad39f906751764840d3d6c301e5e25de29d76faf86
+size 9992125001

pytorch_model-00002-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c875867e1b9a8d6af8f1b793074de62d1da5772064180973db67a671eafa5ce0
+size 9690151899

pytorch_model-00003-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d4112c53c0e5de38baf8fc27ef47e1b869d35ce115f9759151acf6cd9f581f7
+size 9690151963

pytorch_model-00004-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:877a6f7b57821a80715c8bf38c3eb37ce1f5645936b64e3ce240527d90895629
+size 9690151963

pytorch_model-00005-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46a6dc01123416e7f76388f0ca28780c8df7a71534db93dd5642ab8c955514ef
+size 9690151963

pytorch_model-00006-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dcaf6a8ba680a21b742a8eff59e8695900527b528f375bf2eb0d0f9ff44f8865
+size 9690151963

pytorch_model-00007-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf28019c7eb3cc4913e34719a4654b43b81818989aca38ee3dac741153036be5
+size 9690151963

pytorch_model-00008-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d5dad37e3b7973b01db3aad57ef123383694a25045431dfe97d2ae3fd4d8fca
+size 9690151963

pytorch_model-00009-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a93fa34b1b10f1524527086e9d7c79488a12024c9d864b23fb9109357f72873a
+size 9690151963

pytorch_model-00010-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a73a5c570feb117fc7f8024cf08b4b92550f70c58684dbba0249ed94c8f58c77
+size 9690151963

pytorch_model-00011-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abf691c3f0661c640bd5cdd2f117ffb5f1046e7129aacce58155a1baf03462b6
+size 9690151963

pytorch_model-00012-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:baa7ef6e4ec69d7624410623eac6bef6b3f70262e3d688dc72bde32c86e949c1
+size 9690151963

pytorch_model-00013-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:129bd9c4591d055821cf960c8821ed9b0f93d844e95c8885d7fb41284a8d4eb6
+size 9690151963

pytorch_model-00014-of-00014.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:088716553a3001db257b113baef3feb8c13b3053d9abcb2b296197b0423cc53e
+size 4605832921

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cc4aebe7cab049f43d2666c5250b5bfd97a13e0e7af80b584876c448d636e59
+size 1198392

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+    "extra_ids": 0,
+    "do_lower_case": false,
+    "keep_accents": true,
+    "bos_token": "<s>",
+    "eos_token": "</s>",
+    "unk_token": "<unk>",
+    "pad_token": "<pad>",
+    "mask_token": "<mask>",
+    "cls_token": "<cls>",
+    "sep_token": "<sep>",
+    "sp_model_kwargs": {},
+    "special_tokens_map_file": null,
+    "tokenizer_class": "T5Tokenizer"
+}