|
---
|
|
license: mit
|
|
language:
|
|
- ja
|
|
- en
|
|
---
|
|
|
|
# Sarashina2-13B
|
|
|
|
This repository provides large language models trained by [SB Intuitions](https://www.sbintuitions.co.jp/).
|
|
|
|
|
|
## How to use
|
|
|
|
```
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2-13b", torch_dtype=torch.bfloat16)
|
|
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-13b", use_fast=False)
|
|
generator = pipeline("text-generation", model=model, tokenizer=tokenizer, device_map="auto")
|
|
set_seed(123)
|
|
|
|
text = generator(
|
|
"おはようございます、今日の天気は",
|
|
max_length=30,
|
|
do_sample=True,
|
|
pad_token_id=tokenizer.pad_token_id,
|
|
num_return_sequences=3,
|
|
)
|
|
|
|
for t in text:
|
|
print(t)
|
|
|
|
```
|
|
|
|
## Configuration
|
|
|
|
| Parameters | Vocab size | Trainning tokens | Architecture | Position type | Layers | Hidden dim | Attention heads |
|
|
| :-----: | :-----------: | :-------------: | :------------ | :-----------: | :----: | :--------: | :-------------: |
|
|
| [7B](https://huggingface.co/sbintuitions/sarashina2-7b) | 102400 | 2.1T | Llama2 | RoPE | 32 | 4096 | 32 |
|
|
| [13B](https://huggingface.co/sbintuitions/sarashina2-13b) | 102400 | 2.1T | Llama2 | RoPE | 40 | 5120 | 40 |
|
|
| 70B (TBA)| | | | | | |
|
|
|
|
## Training Corpus
|
|
|
|
For our Japanese training data, we used a Japanese portion of the [Common Crawl corpus](https://commoncrawl.org/), which is the largest Web corpus, as our training dataset.
|
|
To clean the training corpus, we used [CCNet](https://github.com/facebookresearch/cc_net) and [HojiChar](https://github.com/HojiChar/HojiChar).
|
|
After cleaning, our Japanese training data contains about 1T tokens.
|
|
|
|
For our English training data, we extracted English documents from [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B) but we removed books3 corpus due to copyright infringement.
|
|
|
|
## Tokenization
|
|
|
|
We use a [sentencepiece](https://github.com/google/sentencepiece) tokenizer with a unigram language model and byte-fallback.
|
|
We do not apply pre-tokenization with Japanese tokenizer.
|
|
Thus, a user may directly feed raw sentences into the tokenizer.
|
|
|
|
|
|
## Ethical Considerations and Limitations
|
|
Sarashina2 has not been tuned to follow an instruction yet.
|
|
Therefore, sarashina2 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs.
|
|
Before using sarashina2, we would like developers to tune models based on human preferences and safety considerations.
|
|
|
|
## License
|
|
|
|
[MIT License](https://huggingface.co/sbintuitions/sarashina2-7b/blob/main/LICENSE) |
