karakuri-ai/karakuri-lm-8x7b-chat-v0.1

KARAKURI LM 8x7B Chat v0.1

Model Details

Model Description

• Developed by: KARAKURI Inc.
• Model type: Mixture of Experts (MoE)
• Languages: Primarily English and Japanese
• License: Apache 2.0
• Finetuned from model: tokyotech-llm/Swallow-MX-8x7b-NVE-v0.1
• Contact: For questions and comments about the model, please email karakuri-rd@karakuri.ai
• Demo: https://lm.karakuri.cc/

Usage

Warning

The prompt format has been changed from KARAKURI LM 70B Chat v0.1. Please make sure to follow the correct format. Otherwise, the model will generate sub-optimal outputs.

Prompt Format

We use the following prompt template of multi-turn conversation in the Mistral format, which includes an encoded string of multiple attribute values.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("karakuri-ai/karakuri-lm-8x7b-chat-v0.1")

messages = [
    {"role": "system", "content": "System prompt"},
    {"role": "user", "content": "User prompt"},
    {"role": "assistant", "content": "Model response"},
    {"role": "user", "content": "User prompt"},
]
tokenizer.apply_chat_template(messages, tokenize=False)
# <s>[INST] <<SYS>>
# System prompt
# <</SYS>>
#
# User prompt [ATTR] helpfulness: 4 correctness: 4 coherence: 4 complexity: 4 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]Model response</s>[INST] User prompt [ATTR] helpfulness: 4 correctness: 4 coherence: 4 complexity: 4 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]

The prompt template contains nine attributes. The first five are derived from HelpSteer, while the remaining four are derived from OASST2. The values are represented by integers ranging from 0 to 4, with 0 being the lowest and 4 being the highest.

• helpfulness (default: 4)
• correctness (default: 4)
• coherence (default: 4)
• complexity (default: 4)
• verbosity (default: 4)
• quality (default: 4)
• toxicity (default: 0)
• humor (default: 0)
• creativity (default: 0)

If you want to change attribute values from the default values specified in the template, you can modify them to any values by adding the attribute values to the user messages:

messages = [
    {"role": "user", "content": "User prompt", "helpfulness": 0, "complexity": 0},
]
tokenizer.apply_chat_template(messages, tokenize=False)
# <s>[INST] User prompt [ATTR] helpfulness: 0 correctness: 4 coherence: 4 complexity: 0 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]

Run the model

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "karakuri-ai/karakuri-lm-8x7b-chat-v0.1",
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": "週末に日帰りで東京に遊びに行こうと思っています。日帰りなので、短時間で回れるおすすめの観光プランを教えてください。",
    },
]

input_ids = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
).to(model.device)
outputs = model.generate(input_ids, max_new_tokens=512)
tokenizer.decode(outputs[0][input_ids.shape[-1]:])

Performance

Model	# Active Params	Alignment	MT-Bench-jp
Qwen1.5 72B Chat	72B	DPO	8.19
KARAKURI LM 8x7B Chat v0.1	13B	SteerLM	7.54
Command R+	104B	-	7.31
Mixtral 8x7B Instruct v0.1	13B	DPO	7.24
Llama 3 70B Instruct	70B	RLHF	7.13
KARAKURI LM 70B Chat v0.1	70B	SteerLM	6.43
Llama 2 70B Chat	70B	RLHF	5.23

Training Details

Training Data

• OASST2
• HelpSteer
• Internal Japanese dataset

Training Infrastructure

• Hardware: The model was trained on 8 nodes of an Amazon EC2 trn1.32xlarge instance.
• Software: We use code based on neuronx-nemo-megatron.

Citation

@misc {karakuri_lm_8x7b_chat_v01,
    author       = { {KARAKURI} {I}nc. },
    title        = { {KARAKURI} {LM} 8x7{B} {C}hat v0.1 },
    year         = { 2024 },
    url          = { https://huggingface.co/karakuri-ai/karakuri-lm-8x7b-chat-v0.1 },
    publisher    = { Hugging Face },
    journal      = { Hugging Face repository }
}