metadata

license: gemma
library_name: transformers
pipeline_tag: text-generation
base_model: google/gemma-2-9b-it
language:
  - en
  - zh
tags:
  - llama-factory
  - orpo

Model Summary

Gemma-2-9B-Chinese-Chat is the first instruction-tuned language model built upon google/gemma-2-9b-it for Chinese & English users with various abilities such as roleplaying & tool-using.

Developed by: Shenzhi Wang (王慎执) and Yaowei Zheng (郑耀威)

License: Gemma License
Base Model: google/gemma-2-9b-it
Model Size: 9.24B
Context length: 8K

1. Introduction

This is the first model specifically fine-tuned for Chinese & English users based on the google/gemma-2-9b-it with a preference dataset with more than 100K preference pairs. The fine-tuning algorithm we employ is ORPO [1].

Compared to the original google/gemma-2-9b-it, our Gemma-2-9B-Chinese-Chat model significantly reduces the issues of "Chinese questions with English answers" and the mixing of Chinese and English in responses, with enhanced performance in roleplay, tool using, and math.

[1] Hong, Jiwoo, Noah Lee, and James Thorne. "Reference-free Monolithic Preference Optimization with Odds Ratio." arXiv preprint arXiv:2403.07691 (2024).

Training framework: LLaMA-Factory.

Training details:

epochs: 3
learning rate: 3e-6
learning rate scheduler type: cosine
Warmup ratio: 0.1
cutoff len (i.e. context length): 8192
orpo beta (i.e. $\lambda$ in the ORPO paper): 0.05
global batch size: 128
fine-tuning type: full parameters
optimizer: paged_adamw_32bit

2. Usage

2.1 Usage of Our BF16 Model

Please upgrade the transformers package to ensure it supports Gemma-2 models. The current version we are using is 4.42.2.
Use the following Python script to download our BF16 model

from huggingface_hub import snapshot_download
snapshot_download(repo_id="shenzhi-wang/Gemma-2-9B-Chinese-Chat", ignore_patterns=["*.gguf"])  # Download our BF16 model without downloading GGUF models.

Inference with the BF16 model

import torch
import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "/Your/Local/Path/to/Gemma-2-9B-Chinese-Chat"
dtype = torch.bfloat16

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda",
    torch_dtype=dtype,
)

chat = [
    {"role": "user", "content": "写一首关于机器学习的诗。"},
]
input_ids = tokenizer.apply_chat_template(
    chat, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=8192,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
response = outputs[0][input_ids.shape[-1] :]
print(tokenizer.decode(response, skip_special_tokens=True))