metadata

library_name: transformers
tags:
  - llm-jp
  - japanese
  - instruction-tuning

Model Card for yuhkis/llm-jp-3-13b-finetune

Model Details

Model Description

This is a LoRA-tuned version of LLM-jp-3-13b, fine-tuned on the Ichikara Instruction dataset.

Developed by: Yuhki Shiraishi
Model type: Instruction-tuned Japanese Language Model
Language: Japanese
License: CC-BY-NC-SA
Finetuned from model: llm-jp/llm-jp-3-13b

Uses

Direct Use

To use this model for inference:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "yuhkis/llm-jp-3-13b-finetune"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    token=HF_TOKEN
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True, token=HF_TOKEN)

Output Format

The model outputs results in JSONL format with required fields:

task_id: Task identifier
output: Generated response

Example output:

{"task_id": 0, "output": "応答テキスト"}

Out-of-Scope Use

This model should not be used for:

Commercial applications due to license restrictions
Critical decision-making without human oversight
Applications requiring strict reliability guarantees

Bias, Risks, and Limitations

The model inherits biases from its training data
Output quality may vary depending on input complexity
The model should not be used for making critical decisions without human oversight

Recommendations

Users should be aware of the model's limitations and verify outputs when used in applications.

Training Details

Training Data

Dataset: Ichikara Instruction Dataset

Training Procedure

Training regime: bf16 mixed precision
Library: 🤗 Transformers
Optimization: LoRA (Low-Rank Adaptation)

Technical Specifications

Model Architecture

Base model: LLM-jp-3-13b
Adaptation method: LoRA

Citation

BibTeX:

@misc{shiraishi2024llm,
    title={LLM-jp-3-13b-finetune: Instruction-tuned Japanese Language Model},
    author={Yuhki Shiraishi},
    year={2024},
    publisher={Hugging Face},
    howpublished={\url{https://huggingface.co/yuhkis/llm-jp-3-13b-finetune}}
}

Base Model Citation:

@misc{llm-jp2024,
    title={LLM-jp-3: Large Language Model for Japanese},
    author={LLM-jp Project Team},
    year={2024},
    publisher={Hugging Face},
    howpublished={\url{https://huggingface.co/llm-jp/llm-jp-3-13b}}
}

Training Data Citation:

関根聡, 安藤まや, 後藤美知子, 鈴木久美, 河原大輔, 井之上直也, 乾健太郎. 
ichikara-instruction: LLMのための日本語インストラクションデータの構築. 
言語処理学会第30回年次大会(2024)

Model Card Contact

Primary Contact:

Name: Yuhki Shiraishi
GitHub: @yuhkis

For questions regarding this model, please open an issue in the GitHub repository or contact via HuggingFace discussion forum.

Please include "LLM-jp-3-13b-finetune" in the subject line of any correspondence.