soka0000/vclm-KoCoder-7B

VCLM KoCoder 7B

Korean Code Generation Model based on Qwen2.5-7B-Instruct

---

📋 Model Description

VCLM KoCoder 7B is a specialized code generation model fine-tuned from soka0000/vclm-korean-7b using QLoRA on 40,000 high-quality code instruction datasets.

Key Features

• Korean & English: Bilingual code generation
• Multi-language: Python, Java, C++, JavaScript, SQL, etc.
• Math Preserved: Maintains strong mathematical reasoning (GSM8K: 77%+)
• Knowledge Retained: Korean language understanding preserved
• Efficient: Fine-tuned with QLoRA (4-bit quantization)

Base Model

• Base: soka0000/vclm-korean-7b (Soka1.0 architecture)
• Architecture: SokaForCausalLM (based on Qwen2.5-7B)
• Parameters: 7.62B
• Context Length: 32,768 tokens
• Base Training: 1M instruction tuning datasets (Science/Math, Humanities, Code, Korean)

---

Quick Start

Installation

pip install transformers torch accelerate

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "Soka0000/vclm-KoCoder-7B"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Generate code
messages = [
    {"role": "system", "content": "You are SokaAI, created by Soka0000 Korea. You are a helpful AI Assistant."},
    {"role": "user", "content": "Write a Python function to implement quicksort algorithm."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Korean Example

messages = [
    {"role": "system", "content": "You are SokaAI, created by Soka0000 Korea. You are a helpful AI Assistant."},
    {"role": "user", "content": "이진 탐색 트리를 구현하는 파이썬 클래스를 작성해줘."}
]

# ... (same generation code as above)

---

Performance

Training Results

Metric	Value
Final Loss	0.554
Token Accuracy	86.2%
Training Samples	40,000
Training Time	~3.5 hours (H100 80GB)

Inference Speed

• Generation Speed: 17-22 tokens/sec (H100)
• Context Window: 32,768 tokens
• Batch Size: Configurable

Code Generation Examples

Binary Search Tree (880 tokens generated)

• ✅ Complete class implementation
• ✅ Proper structure and methods
• ✅ Correct logic and syntax

---

🔧 Training Details

Fine-tuning Method

• Technique: QLoRA (Quantized Low-Rank Adaptation)
• Quantization: 4-bit (bitsandbytes)
• LoRA Rank: 32
• LoRA Alpha: 16
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Configuration

Base Model: soka0000/vclm-korean-7b
Fine-tuning: QLoRA
Epochs: 1
Batch Size: 2 (per device)
Gradient Accumulation: 8 (effective batch size: 16)
Learning Rate: 5e-5
Optimizer: paged_adamw_8bit
Scheduler: cosine with warmup (10%)
Max Length: 2048 tokens
Precision: bfloat16
Flash Attention: 2

Dataset Composition

Dataset	Samples	Weight
CodeAlpaca-20k	18,000	50%
Python Code Instructions	12,000	35%
Code Instructions 122k	10,000	15%
Total	40,000	100%

---

💡 Use Cases

1. Code Generation

# Generate functions, classes, algorithms
"Write a Python function to calculate factorial using recursion."

2. Code Explanation

# Explain code concepts
"Explain how binary search works with a C++ example."

3. Debugging & Optimization

# Help with code issues
"이 코드를 최적화해줘: [code snippet]"

4. Multi-language Support

# Python, Java, C++, JavaScript, SQL, etc.
"Implement quicksort in Java."

---

Limitations

• Code Execution: Model generates code but cannot execute it
• Complex Systems: May struggle with very large, multi-file projects
• Domain-Specific: May not cover all niche frameworks/libraries
• Verification: Always test generated code before production use

---

License

This model inherits the license from its base model soka0000/vclm-korean-7b.

Important: The base model has a custom license (LICENSE_NC). Please check the base model's license terms before commercial use.

---

Acknowledgments

• Base Model: soka0000/vclm-korean-7b
• Original Architecture: Qwen2.5-7B-Instruct by Alibaba Cloud
• Framework: Hugging Face Transformers
• Training Library: TRL, PEFT

---

Contact

• Author: Soka0000 Korea
• Model: SokaAI
• HuggingFace: @Soka0000

---

Citation

If you use this model, please cite:

@model{vclm-kocoder-7b,
  title={VCLM KoCoder 7B: Korean Code Generation Model},
  author={Soka0000},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/Soka0000/vclm-KoCoder-7B}
}

---

Made with ❤️ by Soka0000 Korea

🤗 Model • Documentation • Discussions