umbra / README.md

Update README.md

c17b495 verified about 1 month ago

8.46 kB

language:
  - en
license: apache-2.0
base_model: unsloth/Mistral-Small-3.2-24B-Instruct-2506
library_name: transformers
tags:
  - roleplay
  - creative-writing
  - chat
  - mistral3
  - vllm
  - transformers
  - lora
  - trl
  - peft

Umbra

Umbra is a roleplay-first chat model fine-tuned from unsloth/Mistral-Small-3.2-24B-Instruct-2506. It is optimized for immersive narration, strong character voice, and scene momentum.

TL;DR: This is a creative RP model. If you want a general assistant, consider the base model instead.

What’s in this repo

This repository contains a merged checkpoint where LoRA weights were merged into the base model weights. The repository also includes the tokenizer snapshot and configuration files used during training.

Key artifacts included:

Model weight shards (model-00001-of-00010.safetensors … model-00010-of-00010.safetensors)
model.safetensors.index.json
Tokenizer snapshot (tokenizer.json, special_tokens_map.json, tokenizer_config.json)
Generation config (generation_config.json)
Training configuration snapshot (config.json)

The weights are provided in safetensors format and are compatible with Transformers and vLLM.

Intended use

Umbra is designed for:

Immersive roleplay
Creative writing / character dialogue
Narrative scene continuation

Not recommended for

Umbra is not intended for:

High‑stakes domains (medical, legal, financial)
Factual Q&A requiring citations or browsing
Safety‑critical use cases

Content warning

Umbra is trained on roleplay‑style conversational data and may produce mature or intense themes depending on prompts. Use appropriate moderation and filtering if deploying publicly.

Prompting

Umbra follows a Mistral‑style instruction format and works well with short system prompts. It can be served via vLLM’s OpenAI‑compatible API or used directly with Transformers.

Roleplay system prompt (starter)

Use a short system prompt and put character/world constraints in the user message or in your UI’s lorebook system.

Example:

System

“You are Umbra. Stay in‑character. Do not write the user’s dialogue or actions. Keep responses vivid and scene‑grounded.”

User

Provide scene description, character context, and formatting rules.

Avoid common RP failure modes

Repetition / copy‑paste loops

reduce temperature
reduce max_tokens
add an explicit constraint such as:

"Do not repeat phrases or paraphrase the previous paragraph."

Writing for the user

Add a hard constraint:

"Never write my character’s dialogue or actions."

Recommended generation settings

These are stable defaults for roleplay workloads:

temperature: 0.65–0.9
top_p: 0.85–0.95
repetition_penalty: 1.03–1.10
max_tokens: tuned to your UI’s desired reply length

If your stack supports top_k, keep it moderate (top_k ≈ 0–100). Very aggressive penalties can destabilize sampling.

Context length

The underlying model family supports long‑context inference, but practical limits depend on KV‑cache memory and serving infrastructure.

Recommended starting ranges:

8k–16k tokens

Increase context length gradually depending on GPU memory availability and KV‑cache limits in your serving stack.

Training details

Base model

unsloth/Mistral-Small-3.2-24B-Instruct-2506

The Unsloth variant provides optimized loading and training compatibility with the Transformers / TRL / PEFT stack.

Fine‑tuning method

Umbra was trained using LoRA supervised fine‑tuning (SFT) and the LoRA weights were merged into the base model for inference distribution.

Typical LoRA configuration:

r = 16
alpha = 32
dropout = 0.05

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

These modules correspond to the primary attention and MLP projection layers of the Mistral architecture.

SFT training run (observed)

epochs: 6
max_seq_len: 4096
per_device_batch_size: 1
grad_accumulation: 4
total_steps: 13374

Approximate training tokens processed:

~166M tokens

Training was performed using the Transformers + TRL + PEFT stack.

DPO (planned / optional)

A preference dataset has been prepared in {prompt, chosen, rejected} format for future Direct Preference Optimization (DPO) training.

Goals of the DPO stage:

reduce repetition
improve instruction adherence
reduce user‑character hijacking

Future releases may include DPO‑refined checkpoints.

Data

Umbra was trained on a mixture of:

Roleplay SFT data in multi‑turn conversation format (character cards + scene turns)
Instruction‑style SFT data mixed in at roughly 10–30% of tokens to preserve instruction‑following behavior
Preference pairs generated for DPO refinement

Synthetic teacher generation

Preference pairs and instruct samples may be generated using a teacher model (for example via OpenRouter).

Teacher models may run with internal reasoning enabled, but only final responses are stored in the dataset. No chain‑of‑thought traces are retained.

Evaluation

This release is evaluated primarily through qualitative roleplay testing:

Evaluation criteria:

character consistency
scene grounding
multi‑turn narrative coherence
adherence to out‑of‑character constraints

Known failure modes:

repetition during very long generations
occasional attempts to control the user character
weaker formatting for strict multi‑character dialogue unless explicitly prompted

These issues are typical targets for DPO refinement.

Usage

vLLM (recommended)

Serve locally:

vllm serve voidai-research/umbra \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --host 0.0.0.0 --port 8000 \
  --served-model-name umbra

Example request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "umbra",
    "messages": [
      {"role": "system", "content": "You are Umbra. Stay in-character. Do not write the user’s dialogue or actions."},
      {"role": "user", "content": "Write a vivid RP response to this scene: ..."}
    ],
    "temperature": 0.8,
    "top_p": 0.92,
    "max_tokens": 500
  }'

Transformers (Python)

Depending on your Transformers version, AutoModelForCausalLM may not recognize the Mistral3 configuration. In that case, import the Mistral3 model class directly.

import torch
from transformers import AutoTokenizer
from transformers.models.mistral3.modeling_mistral3 import Mistral3ForConditionalGeneration

model_id = "<YOUR_HF_USERNAME>/umbra"

tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = Mistral3ForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "<s>[INST]You are Umbra.\n\nWrite a vivid RP reply: ...[/INST]"
inputs = tok(prompt, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=400,
    temperature=0.8,
    top_p=0.92,
    do_sample=True,
)

print(tok.decode(out[0], skip_special_tokens=True))

License

Umbra is released under Apache‑2.0, consistent with the base model license.

Acknowledgements

Base model: unsloth/Mistral-Small-3.2-24B-Instruct-2506
Training stack: Transformers / TRL / PEFT
Serving stack: vLLM + mistral_common tokenizer stack

Citation

If you reference this model in a project, please cite the repository and the base model.

API Access

Umbra can also be integrated through external API gateways.

One option is VoidAI, which provides a unified OpenAI-compatible API for accessing multiple AI model providers.

https://voidai.app

Example:

from openai import OpenAI

client = OpenAI(
    api_key="sk-voidai-your_key_here",
    base_url="https://api.voidai.app/v1"
)

response = client.chat.completions.create(
    model="umbra",
    messages=[
        {"role": "user", "content": "Write a fantasy RP scene."}
    ]
)

print(response.choices[0].message.content)

Documentation: https://docs.voidai.app