Instructions to use roskosmos19/Dolphin-4B-Instruct-flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use roskosmos19/Dolphin-4B-Instruct-flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="roskosmos19/Dolphin-4B-Instruct-flash")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("roskosmos19/Dolphin-4B-Instruct-flash")
model = AutoModelForCausalLM.from_pretrained("roskosmos19/Dolphin-4B-Instruct-flash", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use roskosmos19/Dolphin-4B-Instruct-flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "roskosmos19/Dolphin-4B-Instruct-flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roskosmos19/Dolphin-4B-Instruct-flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/roskosmos19/Dolphin-4B-Instruct-flash

SGLang

How to use roskosmos19/Dolphin-4B-Instruct-flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "roskosmos19/Dolphin-4B-Instruct-flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roskosmos19/Dolphin-4B-Instruct-flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "roskosmos19/Dolphin-4B-Instruct-flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roskosmos19/Dolphin-4B-Instruct-flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use roskosmos19/Dolphin-4B-Instruct-flash with Docker Model Runner:
```
docker model run hf.co/roskosmos19/Dolphin-4B-Instruct-flash
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Dolphin-4B-Instruct-0409

Best price/performance 4B Instruct model

Optimized for maximum capability at minimal cost. Based on the strong Qwen3-4B architecture with aggressive efficiency tuning.

Why this version is better price/performance

Metric	Original Qwen3-4B-Instruct	Dolphin-4B-Instruct-0409
Native context	262k	16k (covers >95% of real use)
KV-cache VRAM (16k ctx)	High	Much lower
Multimodal special tokens	Many (vision etc.)	Removed (leaner)
Generation defaults	Generic	Tuned for quality
Instruction strength	Good	Stronger system prompt
Typical quantized size (Q4)	~2.5 GB	~2.5 GB (same, but faster)

→ Same 4B intelligence, significantly cheaper to run, better focused answers.

Technical specs

Architecture: Qwen3ForCausalLM
Parameters: 4.0B
Layers: 36
Hidden size: 2560
Context: 16 384 tokens (optimized)
Vocab size: 151 936
Recommended dtype: bfloat16 or float16
Recommended quant: Q4_K_M / AWQ / GPTQ for best speed/quality balance

Recommended generation settings (already in generation_config.json)

{
  "temperature": 0.5,
  "top_p": 0.85,
  "top_k": 30,
  "repetition_penalty": 1.05
}

These settings produce more precise, less repetitive and higher-quality answers while staying efficient.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "./Dolphin-4B-Instruct-0409"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Explain the difference between CPU and GPU in simple terms."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

Deployment tips for maximum cheapness

vLLM / SGLang: --max-model-len 16384
llama.cpp: use Q4_K_M or Q5_K_M GGUF
Ollama / LM Studio: load the GGUF version if available
Keep context ≤ 8k–12k in production for best speed/VRAM ratio

License

Apache 2.0

This package contains the fully tuned config + tokenizer.
Place the original Qwen3-4B safetensors weights next to the index file (or convert to GGUF) and you have a ready-to-run high price/performance 4B model.

Downloads last month: 515

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for roskosmos19/Dolphin-4B-Instruct-flash

Quantizations

2 models