Instructions to use jialinyyzz/humanizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jialinyyzz/humanizer with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jialinyyzz/humanizer")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("jialinyyzz/humanizer")
model = AutoModelForMultimodalLM.from_pretrained("jialinyyzz/humanizer", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use jialinyyzz/humanizer with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jialinyyzz/humanizer:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jialinyyzz/humanizer:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jialinyyzz/humanizer:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jialinyyzz/humanizer:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf jialinyyzz/humanizer:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf jialinyyzz/humanizer:Q4_K_M

Use Docker

docker model run hf.co/jialinyyzz/humanizer:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use jialinyyzz/humanizer with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jialinyyzz/humanizer"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jialinyyzz/humanizer",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jialinyyzz/humanizer:Q4_K_M

SGLang

How to use jialinyyzz/humanizer with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jialinyyzz/humanizer" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jialinyyzz/humanizer",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jialinyyzz/humanizer" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jialinyyzz/humanizer",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use jialinyyzz/humanizer with Ollama:
```
ollama run hf.co/jialinyyzz/humanizer:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use jialinyyzz/humanizer with Docker Model Runner:
```
docker model run hf.co/jialinyyzz/humanizer:Q4_K_M
```

Lemonade

How to use jialinyyzz/humanizer with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jialinyyzz/humanizer:Q4_K_M

Run and chat with the model

lemonade run user.humanizer-Q4_K_M

List all available models

lemonade list

Atomic Chat

Speed on M5 Ultra: MLX 8-bit 69 tok/s, llama.cpp Q8_0 61, DSpark draft x1.3

#30

by tristanbes - opened 17 days ago

Discussion

tristanbes

17 days ago

•

edited 17 days ago

Hugging Face CLI

Speed numbers for the 12B on a Mac Studio M5 Ultra (256 GB), in case they help people pick a runtime. The README has M5 Max figures; on the Ultra the order flips and MLX comes out ahead of llama.cpp.

Runtime	File	Decode tok/s (median)	Prefill tok/s (median)	Peak memory
MLX 8-bit (mlx-lm 0.32.2)	converted from `model.safetensors` with the README command (`-q --q-bits 8 --q-group-size 64`)	69.3 EN / 69.1 ZH	1,627	13.0 GB
llama.cpp Q8_0, Metal (0.7.0, build 11669)	`humanizer-12b-Q8_0.gguf`, sha256 checked	61.0 EN / 61.0 ZH	1,333
MLX 4-bit target + Draftfit DSpark draft	`mlx-community/humanizer-12b-OptiQ-4bit` @ e0e2552, runner settings from the draft card	92.6 (vs 71.4 target alone, ×1.30)

How it was measured

5 drafts (email, blog, report, essay, Chinese email; 50–80 words each), 3 runs each, medians. Rewrites were 60–105 tokens, so these are short-output numbers.
MLX 8-bit and llama.cpp: exact prompt from prompt_format.json, temperature 1.0, top-p 0.95, top-k 0, min-p 0, repeat penalty 1.0, stop on EOS only. llama-server with -ngl 99 -c 8192 -fa on, cache_prompt: false.
DSpark: the draft card's setup (mlx 0.32.0, mlx-lm 0.31.3, patched mlx-dspark @ 4143092, FP32 target, 4-bit draft, greedy, draft cap 3, no lookup), up to 256 tokens. 15 of 15 outputs matched the target-only tokens exactly. Mean accepted length 2.4–2.9 per round; the Chinese draft gained least (×1.21; the report gained most, ×1.51).
Machine otherwise idle, on AC power, no thermal warnings. Served through oMLX the 8-bit model gives about 61 tok/s end to end, HTTP included.

Takeaways for Apple silicon owners

On an M5 Ultra, MLX 8-bit is about 14% faster than llama.cpp Q8_0 at decode and about 22% faster at prefill.
The DSpark draft reproduces on the Ultra: about ×1.3 with identical output (the card reports +43% on an M5 Pro).

Thanks for the model and the careful docs, they made this easy to reproduce.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment