neural-nova/gpt-oss-120b-Instruct-Optimized

GPT-OSS-120B — Tuned vLLM Serving Recipe

Docker build, launch script, and benchmark-reproduction script for serving openai/gpt-oss-120b with vLLM. Source benchmark: neural-nova.com/benchmark/gpt-oss-120b, reporting +24.5% output tok/s and ~20% lower cost per token for the optimized config over baseline on a single H100.

This repository does not contain model weights. vllm serve pulls them directly from the upstream repo above (and caches them) when the server starts.

Model Overview

• Sparse MoE causal LM with attention sinks
• MoE expert weights in MXFP4; attention, norms, embeddings, and router remain BF16

Environment

Hardware	1x NVIDIA H100-80GB
Framework	vllm bench serve
Base Docker image	vllm/vllm-openai:v0.24.0
Dataset	random synthetic dataset, range ratio 0.8
vLLM version	v0.24.0
Parallelism	Tensor-parallel 1
Features	FlashAttention, prefix caching, chunked prefill, async scheduling
Input Length	409–3,686 tokens (nominal 2,048)
Output Length	51–460 tokens (nominal 256)
Prompts Measured	200 prompts, 10 warm-ups excluded
Request Rate	inf
Tested Concurrency	16

Optimization Changes

• --gpu-memory-utilization 0.92
• --max-num-batched-tokens 32768, --max-num-seqs 128
• --watermark 0.01, --block-size 16
• --attention-backend FLASH_ATTN
• Enabled prefix caching, chunked prefill, and async scheduling
• VLLM_FLOAT32_MATMUL_PRECISION=medium, plus usage tracking disabled (VLLM_NO_USAGE_STATS=1, VLLM_DO_NOT_TRACK=1)

1. Docker

Installation

Build the image:

docker build -t your-org/gpt-oss-120b-vllm:latest .

Run it (weights download into a cache volume on first start):

docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  your-org/gpt-oss-120b-vllm:latest optimized

Pass baseline instead of optimized as the final argument to run the unoptimized configuration used for the comparison below.

Baseline

vllm serve openai/gpt-oss-120b --tensor-parallel-size 1 --max-model-len 16384

Optimized

VLLM_FLOAT32_MATMUL_PRECISION=medium · VLLM_NO_USAGE_STATS=1 · VLLM_DO_NOT_TRACK=1 · vllm serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 --trust-remote-code --gpu-memory-utilization 0.92 --max-num-batched-tokens 32768 --max-num-seqs 128 --watermark 0.01 --block-size 16 --attention-backend FLASH_ATTN --enable-prefix-caching --performance-mode throughput --async-scheduling --enable-chunked-prefill --max-model-len 16384

Validation

Once the container is up, confirm the server is healthy and serving:

curl -sf http://localhost:8000/health && echo "server is healthy"

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-oss-120b", "prompt": "The capital of France is", "max_tokens": 16}'

Files in this repository

File	Purpose
Dockerfile	vllm/vllm-openai:v0.24.0-based image with the two scripts below baked in
launch_server.sh	Starts vllm serve with the baseline or optimized flags
run_benchmark.sh	Starts baseline + optimized servers, benchmarks each, and prints the comparison table below — fully automated
README.md	This model card

2. Shell Script

Launch Server

./launch_server.sh optimized   # or: ./launch_server.sh baseline

This is also the image's default command — docker run ... <image> optimized runs the same thing. Inside a running container, invoke it with docker exec -it <container> ./launch_server.sh <mode>.

Reproduce Benchmark

This is a separate container invocation from "Launch Server" above — it overrides the entrypoint so run_benchmark.sh (not launch_server.sh) drives the container. It starts the baseline server itself, benchmarks it, shuts it down, starts the optimized server, benchmarks that, shuts it down, and prints the comparison table — nothing to copy in by hand:

docker run --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint ./run_benchmark.sh \
  your-org/gpt-oss-120b-vllm:latest both

That prints the full table in the terminal when it finishes. Pass baseline or optimized instead of both to run just one side. The raw per-run JSON (benchmark_baseline.json, benchmark_optimized.json) and each server's startup log (server_baseline.log, server_optimized.log) are left in the container's /workspace if you want them too:

docker cp <container>:/workspace/benchmark_optimized.json .

Methodology (vllm bench serve):

• Dataset: random (synthetic), range ratio 0.8
• Input length: 409–3,686 tokens (nominal 2,048, uniform)
• Output length: 51–460 tokens (nominal 256, uniform)
• 200 measured prompts, 10 warm-up prompts (discarded, not saved)
• Request rate: unbounded (inf) — max concurrency is the only throttle
• Max concurrency: 16 (override with -e MAX_CONCURRENCY=...)

Results

1x H100-80GB, vLLM v0.24.0. This is exactly the table run_benchmark.sh both prints — running it is the only step required to reproduce it:

Metric	Baseline	Optimized
Output tok/s	364	453
TTFT median (s)	14.6	11.4
TPOT median (ms)	169	127
ITL mean (ms)	65	51
Request waiting (avg)	35.1	35.2