hiepnh137/llm-serving

gpt-oss serving

vLLM serves gpt-oss on one H200 as a Slurm job. Other jobs on the cluster reach it over the internal network through its OpenAI-compatible API.

Copy this directory anywhere you can write to, preferably a large filesystem, since the weights and env take about 90 GB. Run every command from inside it. All paths are relative to it, so each user gets their own env, weights, API key and endpoints. To keep the env or weights somewhere else, set LLM_VLLM_ENV or LLM_HF_HOME, or make env / hf_cache symlinks here.

1. Build the environment (login node, once)

./install_env.sh                           # conda env in ./env

This creates a Python 3.12 conda env with vllm==0.19.1 (torch 2.10, CUDA 12.8) and the hf CLI, then checks it by importing vLLM. It takes about 10 minutes and 12 GB. Pip's cache goes to .cache/pip here, because a small home quota can't hold it. conda comes from module load Anaconda3 if it isn't already on your PATH. To build somewhere else, pass the path (./install_env.sh /path/to/env) and set LLM_VLLM_ENV to it, or symlink it as env.

Don't upgrade vLLM past 0.19.1: see the driver note under Notes.

2. Download weights (login node, once)

./download_weights.sh                      # 20b (14 GB) + 120b (65 GB) into ./hf_cache

3. Start the server

Two instances, 2 H200s each

Two separate 5-day jobs. Each one runs a single server split across its 2 GPUs with tensor parallelism:

mkdir -p logs
sbatch --gres=gpu:h200:2 serve_gpt_oss.sbatch openai/gpt-oss-120b --tensor-parallel-size 2
LLM_INSTANCE=2 sbatch --gres=gpu:h200:2 serve_gpt_oss.sbatch openai/gpt-oss-120b --tensor-parallel-size 2

• --gres=gpu:h200:2 asks Slurm for 2 GPUs. --tensor-parallel-size 2 tells vLLM to use both. Keep the two numbers equal.

• The first job publishes endpoints/gpt-oss-120b.env and the second publishes endpoints/gpt-oss-120b-2.env. Clients pick one with client_env.sh gpt-oss-120b or client_env.sh gpt-oss-120b-2. Both serve the model name gpt-oss-120b.

• Each job holds 2 GPUs for 5 days, so a 2-GPU slot can take longer to free up than a 1-GPU one. sbatch --test-only ... shows the estimated start time without submitting.

• To chain these jobs, pass the GPU count as SBATCH_GRES=gpu:h200:2 in place of --gres. A chained successor inherits environment variables, but it doesn't inherit command-line flags. With --gres, the successor would get the script's default of 1 GPU and fail with --tensor-parallel-size 2:

  LLM_CHAIN=1 SBATCH_GRES=gpu:h200:2 sbatch serve_gpt_oss.sbatch openai/gpt-oss-120b --tensor-parallel-size 2
  LLM_CHAIN=1 SBATCH_GRES=gpu:h200:2 LLM_INSTANCE=2 sbatch serve_gpt_oss.sbatch openai/gpt-oss-120b --tensor-parallel-size 2
  

Check both with squeue -u $USER. Each job is ready when its log prints ready: http://....

Loading takes a few minutes (about 2.5 for the 20b). When it's ready, logs/gpt-oss-serve-<jobid>.log prints ready: http://..., and endpoints/<model>.env holds the URL. vLLM's startup log also reports the real KV cache size (GPU KV cache size: N tokens, Maximum concurrency for 131,072 tokens per request — 2.44 M tokens and 35x for the 20b on one H200).

Pass extra vLLM flags after the model name, e.g. sbatch serve_gpt_oss.sbatch openai/gpt-oss-120b --max-num-seqs 64.

4. Use it from another job

source <llm-serving dir>/client_env.sh gpt-oss-120b   # waits for the server (up to 1 h)
python my_script.py

import os
from openai import OpenAI

client = OpenAI()  # picks up OPENAI_BASE_URL and OPENAI_API_KEY
resp = client.chat.completions.create(
    model=os.environ["LLM_MODEL"],
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Check it by hand with ./smoke_test.sh gpt-oss-120b.

5. Use it from outside the cluster

A machine with no cluster filesystem and no route to the compute nodes can still reach the server by forwarding a port through the login node, which does have both:

LLM_REMOTE_DIR=<llm-serving dir on the cluster> ./tunnel.sh gpt-oss-120b 8000   # prints env vars, holds the tunnel

Then, in another shell on that machine, OPENAI_BASE_URL=http://localhost:8000/v1 works like any other client. The tunnel dies with the server job, and the port changes on every restart, so re-run tunnel.sh after resubmitting.

Notes

• The API key is generated on the first server run in .secrets/api_key (mode 600). Only your own jobs can read it, and the server requires it on every /v1 request.
• The server picks a free port each run, so always read the URL from client_env.sh and never hard-code it.
• The vLLM env defaults to ./env (vllm 0.19.1, torch 2.10+cu128, built by install_env.sh). Override with LLM_VLLM_ENV=... sbatch ....
• Submit serve_gpt_oss.sbatch from this directory. Slurm runs a copy of the script, so it finds config.sh through the submit directory, and the log goes to logs/ there.
• Keep vLLM at 0.19.1 or older: it is the last release on torch 2.10 / CUDA 12.8. The nodes' driver 565 is CUDA 12.6, and 0.20+ pulls torch 2.11 / CUDA 13, which that driver cannot run.
• The server is handed the cached snapshot directory, not the repo id: download_weights.sh skips the original/ and metal/ copies (~130 GB extra for the 120b), and vLLM's own lookup by repo id rejects a cache missing them. Both scripts read the skip list from LLM_HF_IGNORE in config.sh, so change it in one place.
• torch.compile and Triton caches go to .cache/ here rather than ~/.cache, which would eat a small home quota.
• --gpu-memory-utilization is 0.88 because the 120b OOMed at 0.92 while capturing CUDA graphs, which happens after the KV cache is allocated: 61 GB of weights left no headroom. Measured on one H200: 792 k tokens of KV cache (11.4x concurrency at 131 k context) for the 120b, 2.44 M (35x) for the 20b. PYTORCH_ALLOC_CONF=expandable_segments:True is set for the same reason - the failed run had 12 GiB reserved but unallocated.
• The job stops at its time limit. Resubmit it, submit with LLM_CHAIN=1, or queue a follow-on by hand with sbatch --dependency=afterany:<jobid> serve_gpt_oss.sbatch.