Performance gpt-oss-120b-dflash with aipeft
Hi team,
Thank you for contributing such a high-quality technique and repository. The idea of using diffusion to accelerate inference is quite interesting.
We conducted a few small experiments comparing inference speed of the model with and without DFlash on two H100 GPUs. However, as we increased the CCU, it seems that the version using DFlash became significantly slower.
From the vLLM logs, we noticed that enabling DFlash may reduce much of the effectiveness of the KV cache.
Could your team share the benchmarking configurations and results when running on H100 or H200, especially under high CCU scenarios?
gpt-oss-120b with dflash
without dflash
This is config we use for serve with vllm :
maxNumSeqs: 256
maxNumBatchedTokens: 32768
maxModelLen: 131072
tensorParallel: 2
enableChunkedPrefill: false
numGPUs: 2
numCPUs: 48
memory: 256Gi
gpuMemoryUtilization: 0.75
overrideGenerationConfig: "{"temperature": 0.6, "top_p": 0.9, "repetition_penalty": 1.025}"
trustRemoteCode: true
visionModel: false
functionCall: true
toolCallParser: "openai"
reasoning: true
reasoningParser: "openai_gptoss"
temperature: 0.5
top_p: 0.9
useVersion0: false
disableFlashinferSampler: false
enableExpertParallel: false
attentionBackend: "TRITON_ATTN"
prefixCaching:
enable: true
speculativeDecoding:
enabled: true
config: '{"method":"dflash","model":"/models/fp16_models/z-lab/gpt-oss-120b-DFlash","num_speculative_tokens":5}'
nccl:
gpuMem: true
shmSize: 32Gi
CCU_LIST="10 20 30 40 50 60 70 80 90 100 200 300 400 500"
for CCU in $CCU_LIST; do
echo "=========================================="
echo "[$(date)] Starting benchmark: CCU=$CCU"
echo "=========================================="
CURRENT_ARTIFACT_DIR="${BASE_ARTIFACT_DIR}/CCU-${CCU}"
mkdir -p "${CURRENT_ARTIFACT_DIR}"
aiperf profile \
--model ${MODEL_NAME} \
--url ${SERVICE_URL} \
--endpoint-type chat \
--streaming \
--concurrency ${CCU} \
--request-count 1000 \
--isl ${ISL} \
--osl ${OSL} \
--goodput 'time_to_first_token:200' \
--tokenizer ${TOKENIZER} \
--artifact-dir ${CURRENT_ARTIFACT_DIR} \
--gpu-telemetry
env :
- name: ISL
value: "5000"
- name: OSL
value: "1000"
and config for aiperf
It would be really great if you could help us reproduce the H200 inference results reported in your paper.