patch/solar_open2.py is incompatible with Upstage vLLM v0.22.0-solar-open2
Hi Nota AI team,
I am trying to serve nota-ai/Solar-Open2-250B-Nota-INT4-GlobalPruned.
The model card says that the model uses a variable number of experts per layer and instructs users to replace vLLM's Solar Open2 implementation with the provided patched file:
cp patch/solar_open2.py \
path/to/vllm/model_executor/models/solar_open2.py
However, the official Solar Open2 model card points users to the following Upstage vLLM fork:
https://github.com/UpstageAI/vllm/tree/v0.22.0-solar-open2
I therefore tested the model with:
Docker image: upstage/vllm-solar-open2:1.0.0
vLLM version: 0.22.0
Base implementation:
vllm/model_executor/models/solar_open2.py
1. Without the Nota patch
The Upstage implementation successfully resolves SolarOpen2ForCausalLM, but model loading fails because the original implementation assumes a uniform number of experts across all MoE layers:
AssertionError: Attempted to load weight (torch.Size([256]))
into parameter (torch.Size([297]))
This appears to be the expected failure caused by the globally pruned model having different expert counts per layer.
2. With the provided Nota patch/solar_open2.py
When I replace the Upstage implementation with the patch included in this repository, model inspection fails immediately:
ImportError: cannot import name 'SharedFusedMoE'
from 'vllm.model_executor.layers.fused_moe'
In vLLM 0.22.0, SharedFusedMoE appears to have been merged into FusedMoE through the shared_experts argument. The current Upstage implementation uses the newer API:
FusedMoE(
shared_experts=self.shared_experts,
...
)
This suggests that the provided Nota patch was written against an older Solar Open2/vLLM implementation and has not yet been rebased onto v0.22.0-solar-open2.
Could you please clarify the following?
- Which exact vLLM repository and commit was used to test this model?
- Which Docker image and tag should be used?
- Is there a
v0.22.0-solar-open2compatible version ofpatch/solar_open2.py?
A tested solar_open2.py, commit hash, or Docker image would be very helpful.
Thank you.
The vLLM version used for the newly released Solar Open2 differs from the version we tested. We are currently working on updating the implementation for compatibility and plan to upload a new patch shortly. We will let you know once it is ready.
@hancheolp Thank you for the quick update!
I confirmed that the updated patch works successfully with the following Docker Compose configuration:
services:
vllm-server:
image: upstage/vllm-solar-open2:1.0.0
container_name: vllm-solar_open2_250b_nota_int4_globalpruned
runtime: nvidia
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
device_ids: ["0", "1", "2", "3"]
environment:
- HF_HUB_OFFLINE=1
ports:
- "8000:8000"
ipc: host
volumes:
- /home/user/vllm/model:/app/model
- /home/user/vllm/patch/solar_open2.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/solar_open2.py:ro
command: >
--model /app/model/solar-open2-250b-nota-int4-globalpruned
--served-model-name upstage/Solar-Open2-250B
--moe-backend triton
--default-chat-template-kwargs '{"think_render_option":"preserved"}'
--reasoning-parser solar_open2
--enable-auto-tool-choice
--tool-call-parser solar_open2
--logits-processors vllm.v1.sample.logits_processor.solar_open2:SolarOpen2TemplateLogitsProcessor
--trust-remote-code
--max-model-len 262144
--max-num-batched-tokens 8192
--max-num-seqs 32
--enable-prefix-caching
--pipeline-parallel-size 4
--tensor-parallel-size 1
--data-parallel-size 1
--enable-expert-parallel
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 20s
retries: 3
The model loaded successfully, and inference is working as expected.
Thank you again for addressing the compatibility issue so quickly. I also appreciate your work in providing such a high-quality quantized model. :)