DFlashMuseGlimmerAssistantModel not supported vllm
#3
by ndurkee - opened
I'm using the vllm:muse-glimmer docker image and getting an error that the tag is not found. I modified the image to update transformers to 5.15.0 and it's still not found
Same here. Upgrade didn't work.
Confirmed, it does not work yet
I misread the comment, it works for me. You need vllm/vllm-openai:muse-glimmer. Spec decoding doesn't work since the draft model class isn't part of the VLLM image.
Same I think its just a mismatch in the name. Its saying DFlashMuseGlimmerAssistantModel is not a supported architecture but MuseGlimmerAssistantModel is in the supported list.
(APIServer pid=3705) Value error, Model architectures ['DFlashMuseGlimmerAssistantModel'] are not supported for now. Supported architectures: dict_keys(['AfmoeForCausalLM', 'ApertusForCausalLM', 'ArceeForCausalLM', 'ArcticForCausalLM', 'AXK1ForCausalLM', 'BailingMoeForCausalLM', 'BailingMoeV2ForCausalLM', 'BailingMoeV2_5ForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'Cohere2MoeForCausalLM', 'CwmForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'DeepseekV3ForCausalLM', 'DeepseekV32ForCausalLM', 'DeepseekV4ForCausalLM', 'Ernie4_5ForCausalLM', 'Ernie4_5_MoeForCausalLM', 'ExaoneForCausalLM', 'Exaone4ForCausalLM', 'ExaoneMoEForCausalLM', 'Fairseq2LlamaForCausalLM', 'FalconForCausalLM', 'FalconMambaForCausalLM', 'FalconH1ForCausalLM', 'FlexOlmoForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'Gemma3ForCausalLM', 'Rnj1ForCausalLM', 'Gemma3nForCausalLM', 'Gemma4ForCausalLM', 'Qwen3NextForCausalLM', 'GlmForCausalLM', 'Glm4ForCausalLM', 'Glm4MoeForCausalLM', 'Glm4MoeLiteForCausalLM', 'GlmMoeDsaForCausalLM', 'GptOssForCausalLM', 'GPT2LMHeadModel', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'GraniteForCausalLM', 'GraniteMoeForCausalLM', 'GraniteMoeHybridForCausalLM', 'GraniteMoeSharedForCausalLM', 'GritLM', 'HrmTextForCausalLM', 'HunYuanMoEV1ForCausalLM', 'HunYuanDenseV1ForCausalLM', 'HYV3ForCausalLM', 'HCXVisionForCausalLM', 'HCXVisionV2ForCausalLM', 'HyperCLOVAXForCausalLM', 'InternLM2ForCausalLM', 'InternLM3ForCausalLM', 'IQuestCoderForCausalLM', 'IQuestLoopCoderForCausalLM', 'Jais2ForCausalLM', 'JambaForCausalLM', 'KimiLinearForCausalLM', 'Lfm2ForCausalLM', 'Lfm2MoeForCausalLM', 'LagunaForCausalLM', 'LlamaForCausalLM', 'Llama4ForCausalLM', 'LLaMAForCausalLM', 'LongcatFlashForCausalLM', 'LongcatFlashNgramForCausalLM', 'MambaForCausalLM', 'Mamba2ForCausalLM', 'MellumForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'MiniMaxM2ForCausalLM', 'MiniMaxM3SparseForCausalLM', 'InklingForCausalLM', 'InklingForConditionalGeneration', 'Ministral3ForCausalLM', 'MistralForCausalLM', 'MistralLarge3ForCausalLM', 'MixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiMoForCausalLM', 'MiMoV2FlashForCausalLM', 'MiMoV2ForCausalLM', 'NemotronForCausalLM', 'NemotronHForCausalLM', 'NemotronHPuzzleForCausalLM', 'Olmo3ForCausalLM', 'MuseGlimmerForCausalLM', 'OlmoHybridForCausalLM', 'OlmoeForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'OuroForCausalLM', 'PanguEmbeddedForCausalLM', 'PanguProMoEV2ForCausalLM', 'PanguUltraMoEForCausalLM', 'Param2MoEForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'PhiMoEForCausalLM', 'Plamo2ForCausalLM', 'Plamo3ForCausalLM', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'Qwen3ForCausalLM', 'Qwen3MoeForCausalLM', 'RWForCausalLM', 'SarvamMoEForCausalLM', 'SarvamMLAForCausalLM', 'SeedOssForCausalLM', 'Step1ForCausalLM', 'Step3TextForCausalLM', 'Step3p5ForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'SolarForCausalLM', 'TeleChat2ForCausalLM', 'TeleChat3ForCausalLM', 'TeleFLMForCausalLM', 'Zamba2ForCausalLM', 'BertModel', 'BertForMaskedLM', 'BertSpladeSparseEmbeddingModel', 'BgeM3EmbeddingModel', 'Gemma2Model', 'Gemma3TextModel', 'GteModel', 'GteNewModel', 'JinaEmbeddingsV5Model', 'LlamaBidirectionalModel', 'LlamaModel', 'MistralModel', 'ModernBertModel', 'NomicBertModel', 'Qwen2Model', 'RobertaForMaskedLM', 'RobertaModel', 'VoyageQwen3BidirectionalEmbedModel', 'XLMRobertaModel', 'CLIPModel', 'ColPaliForRetrieval', 'LlamaNemotronVLModel', 'LlavaNextForConditionalGeneration', 'Phi3VForCausalLM', 'Qwen2VLForConditionalGeneration', 'SiglipModel', 'PrithviGeoSpatialMAE', 'Terratorch', 'HF_ColBERT', 'ColBERTModernBertModel', 'ColBERTJinaRobertaModel', 'ColBERTLfm2Model', 'JinaForRanking', 'ColModernVBertForRetrieval', 'ColQwen3', 'OpsColQwen3Model', 'ColQwen3_5', 'Qwen3VLNemotronEmbedModel', 'InternLM2ForRewardModel', 'Qwen2ForRewardModel', 'Qwen2ForProcessRewardModel', 'BertForTokenClassification', 'ModernBertForTokenClassification', 'OpenAIPrivacyFilterForTokenClassification', 'Qwen3ASRForcedAlignerForTokenClassification', 'RobertaForTokenClassification', 'XLMRobertaForTokenClassification', 'BertForSequenceClassification', 'GPT2ForSequenceClassification', 'GteNewForSequenceClassification', 'JambaForSequenceClassification', 'LlamaBidirectionalForSequenceClassification', 'ModernBertForSequenceClassification', 'RobertaForSequenceClassification', 'XLMRobertaForSequenceClassification', 'JinaVLForRanking', 'LlamaNemotronVLForSequenceClassification', 'AriaForConditionalGeneration', 'AudioFlamingo3ForConditionalGeneration', 'BagelForConditionalGeneration', 'BeeForConditionalGeneration', 'Blip2ForConditionalGeneration', 'ChameleonForConditionalGeneration', 'Cheers', 'CheersForConditionalGeneration', 'Cohere2VisionForConditionalGeneration', 'Cosmos3ForConditionalGeneration', 'Cosmos3EdgeForConditionalGeneration', 'DeepseekVLV2ForCausalLM', 'DeepseekOCRForCausalLM', 'DeepseekOCR2ForCausalLM', 'UnlimitedOCRForCausalLM', 'DotsOCRForCausalLM', 'Eagle2_5_VLForConditionalGeneration', 'Ernie4_5_VLMoeForConditionalGeneration', 'Exaone4_5_ForConditionalGeneration', 'FireRedASR2ForConditionalGeneration', 'FunASRForConditionalGeneration', 'FireRedLIDForConditionalGeneration', 'FunAudioChatForConditionalGeneration', 'Gemma3ForConditionalGeneration', 'Gemma3nForConditionalGeneration', 'DiffusionGemmaForBlockDiffusion', 'Gemma4ForConditionalGeneration', 'Gemma4UnifiedForConditionalGeneration', 'GlmAsrForConditionalGeneration', 'GLM4VForCausalLM', 'Glm4vForConditionalGeneration', 'Glm4vMoeForConditionalGeneration', 'GlmOcrForConditionalGeneration', 'GraniteSpeechForConditionalGeneration', 'GraniteSpeechPlusForConditionalGeneration', 'Granite4VisionForConditionalGeneration', 'H2OVLChatModel', 'HunYuanVLForConditionalGeneration', 'InternVLChatModel', 'InternS1ForConditionalGeneration', 'InternVLForConditionalGeneration', 'InternS1ProForConditionalGeneration', 'InternS2PreviewForConditionalGeneration', 'Idefics3ForConditionalGeneration', 'IsaacForConditionalGeneration', 'KananaVForConditionalGeneration', 'KeyeForConditionalGeneration', 'KeyeVL1_5ForConditionalGeneration', 'KimiVLForConditionalGeneration', 'KimiK25ForConditionalGeneration', 'MoonshotKimiaForCausalLM', 'MossTranscribeDiarizeForConditionalGeneration', 'LightOnOCRForConditionalGeneration', 'Lfm2VlForConditionalGeneration', 'Llama4ForConditionalGeneration', 'Llama_Nemotron_Nano_VL', 'LlavaForConditionalGeneration', 'LlavaNextVideoForConditionalGeneration', 'LlavaOnevisionForConditionalGeneration', 'LlavaOnevision2ForConditionalGeneration', 'MiDashengLMModel', 'MiMoV2OmniForCausalLM', 'MiniMaxM3SparseForConditionalGeneration', 'MiniCPMO', 'MiniCPMV', 'MiniCPMV4_6ForConditionalGeneration', 'Mistral3ForConditionalGeneration', 'MolmoForCausalLM', 'Molmo2ForConditionalGeneration', 'Moondream3ForCausalLM', 'MossAudioModel', 'HfMoondream', 'NemotronH_Nano_VL_V2', 'NemotronH_Nano_Omni_Reasoning_V3', 'NemotronH_Super_Omni_Reasoning_V3', 'NVLM_D', 'MuseGlimmerForConditionalGeneration', 'OpenCUAForConditionalGeneration', 'OpenPanguVLForConditionalGeneration', 'OpenVLAForActionPrediction', 'Ovis', 'Ovis2_5', 'Ovis2_6ForCausalLM', 'Ovis2_6_MoeForCausalLM', 'PaddleOCRVLForConditionalGeneration', 'PaliGemmaForConditionalGeneration', 'Phi4ForCausalLMV', 'Phi4MMForCausalLM', 'PixtralForConditionalGeneration', 'QianfanOCRForConditionalGeneration', 'Qwen2_5_VLForConditionalGeneration', 'Qwen2AudioForConditionalGeneration', 'Qwen2_5OmniModel', 'Qwen2_5OmniForConditionalGeneration', 'Qwen3OmniMoeForConditionalGeneration', 'Qwen3ASRForConditionalGeneration', 'Qwen3ASRRealtimeGeneration', 'Qwen3VLForConditionalGeneration', 'Qwen3VLMoeForConditionalGeneration', 'Qwen3_5ForConditionalGeneration', 'Qwen3_5MoeForConditionalGeneration', 'RForConditionalGeneration', 'SkyworkR1VChatModel', 'SmolVLMForConditionalGeneration', 'StepVLForConditionalGeneration', 'Step3VLForConditionalGeneration', 'Step3p7ForConditionalGeneration', 'UltravoxModel', 'VoxtralForConditionalGeneration', 'VoxtralRealtimeGeneration', 'CohereAsrForConditionalGeneration', 'NemotronParseForConditionalGeneration', 'WhisperForConditionalGeneration', 'ExtractHiddenStatesModel', 'MiMoMTPModel', 'MiMoV2MTPModel', 'MiMoV2OmniMTPModel', 'EagleCohereForCausalLM', 'EagleLlamaForCausalLM', 'EagleLlama4ForCausalLM', 'EagleMiniCPMForCausalLM', 'DFlashDraftModel', 'MuseGlimmerAssistantModel', 'DSparkDraftModel', 'Qwen3DSparkModel', 'DFlashLagunaForCausalLM', 'Gemma4DSparkModel', 'PEagleDraftModel', 'PeagleLlamaForCausalLM', 'Eagle3LlamaForCausalLM', 'Eagle3MiniMaxM2ForCausalLM', 'LlamaForCausalLMEagle3', 'Eagle3Qwen2_5vlForCausalLM', 'Eagle3Qwen3vlForCausalLM', 'Eagle3Qwen3ForCausalLM', 'PeagleQwen3ForCausalLM', 'EagleMistralForCausalLM', 'EagleMistralLarge3ForCausalLM', 'Eagle3DeepseekV2ForCausalLM', 'Eagle3DeepseekV3ForCausalLM', 'EagleDeepSeekMTPModel', 'DeepSeekMTPModel', 'DeepSeekV4MTPModel', 'MiniMaxM3MTP', 'BailingMoeV25MTPModel', 'InklingMTPModel', 'Gemma4MTPModel', 'ErnieMTPModel', 'ExaoneMoeMTP', 'Exaone4_5_MTP', 'NemotronHMTPModel', 'LongCatFlashMTPModel', 'Glm4MoeMTPModel', 'Glm4MoeLiteMTPModel', 'GlmOcrMTPModel', 'MedusaModel', 'OpenPanguMTPModel', 'Qwen3NextMTP', 'Step3p5MTP', 'Qwen3_5MTP', 'Qwen3_5MoeMTP', 'HYV3MTPModel', 'GPTBigCodeForCausalLM', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'SmolLM3ForCausalLM', 'Starcoder2ForCausalLM', 'Emu3ForConditionalGeneration', 'TransformersForCausalLM', 'TransformersMoEForCausalLM', 'TransformersMultiModalForCausalLM', 'TransformersMultiModalMoEForCausalLM', 'TransformersEmbeddingModel', 'TransformersMoEEmbeddingModel', 'TransformersMultiModalEmbeddingModel', 'TransformersForSequenceClassification', 'TransformersMoEForSequenceClassification', 'TransformersMultiModalForSequenceClassification']) [type=value_error, input_value=ArgsKwargs((), {'method':..., _api_process_rank=0)}), input_type=ArgsKwargs]
(APIServer pid=3705) For further information visit https://errors.pydantic.dev/2.13/v/value_error
This setup worked for me if you pull the latest version of the tagged image.
services:
vllm:
image: vllm/vllm-openai:muse-glimmer
container_name: vllm_server
ipc: host
ports:
- "8000:8000"
environment:
- TRANSFORMERS_OFFLINE=1 # Prevents downloading from Hugging Face
volumes:
- /home/$USER/models:/root/models
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all # Or specify GPU IDs
capabilities: [gpu]
command: >
/root/models/meta-models/Muse-Glimmer-30B
--served-model-name muse-glimmer
--enable-auto-tool-choice
--tool-call-parser muse_glimmer
--reasoning-parser muse_glimmer
--no-enable-flashinfer-autotune
--speculative-config '{"method": "dflash", "model": "/root/models/meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 15}'