Hugging Face's logo

yandex
/
AliceAI-T5-35B-A0.6B

Transformers
Safetensors
PyTorch
aliceai_t5_moe
text2text-generation
encoder-decoder
mixture-of-experts
ul2
custom_code
Model card Files
xet
 Community
5

Error run vLLM, llama-cpp

#5
by TimohaS - opened
Discussion
TimohaS

Доброе время суток, напишите пожалуйста на какой версии vLLM и с какими параметрами запускали модель. На версии 0.23.0 и 0.29.0 возникает ошибка:

(EngineCore pid=20) WARNING 09-17 08:45:52 [utils.py:217] TransformersForCausalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
(EngineCore pid=20) INFO 09-17 08:45:52 [base.py:109] Using Transformers modeling backend.
(EngineCore pid=20) INFO 09-17 08:45:52 [base.py:509] Fused: norm (AliceAIT5RMSNorm) -> RMSNorm (CustomOp)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] EngineCore failed to start.
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] Traceback (most recent call last):
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1336, in run_engine_core
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] return func(*args, **kwargs)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) Process EngineCore:
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1093, in init
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] super().init(
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 134, in init
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] self.model_executor = executor_class(vllm_config)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] return func(*args, **kwargs)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 110, in init
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] self._init_executor()
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 74, in _init_executor
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] self.driver_worker.load_model()
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 495, in load_model
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu/model_runner.py", line 384, in load_model
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] self.model = model_loader.load_model(
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] return func(*args, **kwargs)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] model = initialize_model(
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] return func(*args, **kwargs)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 59, in initialize_model
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] model = model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers/causal.py", line 39, in init
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] super(VllmModelForTextGeneration, self).init(
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers/base.py", line 168, in init
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] self.recursive_replace()
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers/base.py", line 520, in recursive_replace
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] _recursive_replace(self.model, prefix="model")
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers/base.py", line 514, in _recursive_replace
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] _recursive_replace(child_module, prefix=qual_name)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers/base.py", line 514, in _recursive_replace
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] _recursive_replace(child_module, prefix=qual_name)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers/base.py", line 475, in _recursive_replace
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] and len(module) == self.text_config.num_hidden_layers
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 484, in getattribute
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] return super().getattribute(key)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py", line 312, in getattribute
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] return super().getattribute(key)
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=20) ERROR 09-17 08:45:53 [core.py:1374] AttributeError: 'AliceAIT5MoEConfig' object has no attribute 'num_hidden_layers'
TimohaS changed discussion title from Error vLLM to Error run vLLM, llama-cpp
TimohaS

Так же при запуске через llama cpp gguf версии моделей возникает ошибка unknown model architecture: 'aliceai_t5_moe'
Czarj

Так же при запуске через llama cpp gguf версии моделей возникает ошибка unknown model architecture: 'aliceai_t5_moe'

В отличие от Python и библиотеки Hugging Face transformers, которая может запускать произвольный код из файла modeling.py модели на лету (trust_remote_code=True), llama.cpp написан на C++ и использует жестко скомпилированный граф вычислений.

Любой тензор, слой attention или роутер MoE должен быть заранее описан в C++ коде движка. Если при конвертации в .gguf в поле general.architecture записывается уникальное имя вроде aliceai_t5_moe или k2-horizon, движок бракует файл на старте, так как не знает, как физически распределять слои по памяти и ядрам GPU.
Короче тут только ждать, пока в lamma добавят поддержку...
shadowklesh

когда ждать vllm поддержку ?

Sign up or log in to comment