Repository: mlcommons/inference
Issue: [Whisper] vLLM's LLM model cannot run with the given Dockerfile. Error: "ValueError: CPU backend only supports V1."
Root cause: External Dependency Breakage

Repro steps:
`docker build -t whisper:latest .`
`docker run -it -t whisper`

prepare dataset, model, ...etc

` bash -x ./reference_mlperf_accuracy.sh `
Error message: 

```
INFO 07-14 21:18:08 [cpu.py:68] Using Torch SDPA backend.
Process Instance-2:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/workspace/reference_SUT.py", line 160, in run
    model = LLM(
            ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 274, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 501, in from_engine_args
    return engine_cls.from_vllm_config(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 477, in from_vllm_config
    return cls(
           ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 265, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
    self.collective_rpc("load_model")
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2956, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 194, in load_model
    self.model_runner.load_model()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/cpu_model_runner.py", line 55, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 59, in get_model
    return loader.load_model(vllm_config=vllm_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 38, in load_model
    model = initialize_model(vllm_config=vllm_config,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 64, in initialize_model
    return model_class(vllm_config=vllm_config, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/whisper.py", line 852, in __init__
    self.model = WhisperModel(vllm_config=vllm_config, prefix=prefix)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/whisper.py", line 568, in __init__
    self.encoder = WhisperEncoder(vllm_config=vllm_config,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/whisper.py", line 485, in __init__
    self.start_layer, self.end_layer, self.layers = make_layers(
                                                    ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 640, in make_layers
    maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/whisper.py", line 487, in <lambda>
    lambda prefix: WhisperEncoderLayer(vllm_config=vllm_config,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/whisper.py", line 367, in __init__
    self.self_attn = WhisperAttention(
                     ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/whisper.py", line 216, in __init__
    self.attn = Attention(
                ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 132, in __init__
    attn_backend = get_attn_backend(head_size,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/attention/selector.py", line 154, in get_attn_backend
    return _cached_get_attn_backend(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/attention/selector.py", line 207, in _cached_get_attn_backend
    attention_cls = current_platform.get_attn_backend_cls(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/cpu.py", line 70, in get_attn_backend_cls
    raise ValueError("CPU backend only supports V1.")
ValueError: CPU backend only supports V1.
```



@keithachorn-intel ^^
We found an issue with the reference impl, also we don't seem to have the full reference output generated by Intel, can you help send it?

Hi @nvzhihanj have you got the full reference output from @wu6u3tw ?

@hanyunfan  We need Intel to provide it. FYI it's not a blocking issue but very nice to have

Hi.  Sorry - I am currently out of the office, but I'll take a look deeper at it tonight (before the WG meeting).

The issue was related to changes in the vllm main branch (compiled during Docker build). I've updated the Dockerfile to pin the original version used while developing the reference implementation.  This appears to resolve the issue.

https://github.com/mlcommons/inference/pull/2273

merged #2273 @nvzhihanj could you give it a try.

WG meeting:  Zhihan will run to verify and close if resolved. 

@nvzhihanj any updates?