Repository: gpustack/gpustack

Issue #1059: Sometimes model instance logs are incomplete
Created: 2025-01-20T13:13:00Z
Steps to reproduce

Deploy a model and view the instance logs.

Result


Sometimes the logs are incomplete.



Check the watch api response.



GET /v1/model-instances/2/logs?tail=499

2025-01-20T20:23:17+08:00 - gpustack.worker.backends.base - DEBUG - Model file size: 49361088
2025-01-20T20:23:17+08:00 - gpustack.worker.downloaders - INFO - Retriving file lock
Downloading Model to directory: /root/workspace/data/cache/model_scope/nomic-ai/nomic-embed-text-v1.5-GGUF

Processing 1 items:   0%|          | 0.00/1.00 [00:00<?, ?it/s]

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:   0%|          | 0.00/47.1M [00:00<?, ?B/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:   2%|▏         | 1.00M/47.1M [00:00<00:21, 2.28MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  11%|█         | 5.00M/47.1M [00:00<00:03, 11.4MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  17%|█▋        | 8.00M/47.1M [00:00<00:03, 11.7MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  21%|██        | 10.0M/47.1M [00:00<00:02, 13.0MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  25%|██▌       | 12.0M/47.1M [00:01<00:02, 13.6MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  32%|███▏      | 15.0M/47.1M [00:01<00:02, 14.0MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  36%|███▌      | 17.0M/47.1M [00:01<00:02, 12.8MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  40%|████      | 19.0M/47.1M [00:01<00:02, 14.3MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  47%|████▋     | 22.0M/47.1M [00:01<00:01, 17.2MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  53%|█████▎    | 25.0M/47.1M [00:01<00:01, 19.1MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  59%|█████▉    | 28.0M/47.1M [00:01<00:00, 20.7MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  66%|██████▌   | 31.0M/47.1M [00:02<00:00, 22.1MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  72%|███████▏  | 34.0M/47.1M [00:02<00:00, 23.3MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  79%|███████▊  | 37.0M/47.1M [00:02<00:00, 24.2MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  85%|████████▍ | 40.0M/47.1M [00:02<00:00, 24.8MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  91%|█████████▏| 43.0M/47.1M [00:02<00:00, 25.5MB/s]�[A

Downloading [nomic-embed-text-v1.5.Q2_K.gguf]:  98%|█████████▊| 46.0M/47.1M [00:02<00:00, 25.8MB/s]�[A
Downloading [nomic-embed-text-v1.5.Q2_K.gguf]: 100%|██████████| 47.1M/47.1M [00:02<00:00, 17.8MB/s]

Processing 1 items: 100%|██████████| 1.00/1.00 [00:02<00:00, 2.77s/it]
Processing 1 items: 100%|██████████| 1.00/1.00 [00:02<00:00, 2.77s/it]
2025-01-20T20:23:21+08:00 - gpustack.worker.backends.base - DEBUG - __init__ execution time: 4.456829309463501 seconds
2025-01-20T20:23:21+08:00 - gpustack.worker.backends.llama_box - INFO - Starting llama-box server
2025-01-20T20:23:21+08:00 - gpustack.worker.backends.llama_box - DEBUG - Run llama-box: /opt/conda/lib/python3.11/site-packages/gpustack/third_party/bin/llama-box/llama-box with arguments: --host 0.0.0.0 --embeddings --gpu-layers 13 --parallel 1 --ctx-size 8192 --port 40726 --model /root/workspace/data/cache/model_scope/nomic-ai/nomic-embed-text-v1___5-GGUF/nomic-embed-text-v1.5.Q2_K.gguf --alias nomic-embed-text-v1.5 --no-mmap --no-warmup
0.00.002.318 I 
0.00.002.352 I arguments  : /opt/conda/lib/python3.11/site-packages/gpustack/third_party/bin/llama-box/llama-box --host 0.0.0.0 --embeddings --gpu-layers 13 --parallel 1 --ctx-size 8192 --port 40726 --model /root/workspace/data/cache/model_scope/nomic-ai/nomic-embed-text-v1___5-GGUF/nomic-embed-text-v1.5.Q2_K.gguf --alias nomic-embed-text-v1.5 --no-mmap --no-warmup
0.00.002.354 I version    : v0.0.107 (3fffafd)
0.00.002.354 I compiler   : cc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
0.00.002.354 I target     : x86_64-redhat-linux
0.00.002.355 I vendor     : llama.cpp adc5dd92 (4490), stable-diffusion.cpp 1ae97f8 (202)
0.00.023.384 I ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
0.00.023.388 I ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
0.00.023.389 I ggml_cuda_init: found 1 CUDA devices:
0.00.023.678 I   Device 0: Tesla P40, compute capability 6.1, VMM: yes
0.00.025.934 I system_info: n_threads = 28 (n_threads_batch = 28) / 56 | CUDA : ARCHS = 600,610,700,750,800,860,890,900 | F16 = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | 
0.00.025.939 I 
0.00.026.030 I srv                      main: listening, hostname = 0.0.0.0, port = 40726, n_threads = 3 + 2
0.00.027.212 I srv                      main: loading model
0.00.027.217 I srv                load_model: loading model '/root/workspace/data/cache/model_scope/nomic-ai/nomic-embed-text-v1___5-GGUF/nomic-embed-text-v1.5.Q2_K.gguf'
0.00.224.070 I llama_model_load_from_file: using device CUDA0 (Tesla P40) - 21751 MiB free
0.00.234.106 I llama_model_loader: loaded meta data with 23 key-value pairs and 112 tensors from /root/workspace/data/cache/model_scope/nomic-ai/nomic-embed-text-v1___5-GGUF/nomic-embed-text-v1.5.Q2_K.gguf (version GGUF V3 (latest))
0.00.234.115 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.00.234.118 I llama_model_loader: - kv   0:                       general.architecture str              = nomic-bert
0.00.234.120 I llama_model_loader: - kv   1:                               general.name str              = nomic-embed-text-v1.5
0.00.234.122 I llama_model_loader: - kv   2:                     nomic-bert.block_count u32              = 12
0.00.234.123 I llama_model_loader: - kv   3:                  nomic-bert.context_length u32              = 2048
0.00.234.124 I llama_model_loader: - kv   4:                nomic-bert.embedding_length u32              = 768
0.00.234.125 I llama_model_loader: - kv   5:             nomic-bert.feed_forward_length u32              = 3072
0.00.234.126 I llama_model_loader: - kv   6:            nomic-bert.attention.head_count u32              = 12
0.00.234.133 I llama_model_loader: - kv   7:    nomic-bert.attention.layer_norm_epsilon f32              = 0.000000
0.00.234.134 I llama_model_loader: - kv   8:                          general.file_type u32              = 10
0.00.234.135 I llama_model_loader: - kv   9:                nomic-bert.attention.causal bool             = false
0.00.234.136 I llama_model_loader: - kv  10:                    nomic-bert.pooling_type u32              = 1
0.00.234.139 I llama_model_loader: - kv  11:                  nomic-bert.rope.freq_base f32              = 1000.000000
0.00.234.140 I llama_model_loader: - kv  12:            tokenizer.ggml.token_type_count u32              = 2
0.00.234.141 I llama_model_loader: - kv  13:                tokenizer.ggml.bos_token_id u32              = 101
0.00.234.141 I llama_model_loader: - kv  14:                tokenizer.ggml.eos_token_id u32              = 102
0.00.234.142 I llama_model_loader: - kv  15:                       tokenizer.ggml.model str              = bert
0.00.245.105 I llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,30522]   = ["[PAD]", "[unused0]", "[unused1]", "...
0.00.257.845 I llama_model_loader: - kv  17:                      tokenizer.ggml.scores arr[f32,30522]   = [-1000.000000, -1000.000000, -1000.00...
0.00.262.425 I llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,30522]   = [3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
0.00.262.431 I llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 100
0.00.262.432 I llama_model_loader: - kv  20:          tokenizer.ggml.seperator_token_id u32              = 102
0.00.262.434 I llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 0
0.00.262.435 I llama_model_loader: - kv  22:               general.quantization_version u32              = 2
0.00.262.435 I llama_model_loader: - type  f32:   51 tensors
0.00.262.436 I llama_model_loader: - type q2_K:   37 tensors
0.00.262.437 I llama_model_loader: - type q3_K:   24 tensors
0.00.262.438 I print_info: file format = GGUF V3 (latest)
0.00.262.439 I print_info: file type   = Q2_K - Medium
0.00.262.442 I print_info: file size   = 46.35 MiB (2.84 BPW) 
0.00.283.566 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.283.600 I load: special tokens cache size = 5
0.00.298.108 I load: token to piece cache size = 0.2032 MB
0.00.298.124 I print_info: arch             = nomic-bert
0.00.298.125 I print_info: vocab_only       = 0
0.00.298.126 I print_info: n_ctx_train      = 2048
0.00.298.126 I print_info: n_embd           = 768
0.00.298.127 I print_info: n_layer          = 12
0.00.298.134 I print_info: n_head           = 12
0.00.298.136 I print_info: n_head_kv        = 12
0.00.298.137 I print_info: n_rot            = 64
0.00.298.137 I print_info: n_swa            = 0
0.00.298.137 I print_info: n_embd_head_k    = 64
0.00.298.138 I print_info: n_embd_head_v    = 64
0.00.298.139 I print_info: n_gqa            = 1
0.00.298.141 I print_info: n_embd_k_gqa     = 768
0.00.298.142 I print_info: n_embd_v_gqa     = 768
0.00.298.144 I print_info: f_norm_eps       = 1.0e-12
0.00.298.144 I print_info: f_norm_rms_eps   = 0.0e+00
0.00.298.145 I print_info: f_clamp_kqv      = 0.0e+00
0.00.298.145 I print_info: f_max_alibi_bias = 0.0e+00
0.00.298.146 I print_info: f_logit_scale    = 0.0e+00
0.00.298.147 I print_info: n_ff             = 3072
0.00.298.148 I print_info: n_expert         = 0
0.00.298.148 I print_info: n_expert_used    = 0
0.00.298.149 I print_info: causal attn      = 0
0.00.298.149 I print_info: pooling type     = 1
0.00.298.149 I print_info: rope type        = 2
0.00.298.150 I print_info: rope scaling     = linear
0.00.298.151 I print_info: freq_base_train  = 1000.0
0.00.298.152 I print_info: freq_scale_train = 1
0.00.298.152 I print_info: n_ctx_orig_yarn  = 2048
0.00.298.153 I print_info: rope_finetuned   = unknown
0.00.298.153 I print_info: ssm_d_conv       = 0
0.00.298.154 I print_info: ssm_d_inner      = 0
0.00.298.154 I print_info: ssm_d_state      = 0
0.00.298.155 I print_info: ssm_dt_rank      = 0
0.00.298.155 I print_info: ssm_dt_b_c_rms   = 0
0.00.298.156 I print_info: model type       = 137M
0.00.298.157 I print_info: model params     = 136.73 M
0.00.298.158 I print_info: general.name     = nomic-embed-text-v1.5
0.00.298.159 I print_info: vocab type       = WPM
0.00.298.159 I print_info: n_vocab          = 30522
0.00.298.160 I print_info: n_merges         = 0
0.00.298.160 I print_info: BOS token        = 101 '[CLS]'
0.00.298.160 I print_info: EOS token        = 102 '[SEP]'
0.00.298.161 I print_info: UNK token        = 100 '[UNK]'
0.00.298.161 I print_info: SEP token        = 102 '[SEP]'
0.00.298.162 I print_info: PAD token        = 0 '[PAD]'
0.00.298.162 I print_info: MASK token       = 103 '[MASK]'
0.00.298.163 I print_info: LF token         = 0 '[PAD]'
0.00.298.163 I print_info: EOG token        = 102 '[SEP]'
0.00.298.164 I print_info: max token length = 21
0.00.300.593 I load_tensors: offloading 12 repeating layers to GPU
0.00.300.599 I load_tensors: offloading output layer to GPU
0.00.300.600 I load_tensors: offloaded 13/13 layers to GPU
0.00.300.603 I load_tensors:    CUDA_Host model buffer size =     0.01 MiB
0.00.300.604 I load_tensors:          CPU model buffer size =     7.34 MiB
0.00.300.605 I load_tensors:        CUDA0 model buffer size =    39.01 MiB
0.00.319.925 W llama_init_from_model: adjust n_ctx of the none causal attention model to the minimum value between given n_ctx and n_ctx_train
0.00.319.933 W llama_init_from_model: align n_batch of the none causal attention model to n_ctx
0.00.319.933 W llama_init_from_model: align n_ubatch of the none causal attention model to n_ctx
0.00.319.935 I llama_init_from_model: n_seq_max     = 1
0.00.319.935 I llama_init_from_model: n_ctx         = 2048
0.00.319.936 I llama_init_from_model: n_ctx_per_seq = 2048
0.00.319.936 I llama_init_from_model: n_batch       = 2048
0.00.319.936 I llama_init_from_model: n_ubatch      = 2048
0.00.319.947 I llama_init_from_model: flash_attn    = 0
0.00.319.953 I llama_init_from_model: freq_base     = 1000.0
0.00.319.955 I llama_init_from_model: freq_scale    = 1
0.00.319.964 I llama_kv_cache_init: kv_size = 2048, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 12, can_shift = 1
0.00.320.603 I llama_kv_cache_init:      CUDA0 KV buffer size =    72.00 MiB
0.00.320.611 I llama_init_from_model: KV self size  =   72.00 MiB, K (f16):   36.00 MiB, V (f16):   36.00 MiB
0.00.320.615 I llama_init_from_model:  CUDA_Host  output buffer size =     0.00 MiB
0.00.342.901 I llama_init_from_model:      CUDA0 compute buffer size =   266.00 MiB
0.00.342.911 I llama_init_from_model:  CUDA_Host compute buffer size =    38.02 MiB
0.00.342.912 I llama_init_from_model: graph nodes  = 453
0.00.342.913 I llama_init_from_model: graph splits = 4 (with bs=2048), 2 (with bs=1)
0.00.342.917 I common_init_from_params: setting dry_penalty_last_n to ctx_size = 2048
0.00.342.920 I srv                load_model: prompt caching enabled
0.00.351.868 I srv                load_model: chat template, built_in: true, alias: chatml, tool call: unsupported, example:

system
You are a helpful assistant.


user
Hello.


assistant
Hi there.


user
What's the weather like in Paris today?


assistant

0.00.351.877 I srv                      main: initializing server
0.00.351.879 I srv                      init: initializing slots, n_slots = 1
0.00.352.082 I srv                      main: starting server
0.03.016.875 I srv        log_server_request: rid 13257738882699 | GET /v1/models 127.0.0.1:47954
0.03.017.012 I srv       log_server_response: rid 13257738882699 | GET /v1/models 127.0.0.1:47954 | status 200 | cost 0.00s
0.03.061.134 I srv        log_server_request: rid 13257738926958 | GET /v1/models 10.18.171.124:57352
0.03.061.264 I srv       log_server_response: rid 13257738926958 | GET /v1/models 10.18.171.124:57352 | status 200 | cost 0.00s
41.26.000.223 I srv        log_server_request: rid 13260221866048 | POST /v1/embeddings 10.18.171.124:57354
41.26.000.281 I srv oaicompat_embeddings_requ: rid 13260221866048 | {"encoding_format":"float","input":"[...]","model":"nomic-embed-text-v1.5"}
41.26.171.259 I srv oaicompat_embeddings_resp: rid 13260221866048 | prompt_tokens: 6
41.26.171.923 I srv       log_server_response: rid 13260221866048 | POST /v1/embeddings 10.18.171.124:57354 | status 200 | cost 0.17s

Expected behavior

Environment

GPUStack version: main 6e4ecf6
OS: Ubuntu 22.04

Additional context
Add any other context about the problem here.

Duplicate issue #1336: View log didn't show the complete instance serving log
Created: 2025-02-25T10:49:40Z
Describe the bug
Deploy ModelScope/ZhipuAI/glm-4-9b-chat-1m on a machine with single 4090, add
--max-model-len=8192
--trust-remote-code
Steps to reproduce

Try to chat with the model in playground.
View log to check the model replica log. It didn't show any access records.



Check the log file on the worker directly; it contains more information not shown in webUI.

root@cuuo5m4p420c73ddmpfg-2jjsf:~/workspace# cat /var/lib/gpustack/log/serve/1.log 
2025-02-25T18:26:19+08:00 - gpustack.worker.downloaders - INFO - Retriving file lock
Downloading Model to directory: /var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m
Downloading Model to directory: /root/.cache/modelscope/hub/ZhipuAI/glm-4-9b-chat-1m
2025-02-25T18:26:22+08:00 - gpustack.worker.backends.vllm - INFO - Starting vllm server
INFO 02-25 18:26:27 __init__.py:190] Automatically detected platform cuda.
INFO 02-25 18:26:28 api_server.py:840] vLLM API server version 0.7.2
INFO 02-25 18:26:28 api_server.py:841] args: Namespace(subparser='serve', model_tag='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', config='', host='0.0.0.0', port=40182, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=8192, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['glm-4-9b-chat-1m'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x7f38fae76c00>)
INFO 02-25 18:26:29 api_server.py:206] Started engine process with PID 6360
INFO 02-25 18:26:33 __init__.py:190] Automatically detected platform cuda.
INFO 02-25 18:26:36 config.py:542] This model supports multiple tasks: {'embed', 'reward', 'generate', 'score', 'classify'}. Defaulting to 'generate'.
WARNING 02-25 18:26:36 tokenizer.py:221] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 02-25 18:26:42 config.py:542] This model supports multiple tasks: {'classify', 'reward', 'score', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 02-25 18:26:42 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.2) with config: model='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', speculative_config=None, tokenizer='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=glm-4-9b-chat-1m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
WARNING 02-25 18:26:42 tokenizer.py:221] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 02-25 18:26:43 cuda.py:230] Using Flash Attention backend.
INFO 02-25 18:26:46 model_runner.py:1110] Starting to load model /var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m...
Loading safetensors checkpoint shards:   0% Completed | 0/10 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  10% Completed | 1/10 [00:00<00:03,  2.96it/s]
Loading safetensors checkpoint shards:  20% Completed | 2/10 [00:00<00:02,  3.07it/s]
Loading safetensors checkpoint shards:  30% Completed | 3/10 [00:01<00:02,  2.70it/s]
Loading safetensors checkpoint shards:  40% Completed | 4/10 [00:01<00:02,  2.74it/s]
Loading safetensors checkpoint shards:  50% Completed | 5/10 [00:01<00:01,  2.92it/s]
Loading safetensors checkpoint shards:  60% Completed | 6/10 [00:02<00:01,  2.77it/s]
Loading safetensors checkpoint shards:  70% Completed | 7/10 [00:02<00:01,  2.77it/s]
Loading safetensors checkpoint shards:  80% Completed | 8/10 [00:02<00:00,  2.91it/s]
Loading safetensors checkpoint shards:  90% Completed | 9/10 [00:03<00:00,  2.98it/s]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:03<00:00,  2.81it/s]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:03<00:00,  2.84it/s]

INFO 02-25 18:26:50 model_runner.py:1115] Loading model weights took 17.8282 GB
INFO 02-25 18:26:51 worker.py:267] Memory profiling takes 1.38 seconds
INFO 02-25 18:26:51 worker.py:267] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.90) = 21.28GiB
INFO 02-25 18:26:51 worker.py:267] model weights take 17.83GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 1.44GiB; the rest of the memory reserved for KV Cache is 1.93GiB.
INFO 02-25 18:26:52 executor_base.py:110] # CUDA blocks: 1581, # CPU blocks: 3276
INFO 02-25 18:26:52 executor_base.py:115] Maximum concurrency for 8192 tokens per request: 3.09x
INFO 02-25 18:26:54 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:17<00:00,  1.97it/s]
INFO 02-25 18:27:12 model_runner.py:1562] Graph capturing finished in 18 secs, took 0.29 GiB
INFO 02-25 18:27:12 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 22.05 seconds
INFO 02-25 18:27:12 api_server.py:756] Using supplied chat template:
INFO 02-25 18:27:12 api_server.py:756] None
INFO 02-25 18:27:12 launcher.py:21] Available routes are:
INFO 02-25 18:27:12 launcher.py:29] Route: /openapi.json, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /docs, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /redoc, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /health, Methods: GET
INFO 02-25 18:27:12 launcher.py:29] Route: /ping, Methods: GET, POST
INFO 02-25 18:27:12 launcher.py:29] Route: /tokenize, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /detokenize, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/models, Methods: GET
INFO 02-25 18:27:12 launcher.py:29] Route: /version, Methods: GET
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/chat/completions, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/completions, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/embeddings, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /pooling, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /score, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/score, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /rerank, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/rerank, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v2/rerank, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /invocations, Methods: POST
INFO:     Started server process [6225]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:40182 (Press CTRL+C to quit)
INFO:     127.0.0.1:34852 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     10.18.239.44:45432 - "GET /v1/models HTTP/1.1" 200 OK
INFO 02-25 18:32:18 chat_utils.py:332] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 02-25 18:32:18 logger.py:39] Received request chatcmpl-62d606d30b9a4068b8c7d0cfdf821b2e: prompt: '[gMASK]<sop>
\ntest
', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4096, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO:     10.18.239.44:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 02-25 18:32:18 engine.py:275] Added request chatcmpl-62d606d30b9a4068b8c7d0cfdf821b2e.
INFO 02-25 18:32:18 metrics.py:455] Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 02-25 18:32:28 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 02-25 18:32:38 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
root@cuuo5m4p420c73ddmpfg-2jjsf:~/workspace# cat /var/lib/gpustack/log/serve/1.log 
2025-02-25T18:26:19+08:00 - gpustack.worker.downloaders - INFO - Retriving file lock
Downloading Model to directory: /var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m
Downloading Model to directory: /root/.cache/modelscope/hub/ZhipuAI/glm-4-9b-chat-1m
2025-02-25T18:26:22+08:00 - gpustack.worker.backends.vllm - INFO - Starting vllm server
INFO 02-25 18:26:27 __init__.py:190] Automatically detected platform cuda.
INFO 02-25 18:26:28 api_server.py:840] vLLM API server version 0.7.2
INFO 02-25 18:26:28 api_server.py:841] args: Namespace(subparser='serve', model_tag='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', config='', host='0.0.0.0', port=40182, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=8192, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['glm-4-9b-chat-1m'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x7f38fae76c00>)
INFO 02-25 18:26:29 api_server.py:206] Started engine process with PID 6360
INFO 02-25 18:26:33 __init__.py:190] Automatically detected platform cuda.
INFO 02-25 18:26:36 config.py:542] This model supports multiple tasks: {'embed', 'reward', 'generate', 'score', 'classify'}. Defaulting to 'generate'.
WARNING 02-25 18:26:36 tokenizer.py:221] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 02-25 18:26:42 config.py:542] This model supports multiple tasks: {'classify', 'reward', 'score', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 02-25 18:26:42 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.2) with config: model='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', speculative_config=None, tokenizer='/var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=glm-4-9b-chat-1m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
WARNING 02-25 18:26:42 tokenizer.py:221] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 02-25 18:26:43 cuda.py:230] Using Flash Attention backend.
INFO 02-25 18:26:46 model_runner.py:1110] Starting to load model /var/lib/gpustack/cache/model_scope/ZhipuAI/glm-4-9b-chat-1m...
Loading safetensors checkpoint shards:   0% Completed | 0/10 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  10% Completed | 1/10 [00:00<00:03,  2.96it/s]
Loading safetensors checkpoint shards:  20% Completed | 2/10 [00:00<00:02,  3.07it/s]
Loading safetensors checkpoint shards:  30% Completed | 3/10 [00:01<00:02,  2.70it/s]
Loading safetensors checkpoint shards:  40% Completed | 4/10 [00:01<00:02,  2.74it/s]
Loading safetensors checkpoint shards:  50% Completed | 5/10 [00:01<00:01,  2.92it/s]
Loading safetensors checkpoint shards:  60% Completed | 6/10 [00:02<00:01,  2.77it/s]
Loading safetensors checkpoint shards:  70% Completed | 7/10 [00:02<00:01,  2.77it/s]
Loading safetensors checkpoint shards:  80% Completed | 8/10 [00:02<00:00,  2.91it/s]
Loading safetensors checkpoint shards:  90% Completed | 9/10 [00:03<00:00,  2.98it/s]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:03<00:00,  2.81it/s]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:03<00:00,  2.84it/s]

INFO 02-25 18:26:50 model_runner.py:1115] Loading model weights took 17.8282 GB
INFO 02-25 18:26:51 worker.py:267] Memory profiling takes 1.38 seconds
INFO 02-25 18:26:51 worker.py:267] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.90) = 21.28GiB
INFO 02-25 18:26:51 worker.py:267] model weights take 17.83GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 1.44GiB; the rest of the memory reserved for KV Cache is 1.93GiB.
INFO 02-25 18:26:52 executor_base.py:110] # CUDA blocks: 1581, # CPU blocks: 3276
INFO 02-25 18:26:52 executor_base.py:115] Maximum concurrency for 8192 tokens per request: 3.09x
INFO 02-25 18:26:54 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:17<00:00,  1.97it/s]
INFO 02-25 18:27:12 model_runner.py:1562] Graph capturing finished in 18 secs, took 0.29 GiB
INFO 02-25 18:27:12 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 22.05 seconds
INFO 02-25 18:27:12 api_server.py:756] Using supplied chat template:
INFO 02-25 18:27:12 api_server.py:756] None
INFO 02-25 18:27:12 launcher.py:21] Available routes are:
INFO 02-25 18:27:12 launcher.py:29] Route: /openapi.json, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /docs, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /redoc, Methods: GET, HEAD
INFO 02-25 18:27:12 launcher.py:29] Route: /health, Methods: GET
INFO 02-25 18:27:12 launcher.py:29] Route: /ping, Methods: GET, POST
INFO 02-25 18:27:12 launcher.py:29] Route: /tokenize, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /detokenize, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/models, Methods: GET
INFO 02-25 18:27:12 launcher.py:29] Route: /version, Methods: GET
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/chat/completions, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/completions, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/embeddings, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /pooling, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /score, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/score, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /rerank, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v1/rerank, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /v2/rerank, Methods: POST
INFO 02-25 18:27:12 launcher.py:29] Route: /invocations, Methods: POST
INFO:     Started server process [6225]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:40182 (Press CTRL+C to quit)
INFO:     127.0.0.1:34852 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     10.18.239.44:45432 - "GET /v1/models HTTP/1.1" 200 OK
INFO 02-25 18:32:18 chat_utils.py:332] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 02-25 18:32:18 logger.py:39] Received request chatcmpl-62d606d30b9a4068b8c7d0cfdf821b2e: prompt: '[gMASK]<sop>
\ntest
', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4096, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO:     10.18.239.44:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 02-25 18:32:18 engine.py:275] Added request chatcmpl-62d606d30b9a4068b8c7d0cfdf821b2e.
INFO 02-25 18:32:18 metrics.py:455] Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 02-25 18:32:28 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 02-25 18:32:38 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.


Confirmed the same problem in Edge, Chrome and Safari in macOS.

Result
Not able to view the complete log
Environment

GPUStack version: v0.5.1
OS: Ubuntu 22.04 image
GPU: v0.5.1

Closing comment (2025-02-25T11:28:33Z): Similar to #1059