/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:17:36] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:17:36] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 20:17:36] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 20:17:36] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 20:17:36] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 20:17:36] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 20:17:36] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 20:17:36] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 20:17:36] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 20:17:36] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 20:17:36] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 20:17:36] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 20:17:36] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 20:17:36] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 20:17:36] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 20:17:36] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 20:17:36] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:17:38] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:17:39] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 20:17:40] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 20:17:42] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 20:17:43] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 20:17:45] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 20:17:45] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 20:17:45] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 20:17:45] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 20:17:45] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 20:17:45] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 20:17:45] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 20:17:45] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:25:03] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:25:03] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 20:25:03] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 20:25:03] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 20:25:03] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 20:25:03] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 20:25:03] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 20:25:03] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 20:25:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 20:25:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 20:25:03] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 20:25:03] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 20:25:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 20:25:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 20:25:03] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 20:25:03] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 20:25:03] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:25:05] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:25:06] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 20:25:07] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 20:25:09] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 20:25:10] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 20:25:12] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 20:25:12] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 20:25:12] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 20:25:12] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 20:25:12] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 20:25:12] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 20:25:12] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 20:25:12] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:31:59] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:31:59] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 20:31:59] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 20:31:59] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 20:31:59] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 20:31:59] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 20:31:59] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 20:31:59] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 20:31:59] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 20:31:59] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 20:31:59] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 20:31:59] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 20:31:59] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 20:31:59] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 20:31:59] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 20:31:59] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 20:31:59] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:32:00] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:32:02] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 20:32:03] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 20:32:04] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 20:32:06] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 20:32:07] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 20:32:07] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 20:32:07] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 20:32:07] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 20:32:07] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 20:32:07] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 20:32:07] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 20:32:07] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:40:03] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:40:03] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 20:40:03] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 20:40:03] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 20:40:03] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 20:40:03] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 20:40:03] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 20:40:03] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 20:40:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 20:40:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 20:40:03] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 20:40:03] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 20:40:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 20:40:03] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 20:40:03] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 20:40:03] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 20:40:03] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 20:40:04] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 20:40:06] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 20:40:07] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 20:40:09] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 20:40:10] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 20:40:12] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 20:40:12] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 20:40:12] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 20:40:12] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 20:40:12] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 20:40:12] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 20:40:12] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 20:40:12] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 21:18:18] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 21:18:18] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 21:18:18] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 21:18:18] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 21:18:18] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 21:18:18] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 21:18:18] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 21:18:18] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 21:18:18] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 21:18:19] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 21:18:19] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 21:18:19] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 21:18:19] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 21:18:19] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 21:18:19] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 21:18:19] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 21:18:19] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 21:18:20] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 21:18:21] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 21:18:23] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 21:18:24] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 21:18:26] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 21:18:27] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 21:18:27] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 21:18:27] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 21:18:27] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 21:18:27] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 21:18:27] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 21:18:27] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 21:18:27] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 22:05:04] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 22:05:04] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 22:05:04] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 22:05:04] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 22:05:04] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 22:05:04] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 22:05:04] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 22:05:04] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 22:05:04] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 22:05:04] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 22:05:04] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 22:05:04] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 22:05:04] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 22:05:04] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 22:05:04] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 22:05:04] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 22:05:04] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 22:05:06] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 22:05:07] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 22:05:09] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 22:05:10] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 22:05:12] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 22:05:13] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 22:05:13] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 22:05:13] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 22:05:13] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 22:05:13] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 22:05:13] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 22:05:13] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 22:05:13] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 22:33:39] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 22:33:39] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-28 22:33:39] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-28 22:33:39] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-28 22:33:39] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-28 22:33:39] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-28 22:33:39] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-28 22:33:39] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-28 22:33:39] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-28 22:33:39] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-28 22:33:39] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-28 22:33:39] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-28 22:33:39] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-28 22:33:39] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-28 22:33:39] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-28 22:33:39] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 /opt/conda/envs/py310/lib/python3.10/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: Expected `int` but got `list` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( [2024-05-28 22:33:39] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-28 22:33:40] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-28 22:33:42] INFO auto_device.py:88: Not found device: cuda:0 [2024-05-28 22:33:43] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-28 22:33:44] INFO auto_device.py:88: Not found device: metal:0 [2024-05-28 22:33:46] INFO auto_device.py:88: Not found device: vulkan:0 [2024-05-28 22:33:48] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-28 22:33:48] INFO auto_device.py:33: Not found: No available device detected [2024-05-28 22:33:48] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-28 22:33:48] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-28 22:33:48] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-28 22:33:48] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-28 22:33:48] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-28 22:33:48] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. Traceback (most recent call last): File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 52, in main() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 29, in main cli.main(sys.argv[2:]) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main convert_weight( File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 180, in convert_weight args.display() File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 49, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 42, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type' /opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Phi-3-mini-128k-instruct --quantization q0f16 --conv-template phi-3 --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-30 02:32:48] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-30 02:32:48] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-30 02:32:48] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-30 02:32:48] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 [2024-05-30 02:32:48] INFO config.py:106: Overriding max_batch_size from 1 to 80 [2024-05-30 02:32:48] INFO gen_config.py:142: [generation_config.json] Setting bos_token_id: 1 [2024-05-30 02:32:48] INFO gen_config.py:142: [generation_config.json] Setting eos_token_id: [32000, 32001, 32007] [2024-05-30 02:32:48] INFO gen_config.py:142: [generation_config.json] Setting pad_token_id: 32000 [2024-05-30 02:32:48] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.model. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.model [2024-05-30 02:32:48] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer.json [2024-05-30 02:32:48] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/vocab.json [2024-05-30 02:32:48] INFO gen_config.py:156: Not found tokenizer config: /models/Phi-3-mini-128k-instruct/merges.txt [2024-05-30 02:32:48] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/added_tokens.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/added_tokens.json [2024-05-30 02:32:48] INFO gen_config.py:154: Found tokenizer config: /models/Phi-3-mini-128k-instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/tokenizer_config.json [2024-05-30 02:32:48] INFO gen_config.py:215: Detected tokenizer info: {'token_postproc_method': 'byte_fallback', 'prepend_space_in_encode': True, 'strip_space_in_decode': True} [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting temperature: 1.0 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting presence_penalty: 0.0 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting frequency_penalty: 0.0 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting repetition_penalty: 1.0 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting top_p: 1.0 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting mean_gen_len: 128 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting max_gen_len: 512 [2024-05-30 02:32:48] INFO gen_config.py:31: [System default] Setting shift_fill_factor: 0.3 [2024-05-30 02:32:48] INFO gen_config.py:222: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC/mlc-chat-config.json /opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Phi-3-mini-128k-instruct --quantization q0f16 --source-format auto --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC [2024-05-30 02:32:50] INFO auto_config.py:115: Found model configuration: /models/Phi-3-mini-128k-instruct/config.json [2024-05-30 02:32:51] INFO auto_device.py:79: Found device: cuda:0 [2024-05-30 02:32:53] INFO auto_device.py:88: Not found device: rocm:0 [2024-05-30 02:32:54] INFO auto_device.py:88: Not found device: metal:0 [2024-05-30 02:32:56] INFO auto_device.py:79: Found device: vulkan:0 [2024-05-30 02:32:56] INFO auto_device.py:79: Found device: vulkan:1 [2024-05-30 02:32:56] INFO auto_device.py:79: Found device: vulkan:2 [2024-05-30 02:32:56] INFO auto_device.py:79: Found device: vulkan:3 [2024-05-30 02:32:57] INFO auto_device.py:88: Not found device: opencl:0 [2024-05-30 02:32:57] INFO auto_device.py:35: Using device: cuda:0 [2024-05-30 02:32:57] INFO auto_weight.py:70: Finding weights in: /models/Phi-3-mini-128k-instruct [2024-05-30 02:32:57] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-05-30 02:32:57] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json [2024-05-30 02:32:57] INFO auto_weight.py:106: Using source weight configuration: /models/Phi-3-mini-128k-instruct/model.safetensors.index.json. Use `--source` to override. [2024-05-30 02:32:57] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override. [2024-05-30 02:32:57] INFO auto_config.py:153: Found model type: phi3. Use `--model-type` to override. [2024-05-30 02:32:57] INFO phi3_model.py:53: context_window_size not found in config.json. Falling back to max_position_embeddings (131072) [2024-05-30 02:32:57] INFO phi3_model.py:68: prefill_chunk_size defaults to 2048 Weight conversion with arguments: --config /models/Phi-3-mini-128k-instruct/config.json --quantization NoQuantize(name='q0f16', kind='no-quant', model_dtype='float16') --model-type phi3 --device cuda:0 --source /models/Phi-3-mini-128k-instruct/model.safetensors.index.json --source-format huggingface-safetensor --output /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC Start storing to cache /models/mlc-delivery/hf/mlc-ai/Phi-3-mini-128k-instruct-q0f16-MLC 0%| | 0/195 [00:00