python -m mlc_chat gen_config /models/Llama-2-7b-chat-hf --quantization q4f16_1 --conv-template llama-2 --output /tmp/tmp1wbp6l29 [2023-12-17 14:49:22] INFO auto_config.py:116: Found model configuration: /models/Llama-2-7b-chat-hf/config.json [2023-12-17 14:49:22] INFO auto_config.py:155: Found model type: llama. Use `--model-type` to override. [2023-12-17 14:49:22] INFO llama_model.py:49: context_window_size not found in config.json. Falling back to max_position_embeddings (4096) [2023-12-17 14:49:22] INFO gen_config.py:114: [generation_config.json] Setting bos_token_id: 1 [2023-12-17 14:49:22] INFO gen_config.py:114: [generation_config.json] Setting eos_token_id: 2 [2023-12-17 14:49:22] INFO gen_config.py:114: [generation_config.json] Setting pad_token_id: 0 [2023-12-17 14:49:22] INFO gen_config.py:114: [generation_config.json] Setting temperature: 0.6 [2023-12-17 14:49:22] INFO gen_config.py:114: [generation_config.json] Setting top_p: 0.9 [2023-12-17 14:49:22] INFO gen_config.py:126: Found tokenizer config: /models/Llama-2-7b-chat-hf/tokenizer.model. Copying to /tmp/tmp1wbp6l29/tokenizer.model [2023-12-17 14:49:22] INFO gen_config.py:126: Found tokenizer config: /models/Llama-2-7b-chat-hf/tokenizer.json. Copying to /tmp/tmp1wbp6l29/tokenizer.json [2023-12-17 14:49:22] INFO gen_config.py:128: Not found tokenizer config: /models/Llama-2-7b-chat-hf/vocab.json [2023-12-17 14:49:22] INFO gen_config.py:128: Not found tokenizer config: /models/Llama-2-7b-chat-hf/merges.txt [2023-12-17 14:49:22] INFO gen_config.py:128: Not found tokenizer config: /models/Llama-2-7b-chat-hf/added_tokens.json [2023-12-17 14:49:22] INFO gen_config.py:126: Found tokenizer config: /models/Llama-2-7b-chat-hf/tokenizer_config.json. Copying to /tmp/tmp1wbp6l29/tokenizer_config.json [2023-12-17 14:49:22] INFO gen_config.py:68: [System default] Setting repetition_penalty: 1.0 [2023-12-17 14:49:22] INFO gen_config.py:68: [System default] Setting mean_gen_len: 128 [2023-12-17 14:49:22] INFO gen_config.py:68: [System default] Setting max_gen_len: 512 [2023-12-17 14:49:22] INFO gen_config.py:68: [System default] Setting shift_fill_factor: 0.3 [2023-12-17 14:49:22] INFO gen_config.py:156: Dumping configuration file to: /tmp/tmp1wbp6l29/mlc-chat-config.json python -m mlc_chat convert_weight /models/Llama-2-7b-chat-hf --quantization q4f16_1 --source-format auto --output /tmp/tmp1wbp6l29 [2023-12-17 14:49:23] INFO auto_config.py:116: Found model configuration: /models/Llama-2-7b-chat-hf/config.json [2023-12-17 14:49:23] INFO auto_device.py:75: Found device: cuda:0 [2023-12-17 14:49:23] INFO auto_device.py:75: Found device: cuda:1 [2023-12-17 14:49:23] INFO auto_device.py:84: Not found device: rocm:0 [2023-12-17 14:49:23] INFO auto_device.py:84: Not found device: metal:0 [2023-12-17 14:49:24] INFO auto_device.py:75: Found device: vulkan:0 [2023-12-17 14:49:24] INFO auto_device.py:75: Found device: vulkan:1 [2023-12-17 14:49:24] INFO auto_device.py:75: Found device: vulkan:2 [2023-12-17 14:49:24] INFO auto_device.py:84: Not found device: opencl:0 [2023-12-17 14:49:24] INFO auto_device.py:33: Using device: cuda:0 [2023-12-17 14:49:24] INFO auto_weight.py:70: Finding weights in: /models/Llama-2-7b-chat-hf [2023-12-17 14:49:24] INFO auto_weight.py:120: Found source weight format: huggingface-torch. Source configuration: /models/Llama-2-7b-chat-hf/pytorch_model.bin.index.json [2023-12-17 14:49:24] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: /models/Llama-2-7b-chat-hf/model.safetensors.index.json [2023-12-17 14:49:24] INFO auto_weight.py:106: Using source weight configuration: /models/Llama-2-7b-chat-hf/pytorch_model.bin.index.json. Use `--source` to override. [2023-12-17 14:49:24] INFO auto_weight.py:110: Using source weight format: huggingface-torch. Use `--source-format` to override. [2023-12-17 14:49:24] INFO auto_config.py:155: Found model type: llama. Use `--model-type` to override. [2023-12-17 14:49:24] INFO llama_model.py:49: context_window_size not found in config.json. Falling back to max_position_embeddings (4096) Weight conversion with arguments: --config /models/Llama-2-7b-chat-hf/config.json --quantization GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7) --model-type llama --device cuda:0 --source /models/Llama-2-7b-chat-hf/pytorch_model.bin.index.json --source-format huggingface-torch --output /tmp/tmp1wbp6l29 0%| | 0/195 [00:00