| (venv2) D:\modelopt-windows-scripts\ONNX_PTQ>python D:\opset21_patrice.py --onnx_path="D:\GenAI\models\FP16_Mistral-Nemo-Instruct-2407_ONNX\model.onnx" --output_path="D:\GenAI\models\FP16_Mistral-Nemo-Instruct-2407_ONNX\opset_21\model.onnx" | |
| Printing opset info of given input model... | |
| Domain: | |
| Version: 14 | |
| Domain: com.microsoft | |
| Version: 1 | |
| Printing opset info of output model... | |
| Domain: | |
| Version: 21 | |
| Domain: com.microsoft | |
| Version: 1 | |
| (venv2) D:\modelopt-windows-scripts\ONNX_PTQ>python quantize_script.py --model_name=mistralai/Mistral-Nemo-Instruct-2407 --onnx_path=D:\GenAI\models\FP16_Mistral-Nemo-Instruct-2407_ONNX\opset_21\model.onnx --output_path="D:\GenAI\models\FP16_Mistral-Nemo-Instruct-2407_ONNX\opset_21\default_quant_cuda_ep_calib\model.onnx" --calibration_eps=cuda | |
| --Quantize-Script-- algo=awq_lite, dataset=cnn, calib_size=32, batch_size=1, block_size=128, add-position-ids=True, past-kv=True, rcalib=False, device=cpu, use_zero_point=False | |
| --Quantize-Script-- awqlite_alpha_step=0.1, awqlite_fuse_nodes=False, awqlite_run_per_subgraph=False, awqclip_alpha_step=0.05, awqclip_alpha_min=0.5, awqclip_bsz_col=1024, calibration_eps=['cuda'] | |
| D:\venv2\Lib\site-packages\transformers\models\auto\configuration_auto.py:1002: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead. | |
| warnings.warn( | |
| D:\venv2\Lib\site-packages\transformers\models\auto\tokenization_auto.py:809: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead. | |
| warnings.warn( | |
| --Quantize-Script-- number_of_batched_samples=32, batch-input-ids-list-len=32, batched_attention_mask=32 | |
| --Quantize-Script-- number of batched inputs = 32 | |
| INFO:root: | |
| Quantizing the model.... | |
| INFO:root:Quantization Mode: int4 | |
| INFO:root:Finding quantizable weights and augmenting graph output with input activations | |
| INFO:root:Augmenting took 0.031656503677368164 seconds | |
| INFO:root:Saving the model took 60.20284128189087 seconds | |
| 2024-11-05 22:37:34.5783341 [W:onnxruntime:, transformer_memcpy.cc:74 onnxruntime::MemcpyTransformer::ApplyImpl] 11 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message. | |
| 2024-11-05 22:37:34.5949880 [W:onnxruntime:, session_state.cc:1168 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf. | |
| 2024-11-05 22:37:34.6026375 [W:onnxruntime:, session_state.cc:1170 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments. | |
| Getting activation names maps...: 100%|██████████████████████████████████████████████████████| 280/280 [00:00<?, ?it/s] | |
| Running AWQ scale search per node...: 100%|██████████████████████████████████████████| 280/280 [17:50<00:00, 3.82s/it] | |
| INFO:root:AWQ scale search took 1070.4731740951538 seconds | |
| Quantizing the weights...: 100%|█████████████████████████████████████████████████████| 280/280 [00:15<00:00, 17.78it/s] | |
| INFO:root:Quantizing actual weights took 15.744078636169434 seconds | |
| INFO:root:Inserting DQ nodes and input_pre_quant_scale node using quantized weights and scales ... | |
| INFO:root:Inserting nodes took 0.17318105697631836 seconds | |
| INFO:root:Exporting the quantized graph ... | |
| Loading extension modelopt_round_and_pack_ext... | |
| INFO:root:Exporting took 59.45134162902832 seconds | |
| INFO:root: | |
| Quantization process took 1223.9775414466858 seconds | |
| INFO:root:Saving to D:\GenAI\models\FP16_Mistral-Nemo-Instruct-2407_ONNX\opset_21\default_quant_cuda_ep_calib\model.onnx took 9.476586818695068 seconds | |
| Done | |
| (venv2) D:\modelopt-windows-scripts\ONNX_PTQ>pip list | |
| Package Version | |
| -------------------- ------------------------- | |
| aiohappyeyeballs 2.4.3 | |
| aiohttp 3.10.10 | |
| aiosignal 1.3.1 | |
| annotated-types 0.7.0 | |
| attrs 24.2.0 | |
| certifi 2024.8.30 | |
| charset-normalizer 3.4.0 | |
| cloudpickle 3.1.0 | |
| colorama 0.4.6 | |
| coloredlogs 15.0.1 | |
| cppimport 22.8.2 | |
| cupy-cuda12x 13.3.0 | |
| datasets 3.1.0 | |
| dill 0.3.8 | |
| fastrlock 0.8.2 | |
| filelock 3.16.1 | |
| flatbuffers 24.3.25 | |
| frozenlist 1.5.0 | |
| fsspec 2024.9.0 | |
| huggingface-hub 0.26.2 | |
| humanfriendly 10.0 | |
| idna 3.10 | |
| Jinja2 3.1.4 | |
| Mako 1.3.6 | |
| markdown-it-py 3.0.0 | |
| MarkupSafe 3.0.2 | |
| mdurl 0.1.2 | |
| mpmath 1.3.0 | |
| multidict 6.1.0 | |
| multiprocess 0.70.16 | |
| networkx 3.4.2 | |
| ninja 1.11.1.1 | |
| numpy 1.26.4 | |
| nvidia-modelopt 0.20.1.dev20+g299b7f8a098 | |
| onnx 1.16.0 | |
| onnx-graphsurgeon 0.5.2 | |
| onnxconverter-common 1.14.0 | |
| onnxmltools 1.12.0 | |
| onnxruntime-gpu 1.20.0 | |
| packaging 24.1 | |
| pandas 2.2.3 | |
| pip 24.0 | |
| propcache 0.2.0 | |
| protobuf 3.20.2 | |
| pyarrow 18.0.0 | |
| pybind11 2.13.6 | |
| pydantic 2.9.2 | |
| pydantic_core 2.23.4 | |
| Pygments 2.18.0 | |
| pyreadline3 3.5.4 | |
| python-dateutil 2.9.0.post0 | |
| pytz 2024.2 | |
| PyYAML 6.0.2 | |
| regex 2024.9.11 | |
| requests 2.32.3 | |
| rich 13.9.4 | |
| safetensors 0.4.5 | |
| scipy 1.14.1 | |
| setuptools 65.5.0 | |
| six 1.16.0 | |
| sympy 1.13.3 | |
| tokenizers 0.20.2 | |
| torch 2.4.0 | |
| tqdm 4.66.6 | |
| transformers 4.46.1 | |
| typing_extensions 4.12.2 | |
| tzdata 2024.2 | |
| urllib3 2.2.3 | |
| xxhash 3.5.0 | |
| yarl 1.17.1 | |
| [notice] A new release of pip is available: 24.0 -> 24.3.1 | |
| [notice] To update, run: python.exe -m pip install --upgrade pip | |
| (venv2) D:\modelopt-windows-scripts\ONNX_PTQ> | |