mradermacher

/

model_requests

like 59

I will probably not have more luck than you, but I can try and see what happens :)

No issue at all here, can't say why.

Anyway, imatrix wuants will be delayed as there is this 1.7T model in the queue which is delaying everything by about 2 days.

I will probably not have more luck than you, but I can try and see what happens :)

I guess we'll see. Sure it could be something with the model itself but It's probably just something in llama.cpp or my python environment that I don't know how to fix. Maybe it's because I still use python v3.1.0 or something else I'm missing. I rarely do quants anymore since I'm focusing more on dataset optimization (synth. data, formatting of the datasets etc.) to train my own lora adapters. So if the normal "python convert_hf_to_gguf.py C:/model --outtype C:/model/modelf16.gguf" doesn't work, I'm pretty much at the end of my llama.cpp knowledge. Back when llama 3 was introduced I needed several hours to finally figure out that I had to use convert_hf_to_gguf_update.py as well as stop using convert.py. So I would say its 99% probability that it's just my lack of knowledge on how to solve issues with llama.cpp if the standard procedure is not working. If anyone can do it though, it's probably you, considering that you have probably the most experience with gguf quantization, at least you provide probably the most gguf quants of the current models.

No issue at all here, can't say why.

Anyway, imatrix wuants will be delayed as there is this 1.7T model in the queue which is delaying everything by about 2 days.

Did you use the normal "python convert_hf_to_gguf.py C:/Model" command or do you have to modify them for qwen2.5 or any other current models?

Thanks so much for your work! It's really amazing that there's someone doing quants that frequently for the community.

At this time (and for some months now), running convert_hf_to_gguf.yp, essentially without any configurability, is the correct and only thing you need to do to convert a model - there are no configurable knobs anymore, sou can't run the wrong converter or with wrong arguments.

The only thing I can imagine is that you didn't install the python requirements (something like pip3 install -r llama.cpp/requirements,txt) or something to that extent.

Back when llama 3 was introduced

It was a disaster. Documentation telling you you just use the wrong converter for ages, the new converter loading everything into ram etc.. Yeah...

At this time (and for some months now), running convert_hf_to_gguf.yp, essentially without any configurability, is the correct and only thing you need to do to convert a model - there are no configurable knobs anymore, sou can't run the wrong converter or with wrong arguments.

The only thing I can imagine is that you didn't install the python requirements (something like pip3 install -r llama.cpp/requirements,txt) or something to that extent.

I deleted all llama.cpp data out of my path folder and used w64devkit for a new llama.cpp (git clone llama.cpp, after loading cd llama.cpp and then just "make" which should install all requirements). I cloned the entire model's repo to my drive and used at first "python convert_hf_to_gguf.py C:/downloadrepofolder" and after that did not work I tried several other things like changing the name of the folder and the drive as well or using variations of the initial command but it always led to the same error that it's an unknown model wrapper. I haven't really found anything out by copy pasting the error into google etc. and I also didn't know where I could try to find a solution since I've never found good resources to solve issues and learn llama.cpp. I mean something like a step by step tutorial for all the features of llama.cpp or a beginner friendly community because the issue section on GitHub is definitely everything but beginner friendly.

I guess I will try making a new conda env or virtual env to rule out that it's a dependency issue but I think normally python would say it's a missing requirement or something that can't be found etc but it's always a good to have a seperate clean env to pinpoint the issue.

Mradermacher was successfully able to quant this model under https://huggingface.co/mradermacher/Eidolon-v2.1-14B-GGUF

The issue must be related to your Python environment. I recommend to completely reinstall python so everything is clean. After that run the following script to get a clean environment. If you are on Windows, the script will probably require some slight modifications.

#!/bin/bash
apt install libopenblas-dev
git clone --recursive https://github.com/ggerganov/llama.cpp.git
cd llama.cpp/
make GGML_CUDA=1 GGML_RPC=1 -j
cd ..
python3 -m venv venv
venv/bin/pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
venv/bin/pip3 install sentencepiece
venv/bin/pip3 install pyyaml
venv/bin/pip3 install safetensors
venv/bin/pip3 install transformers

@WesProfromyourdescrition,you did not install the python requirements. wrapper not found sound slike the error message you would get form python when thisa happens. also, nico's instructions are also wrong, because he also didn't install the requirements :) building (and install) llama.cpp does not install the requirements (from requirements.txt).

@WesProfromyourdescrition,you did not install the python requirements. wrapper not found sound slike the error message you would get form python when thisa happens. also, nico's instructions are also wrong, because he also didn't install the requirements :) building (and install) llama.cpp does not install the requirements (from requirements.txt).

Ok I'll try to do it again clean but I only followed the exact instruction in the github repo fpr building llama.cpp locally for win 10. I followed those exact steps and it worked on another quant 2 days before so I'm not yet sure it's a missing requirement. The steps I followed:

git clone https://github.com/ggerganov/llama.cpp

then

On Windows (x86/x64 only, arm64 requires cmake):

Download the latest fortran version of w64devkit.
Extract w64devkit on your pc.
Run w64devkit.exe.
Use the cd command to reach the llama.cpp folder.
From here you can run:
make

and it took like 20 min or so to finish that's why I thought I got the requirements part right. That doesn't mean a clean install can't be the solution though even if it's not one of the requirements for the llama.cpp build but something else from all the other installs I did on the same environment that may interfere somehow even though I tried using conda envs for everything except llama.cpp and a few essential libraries.

@wespro It's just a strong hint, you don't have to try out what I tell you, but there is no point asking me for hints on what could have gone wrong if you don't do what I say you are probably missing, and no point continuing guessing around until you made sure you have the dependencies installed. Not sure how much clearer I can say that :)

I mean, it's pointless to say "I am not sure" - we know that. Why don't you make sure?

@wespro It's just a strong hint, you don't have to try out what I tell you, but there is no point asking me for hints on what could have gone wrong if you don't do what I say you are probably missing, and no point continuing guessing around until you made sure you have the dependencies installed. Not sure how much clearer I can say that :)

I mean, it's pointless to say "I am not sure" - we know that. Why don't you make sure?

I just used my old laptop because it never had any version of python installed and it worked out. I actually wanted to do a complete new environment but when I was starting to read up on the steps I realized that it's way more work than I initially thought and I wanted to do a complete win10 reset for a long time now so I decided to combine those tasks and do it on my day off next week. Another issue was that my inference tools would also stop working and I needed them to finish a work related thing. I actually really appreciate any help and of course I was planning on applying the advice right away but it seemed to make more sense this way. At least I know now that your hint was definitely the right one and once I set up a new python environment on my fresh new windows install it will work again and hopefully keep working since I will use clean virtual envs for each project which I didn't at first because I just didn't know yet.

Glad that it worked out in the end :)