gguf invalid and no support in llama.cpp
#8
by
cmp-nct
- opened
Hi,
I noticed the gguf you provided, I'm not sure how you converted them but they are not valid gguf binaries and llama.cpp sadly doesn't support the architecture
Hello,
I believe you are right that llama.cpp does not support T5 models, but the binaries are not invalid. You can use them with candle:
cargo run --example quantized-t5 --release -- \
--model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" \
--prompt "<2de> How are you, my friend?" \
--temperature 0
I generated the files using the util described in the quantized-t5 example
jbochi
changed discussion status to
closed
Oh I see, that's really painful that they used the same format and filename in an incompatible fashion.
Thanks for clarification