No correct pipeline_tag
Hi there,
I tried to use the model in a pipeline but it came up with this error.
RuntimeError: The model limcheekin/fastchat-t5-3b-ct2 does not seem to have a correct pipeline_tag set to
infer the task automatically
Not sure exactly how to fix this but could it be as easy as adding a tag like
"Text2Text Generation" to your model? Can you please add that? The same with your flan-alpaca-gpt4-xl-ct2 too. Thanks.
I think you're try to use it in hugging face pipeline, right?
Please look into the ctranslate2 documentation at https://opennmt.net/CTranslate2/, I don't think you can use it in hugging face pipeline.
Hope this help. Thanks.
Thanks for the reply. You sure? I can't find anything particular in the doc about compatibility with huggingface pipeline. I have a pull request to add the pipeline_tag, can you please approve and merge so that I can try? Thanks.
I just merged the pull request. Just get to know the pipeline_tag you mean is the one you added to the readme file.
It is something new to me, may I know what is the use of pipeline_tag? Why you need it?
Thanks. When I tried to load this model in pipeline, it spit out an error saying that this model didn't have a pipeline_tag. I think it needs that inferencing calls.
If you added the pipeline_tag is just to enabled the "Hosted inference API", it will not works, I just tested it myself. I think it is due to reason that the model need to use with the ctranslate2 package as written in the readme.
Hope it is clear.
Bugger, i just tested it as well. Is there other way to quantize it such that it will work with the inference API?
I'm aware of quantization with GPTQ and GGML.
I mainly focus on CTranslate2 and GGML for inferencing with CPU-only compute.
Not sure how it will work with the inference API though, I will close the issue for now.