Using FAST API to query the model.
#99
by
moc1pher
- opened
How to use FAST API to query the model. Any example that I can use ?
tabbyAPI is based on fastAPI - can probably learn a lot from it.
https://github.com/theroyallab/tabbyAPI
Is there an option where it is completely Fastapi ?
Maybe this is better for your use case: https://github.com/c0sogi/LLMChat
based on fastAPI, but without the OpenAI API layer
I just like tabbyAPI for running local models on exl2 quant and consuming them like you would OpenAI API