liuhaotian

/

llava-v1.5-7b

like 264

hey @clem @liuhaotian !
i've integrated the 7b version into a huggingface space, let me know what you think!
https://huggingface.co/spaces/badayvedat/LLaVA

I really want to use this model as a hosted API. Would I be able to run that same space on a GPU?

Update: I tried running that space on a T4 Small and A10G Small, but it errored out.

@sjayaram it requires at least an Nvidia 10G Large to run the space. Let me know if you encounter any errors, I'd gladly help!

@badayvedat nice! the space is paused at the moment but looking forward to trying it!

@clem i stopped it due to the Nvidia 10G large's pricing, resuming it right now.

Thanks for the tag @clem !
The demo works brilliantly @badayvedat . Kudos for setting this up! We can do a few QOL updates to the Space, for example, using the built-in upvote/downvote functionality within the Gradio chatbot, using avatars, and chat bubble size. I'll go through the code, I'll be sure to raise a PR tomorrow.

Thanks, @ysharma , I will gladly help if you need any help with the code!

Also, I know that this is not the right thread to discuss it but it would be great if we could get a GPU grant for it, so that I wouldn't need to stop it.
More on: https://huggingface.co/spaces/badayvedat/LLaVA/discussions/1

Great work @badayvedat , the demo works wonderfully. As I understand it, the API endpoints provided by Gradio for this space don't really offer full functionality, right? I believe this would be due to the frontend's role in persisting chat history (which isn't part of the API) and the streaming functionality in the HTTP bot function. I'm not too familiar with this code base.

I made a space (forked from yours) that provides a gpt-3.5 like API that takes JSON chat history (and an image) and returns a completion.

Also, I'm able to run my version on a T4 small, which is great for ppl who just want to set up a personal API that they can hit for a project.

Its available here: https://huggingface.co/spaces/sjayaram/LLaVA

yes @sjayaram you're right ! and its complexity is due to the "chat-like" functionalities. I don't have the details for it either since it was built on the original gradio app provided by the liuhaotian.
Your space is looking amazing, very clean and organized!
I was planning to implement a similar API-like functionality with fal

qq: is it using LLaVA-1.5 weights or old weights? I saw it using LLaVA-Lightning-MPT-7B-preview, i think they are not the same as LLaVA-1.5, but not sure how they differ?

Hi @sjayaram I am unable to use your demo atm. I think I can help you with your query about using your demo as an API, but I would instead continue this discussion in a separate thread on your app. I'd suggest starting a conversation and tagging me there if you're still looking for a solution.