Spaces:

KoboldAI

/

Koboldcpp-Tiefighter

like 44

Running on T4

This is a showcase of the ability to use Koboldcpp in a Huggingface space, but without a GPU it is very slow and I can not showcase a clone-able GPU capable instance.

A GPU grant would allow the following:

• GPU instance showcase for a model with up to 5 simultaneous users, the free T4 from Google Colab can handle this so I assume T4 Small would be sufficient.
• Users can see a working instance for the GPU which docker file is easy to adapt to a different GGUF model of choice.
• Interactive API documentation for Koboldcpp that is available to more developers at once.
• Faster OpenAI Endpoint both new and old.
  (- I would be able to move this to the KoboldAI organization properly, so if granted I am not sure if I'd have to make a space there first).

Without a GPU grant:

• I could achieve a similar result by using a much smaller model, but everyone will then have poor outputs and it would not demonstrate how fast inference can be done inside a HF GPU space.
• Users will only be able to use this on CPU spaces without manually editing the Dockerfile.
• I can not realistically provide access to simultaneous users, resulting in unnecessary space cloning.
• The same API's as mentioned above exist, but will error out if someone else is already using it.
• I am still willing to help others setup Koboldcpp GPU spaces, but would be unable to test them on my own account.

KoboldAI is a group of hobbyists working on open source AGPLv3 software. Everything in this space is AGPLv3. We do not accept donations, so we have no way of paying for GPU compute for other users. It would be nice if spaces had a feature where the user of the space pays for a private instance of the space with the GPU, so that we can provide the software without having to worry about the cost. But since this is not available I'd love to make this available as a showcase.

The instance can be interruptible as all user data is stored locally on their system, so even if we have to set it to a 1 hour inactivity timeout this is fine.