Spaces:

GT-RIPL

/

GPT-K

like 1

Runtime error

Dear Hugging Face Team,

We are the RIPL academic research lab from Georgia Tech, led by Dr. Zsolt Kira @zsoltkira , and we are excited to submit an application for a GPU grant in support of our research project, GPT-K. GPT-K focuses on incorporating external knowledge for multi-modal vision-and-language (VL) chatting. Through our proposed knowledge augmentations, GPT-K has demonstrated remarkable performance improvement when evaluated on the LLaVA Bench in the Wild dataset.

Resource Constraints

When we attempted to host our demo on Hugging Face Space, we exceeded memory limit of 16GB when loading the model checkpoint. Additionally, our model consists of a ViT-G visual encoder and a 7B LLM, making GPU acceleration necessary for inference speed.

Gradio Interface

To make our research accessible and user-friendly, we have built a user interface using Gradio as shown in the figure below. This interface allows users to upload images and pose questions related to those images. GPT-K, in turn, performs knowledge retrieval and incorporates the retrieved knowledge to provide answers. Moreover, we visualize the retrieved knowledge on the bottom left block of the UI.

Quantitative Results

By introducing external knowledge, GPT-K has achieved a substantial leap in performance on LLaVA Bench when compared to the same model without external knowledge, as demonstrated in the table below:

We believe that our project aligns with the vision and mission of Hugging Face. The GPU grant we are applying for would significantly enhance our ability to continue this vital research, addressing resource limitations and allowing us to contribute even more effectively to the AI communities.