Merge branch 'main' of https://huggingface.co/remyxai/SpaceMinitron-4B into main

README.md

@@ -0,0 +1,67 @@
+---
+license: other
+datasets:
+- remyxai/vqasynth_spacellava
+---
+
+# Model Card for SpaceMinitron-4B
+
+**SpaceMinitron-4B** uses [Minitron-4B-Base](https://huggingface.co/nvidia/Minitron-4B-Base) as the llm backbone along with the fused DINOv2+SigLIP features of [prismatic-vlms](https://github.com/TRI-ML/prismatic-vlms).
+
+
+## Model Details
+
+Uses a full fine-tune including the [spacellava dataset](https://huggingface.co/datasets/remyxai/vqasynth_spacellava) designed with [VQASynth](https://github.com/remyxai/VQASynth/tree/main) to enhance spatial reasoning as in [SpatialVLM](https://spatial-vlm.github.io/).
+
+### Model Description
+
+This model uses data synthesis techniques and publically available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models.
+With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create VQA dataset for spatial reasoning.
+
+
+- **Developed by:** remyx.ai
+- **Model type:** MultiModal Model, Vision Language Model, Prismatic-vlms, Minitron-4B-Base
+- **Finetuned from model:** Minitron-4B-Base [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf)
+
+### Model Sources
+- **Dataset:** [SpaceLLaVA](https://huggingface.co/datasets/remyxai/vqasynth_spacellava)
+- **Repository:** [VQASynth](https://github.com/remyxai/VQASynth/tree/main)
+- **Paper:** [SpatialVLM](https://arxiv.org/abs/2401.12168)
+
+## Usage
+
+Try the `run_inference.py` script to run a quick test:
+```bash
+python run_inference.py --model_location remyxai/SpaceMinitron-4B
+                        --image_source "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg"
+                        --user_prompt "What is the distance between the man in the red hat and the pallet of boxes?"
+                        
+```
+
+## Deploy
+Under the `docker` directory, you'll find a dockerized Triton Server for this model. Run the following:
+
+```bash
+docker build -f Dockerfile -t spacellava-server:latest
+docker run -it --rm --gpus all -p8000:8000 -p8001:8001 -p8002:8002 --shm-size 24G spaceminitron-4B-server:latest
+python3 client.py --image_path "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg" \
+                  --prompt "What is the distance between the man in the red hat and the pallet of boxes?"
+```
+
+## Citation
+```
+@article{chen2024spatialvlm,
+  title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
+  author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
+  journal = {arXiv preprint arXiv:2401.12168},
+  year = {2024},
+  url = {https://arxiv.org/abs/2401.12168},
+}
+
+@inproceedings{karamcheti2024prismatic,
+  title = {Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models},
+  author = {Siddharth Karamcheti and Suraj Nair and Ashwin Balakrishna and Percy Liang and Thomas Kollar and Dorsa Sadigh},
+  booktitle = {International Conference on Machine Learning (ICML)},
+  year = {2024},
+}
+```