Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Paper
•
2412.14171
•
Published
•
21
NYU VisionX
university
AI & ML interests
None defined yet.
Recent Activity
View all activity
nyu-visionx's activity
jihanyang
authored
a
paper
3 days ago
rilynhan
authored
a
paper
3 days ago
sainx
authored
a
paper
3 days ago
ShushengYang
authored
5
papers
4 days ago
Qwen Technical Report
Paper
•
2309.16609
•
Published
•
35
ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers
Paper
•
2305.15272
•
Published
TouchStone: Evaluating Vision-Language Models by Language Models
Paper
•
2308.16890
•
Published
•
1
Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection
Paper
•
2204.02964
•
Published
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Paper
•
2412.14171
•
Published
•
21
ShushengYang
updated
a
dataset
4 days ago
Post
1496
In the past seven days, the Diffusers team has shipped:
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
- 1 reply
rilynhan
updated
a
dataset
5 days ago
jihanyang
updated
a
dataset
5 days ago
Post
2029
Introducing a high-quality open-preference dataset to further this line of research for image generation.
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
- 7 replies
Post
2089
The Control family of Flux from
@black-forest-labs
should be discussed more!
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script 🤗
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script 🤗
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
sayakpaul
authored
a
paper
17 days ago
Post
1465
Let 2024 be the year of video model fine-tunes!
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Post
2595
It's been a while we shipped native quantization support in
We currently support
This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
diffusers 🧨We currently support
bistandbytes as the official backend but using others like torchao is already very simple. This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
enable_model_cpu_offload()5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
- 1 reply
Post
2752
Did some little experimentation to resize pre-trained LoRAs on Flux. I explored two themes:
* Decrease the rank of a LoRA
* Increase the rank of a LoRA
The first one is helpful in reducing memory requirements if the LoRA is of a high rank, while the second one is merely an experiment. Another implication of this study is in the unification of LoRA ranks when you would like to
Check it out here:
sayakpaul/flux-lora-resizing
* Decrease the rank of a LoRA
* Increase the rank of a LoRA
The first one is helpful in reducing memory requirements if the LoRA is of a high rank, while the second one is merely an experiment. Another implication of this study is in the unification of LoRA ranks when you would like to
torch.compile() them. Check it out here:
sayakpaul/flux-lora-resizing
- 1 reply
sayakpaul
authored
a
paper
4 months ago
Post
2945
Here is a hackable and minimal implementation showing how to perform distributed text-to-image generation with Diffusers and Accelerate.
Full snippet is here: https://gist.github.com/sayakpaul/cfaebd221820d7b43fae638b4dfa01ba
With @JW17
Full snippet is here: https://gist.github.com/sayakpaul/cfaebd221820d7b43fae638b4dfa01ba
With @JW17