Instructions to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF", dtype="auto") - llama-cpp-python
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF", filename="LFM2.5-VL-450M-hand-tracking-Q4_K_M.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] ) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Use Docker
docker model run hf.co/luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
- SGLang
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Ollama
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Ollama:
ollama run hf.co/luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
- Unsloth Studio
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF to start chatting
- Pi
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat new
- OpenClaw new
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Docker Model Runner:
docker model run hf.co/luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
- Lemonade
How to use luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.LFM2.5-VL-450M-Hand-Tracking-GGUF-Q4_K_M
List all available models
lemonade list
LFM2.5-VL-450M Hand Tracking
Fine-tuned LFM2.5-VL-450M-Extract for hand bounding box detection in webcam images. Produces [{"label": "hand", "bbox": [x1, y1, x2, y2]}] in 0-1 normalized coordinates.
Disclaimer: This is a personal learning project — my first VLM fine-tune. It went through ~10 iterations of trial and error, documented honestly below (including what didn't work). It is not a research paper, not a production system, and not affiliated with Liquid AI.
The evaluation data is not scientifically rigorous. Val set results (100% accuracy) reflect controlled conditions on curated images. Empirical webcam testing shows more discrepant results, documented transparently below. This model was built for a specific use case (real-time hand tracking via webcam on consumer hardware) and optimized for that scenario, not for general-purpose hand detection.
Use it as a starting point for your own experiments, not as a drop-in solution.
Model Details
- Base model: LiquidAI/LFM2.5-VL-450M-Extract (450M params, LFM2 architecture)
- Fine-tuning method: LoRA (r=16, alpha=16)
- Trainable parameters: 5,013,504 of 453,732,352 (1.10%)
- Architecture: Lfm2VlForConditionalGeneration (lfm2_vl)
- Vision encoder: SigLIP2 NaFlex 86M
- LM backbone: LFM2.5-350M-Base
- Context length: 4096 tokens
- Inference speed: ~200ms per frame on RTX 3050 (290 tok/s decode)
Training Details
Hardware
- GPU: NVIDIA GeForce RTX 3050 6GB Laptop
- CUDA: 12.6, PyTorch: 2.7.0+cu126
- Training framework: Unsloth 2026.6.9 + TRL 0.24.0
Hyperparameters
| Parameter | Value |
|---|---|
| LoRA rank | 16 |
| LoRA alpha | 16 |
| LoRA dropout | 0 |
| Epochs | 2 |
| Learning rate | 2e-4 |
| LR scheduler | linear |
| Warmup steps | 5 |
| Batch size (train) | 2 |
| Batch size (eval) | 1 |
| Gradient accumulation | 4 |
| Effective batch size | 8 |
| Optimizer | adamw_8bit |
| Weight decay | 0.001 |
| Precision | BF16 |
| Max sequence length | 4096 |
| Max response length | 2048 |
| Seed | 3407 |
Training Results
| Metric | Value |
|---|---|
| Total steps | 3,540 |
| Training time | ~118 min |
| Final train loss | ~0.55 |
| Final eval loss | ~0.42 |
| Seed | 3407 (CDA), 42 (split) |
LoRA Configuration
- finetune_vision_layers: False
- finetune_language_layers: True
- finetune_attention_modules: True
- finetune_mlp_modules: True
- use_gradient_checkpointing: "unsloth"
Dataset
Composition
| Category | Count | Source |
|---|---|---|
| Positive (1 hand) | 8,578 | HaGRID (palm + fist) |
| Positive (2 hands, oversampled x2) | 2,844 | HaGRID, duplicated for balance |
| Negative (HaGRID patches) | 2,000 | Cropped regions without hands |
| Negative (COCO val2017, no person) | 2,307 | Images without any person annotation |
| Negative (COCO val2017, person no wrists) | 594 | Images with person but wrists not visible |
| Total | 16,323 | 30% negatives |
Data Sources
HaGRID (HAnd Gesture Recognition Image Dataset)
- 554,800 FullHD RGB images, 18 gesture classes
- I used 2 gestures: palm (open hand) + fist (closed hand) for maximum pose diversity
- Bboxes are normalized [x, y, w, h] in 0-1 range, converted to [x1, y1, x2, y2]
- Citation: Kapitanov et al., "HaGRID -- HAnd Gesture Recognition Image Dataset", WACV 2024
COCO val2017 (negative samples)
- 5,000 validation images from COCO 2017
- 2,307 images without any "person" annotation (guaranteed no hands)
- 594 images with person annotation but wrists not visible in keypoints (person on screen, hands occluded)
- The person-no-wrists subset addresses a critical failure mode: the model associated "person in frame" with "hand must exist", producing false positives whenever a person was on camera without hands raised
- Citation: Lin et al., "Microsoft COCO: Common Objects in Context", ECCV 2014
Dataset iteration The negative sampling went through four iterations, each addressing a failure mode observed in testing:
- No negatives — model always detected a hand, even on empty scenes
- HaGRID patches only — reduced empty-scene false positives, but still detected hands whenever a person was visible
- Added COCO no-person — improved diversity, but person-on-screen still triggered false positives
- Added COCO person-no-wrists — resolved the "person = hand" bias
Content order augmentation (positional bias)
During testing, I discovered that the Q8_0 quantization was sensitive to
the order of items in the user content. With image before text, the
model correctly returned [] when no hands were visible. With text
before image, the same model produced false positive hand detections.
This is a form of positional bias. To mitigate this, I applied Counterfactual Data Augmentation (CDA): each training example has a 50% chance of having the image/text order swapped. This teaches the model that content order should not affect the response.
References:
- AAAI 2024: "Enhance Modality Robustness in Text-Centric Multimodal Alignment"
- ACL 2024: "Unveiling Selection Biases: Order and Token Sensitivity in LLMs"
- arXiv 2404.01430: "Position-Aware PEFT for Reducing Positional Bias"
Seed: 3407 (reproducible augmentation)
HaGRID patches (negative samples)
- 2,000 crops from HaGRID training images, regions outside hand bboxes
- Various sizes (256-640px), same domain as positives
Data Format
{
"conversations": [
{
"role": "user",
"content": [
{"type": "image", "image": "path/to/image.jpg"},
{"type": "text", "text": "Detect all hands in the image."}
]
},
{
"role": "assistant",
"content": [
{"type": "text", "text": "[{\"label\":\"hand\",\"bbox\":[0.23,0.45,0.67,0.89]}]"}
]
}
]
}
Negative examples use "[]" as the assistant response.
Published Files
Model repo: luksamuk/LFM2.5-VL-450M-Hand-Tracking
| File | Description |
|---|---|
adapter_config.json |
LoRA adapter config |
adapter_model.safetensors |
LoRA weights (5M params, 24MB) |
tokenizer.json + config |
Tokenizer files |
README.md |
This model card |
GGUF repo: luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF
| File | Size | Description |
|---|---|---|
LFM2.5-VL-450M-Hand-Tracking-BF16.gguf |
~900MB | Full precision (for re-quantization) |
LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf |
379MB | Recommended (best quality) |
LFM2.5-VL-450M-Hand-Tracking-Q4_K_M.gguf |
229MB | Edge / lightweight |
mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf |
190MB | Vision encoder (required) |
The BF16 file is the merged model (LoRA + base) in full precision. It allows others to create custom quantizations or use the model without quantization. Q8_0 is recommended for PC/laptop. Q4_K_M is recommended for edge devices (Raspberry Pi).
Usage
llama.cpp (GGUF)
llama-server \
--model LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf \
--mmproj mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf \
--port 8001 \
--temp 0.1 \
--jinja
Python (transformers)
from unsloth import FastVisionModel
import torch
model, tokenizer = FastVisionModel.from_pretrained(
model_name="luksamuk/LFM2.5-VL-450M-Hand-Tracking",
max_seq_length=4096,
load_in_4bit=False,
)
Example Application
A demo harness (VLM Webcam Vision) is available at github.com/luksamuk/vlm-webcam-vision. It provides real-time webcam hand tracking with LERP smoothing, label sanitization, and camera controls. Supports any OpenAI-compatible endpoint (llama-swap, Ollama, llama-server) via environment variables.
The llama.cpp server also supports multi-model orchestration natively via model and mmproj flags, enabling hand tracking alongside other models without a proxy.
Output Format
[{"label": "hand", "bbox": [0.23, 0.45, 0.67, 0.89]}]
Coordinates are normalized 0-1. Empty array [] when no hands are visible.
Training Plots
Evaluation Results
| Category | Q8_0 | Q4_K_M |
|---|---|---|
| 1 hand (recall) | 100% | 100% |
| 2 hands (recall) | 100% | 100% |
| Negative HaGRID (specificity) | 100% | 100% |
| Negative COCO no-person | 100% | 100% |
| Negative COCO person-no-wrists | 100% | 100% |
| Overall accuracy | 100% | 100% |
250 samples (50 per category). Zero false positives, zero false negatives.
Important: Val set accuracy reflects controlled conditions (curated dataset, ideal images). Empirical webcam testing shows more discrepant results — see below.
Empirical Webcam Testing
Structured protocol: ~10s setup + 5s phases (no hands / 1 hand / 2 hands / alternating) + 3s final.
| Model | Frames | [] (empty) |
1 hand | 2 hands | Latency | Flicker |
|---|---|---|---|---|---|---|
| 450M Q8_0 | 356 | 244 (69%) | 100 (28%) | 12 (3%) | 130ms | 20 |
| 450M Q4_K_M | 305 | 126 (41%) | 123 (40%) | 56 (18%) | 154ms | 29 |
| 1.6B Q8_0 | 90 | 42 (47%) | 48 (53%) | 0 (0%) | 462ms | 2 |
| 1.6B Q4_K_M | 113 | 53 (47%) | 60 (53%) | 0 (0%) | 405ms | 4 |
Key findings:
- 2-hand detection is the main weakness across all models in real webcam conditions
- 450M Q8_0 has the best specificity (69% empty frames) and lowest latency (130ms)
- 1.6B models show zero 2-hand detection despite 100% val set accuracy — domain gap
- Both 1.6B variants were trained with QLoRA (4-bit base) due to VRAM constraints
1.6B experiment (not published)
A 1.6B Extract variant was trained with the same dataset using QLoRA (4-bit base, r=16). Val set showed 100% accuracy, identical to 450M. However, empirical webcam testing revealed zero 2-hand detection and 3x higher latency (400-460ms vs 130-154ms).
The 1.6B is excluded from publication because:
- It does not improve 2-hand detection in real conditions (the main weakness)
- Latency is too high for real-time webcam use (target: <200ms)
- QLoRA may have degraded quality — full LoRA on more capable hardware is left as future work
The domain gap (HaGRID studio images vs webcam real-world) appears to be the primary bottleneck for 2-hand detection, not model size. This affects both 450M and 1.6B equally.
Limitations
- Trained on 2 gestures only (palm + fist); may not generalize to all hand poses
- 2-hand detection in webcam scenarios may flicker between 1 and 2 detections
- Optimized for webcam-resolution images (640x480); may need adjustment for higher resolutions
- Single-person training data; multi-person scenes not tested
What was tried and discarded
- Horizontal flip augmentation: flipped images + bboxes. Introduced label noise (bbox-only flip without image flip in early versions). Corrected version showed no improvement. Discarded.
- Distilling with Qwen3-VL (4B): used Qwen3-VL as teacher to generate hand detection labels on COCO images. IoU 0.805 vs HaGRID ground truth, but introduced patterns that degraded person-no-wrists specificity from 100% to 86%. Discarded.
- Q6_K and Q5_K_M quants: inconsistent behavior across quantization levels. Q6_K produced broken JSON. Q5_K_M over-detected. Only Q8_0 and Q4_K_M published.
Attribution
- Base model: LiquidAI/LFM2.5-VL-450M-Extract -- LFM Open License v1.0
- Training dataset: HaGRID by Kapitanov et al. -- CC BY-SA 4.0
- Negative samples: COCO by Lin et al. -- CC BY 4.0
- Training framework: Unsloth -- Apache 2.0
References
- Kapitanov, A., Kvanchiani, K., Nagaev, A., Kraynov, R., Makhliarchuk, A. "HaGRID -- HAnd Gesture Recognition Image Dataset." WACV 2024. arXiv:2206.08219
- Nuzhdin, A., et al. "HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition." 2024. arXiv:2412.01508
- Lin, T.-Y., et al. "Microsoft COCO: Common Objects in Context." ECCV 2014. arXiv:1405.0312
- Unsloth LFM2.5-VL fine-tuning guide: https://unsloth.ai/docs/models/tutorials/lfm2.5
- LFM2 Technical Report: arXiv:2511.23404
- Downloads last month
- 81
4-bit
8-bit
Model tree for luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF
Base model
LiquidAI/LFM2.5-350M-Base