LFM2.5-VL-450M Hand Tracking

Fine-tuned LFM2.5-VL-450M-Extract for hand bounding box detection in webcam images. Produces [{"label": "hand", "bbox": [x1, y1, x2, y2]}] in 0-1 normalized coordinates.

Disclaimer: This is a personal learning project — my first VLM fine-tune. It went through ~10 iterations of trial and error, documented honestly below (including what didn't work). It is not a research paper, not a production system, and not affiliated with Liquid AI.

The evaluation data is not scientifically rigorous. Val set results (100% accuracy) reflect controlled conditions on curated images. Empirical webcam testing shows more discrepant results, documented transparently below. This model was built for a specific use case (real-time hand tracking via webcam on consumer hardware) and optimized for that scenario, not for general-purpose hand detection.

Use it as a starting point for your own experiments, not as a drop-in solution.

Model Details

  • Base model: LiquidAI/LFM2.5-VL-450M-Extract (450M params, LFM2 architecture)
  • Fine-tuning method: LoRA (r=16, alpha=16)
  • Trainable parameters: 5,013,504 of 453,732,352 (1.10%)
  • Architecture: Lfm2VlForConditionalGeneration (lfm2_vl)
  • Vision encoder: SigLIP2 NaFlex 86M
  • LM backbone: LFM2.5-350M-Base
  • Context length: 4096 tokens
  • Inference speed: ~200ms per frame on RTX 3050 (290 tok/s decode)

Training Details

Hardware

  • GPU: NVIDIA GeForce RTX 3050 6GB Laptop
  • CUDA: 12.6, PyTorch: 2.7.0+cu126
  • Training framework: Unsloth 2026.6.9 + TRL 0.24.0

Hyperparameters

Parameter Value
LoRA rank 16
LoRA alpha 16
LoRA dropout 0
Epochs 2
Learning rate 2e-4
LR scheduler linear
Warmup steps 5
Batch size (train) 2
Batch size (eval) 1
Gradient accumulation 4
Effective batch size 8
Optimizer adamw_8bit
Weight decay 0.001
Precision BF16
Max sequence length 4096
Max response length 2048
Seed 3407

Training Results

Metric Value
Total steps 3,540
Training time ~118 min
Final train loss ~0.55
Final eval loss ~0.42
Seed 3407 (CDA), 42 (split)

LoRA Configuration

  • finetune_vision_layers: False
  • finetune_language_layers: True
  • finetune_attention_modules: True
  • finetune_mlp_modules: True
  • use_gradient_checkpointing: "unsloth"

Dataset

Composition

Category Count Source
Positive (1 hand) 8,578 HaGRID (palm + fist)
Positive (2 hands, oversampled x2) 2,844 HaGRID, duplicated for balance
Negative (HaGRID patches) 2,000 Cropped regions without hands
Negative (COCO val2017, no person) 2,307 Images without any person annotation
Negative (COCO val2017, person no wrists) 594 Images with person but wrists not visible
Total 16,323 30% negatives

Data Sources

HaGRID (HAnd Gesture Recognition Image Dataset)

  • 554,800 FullHD RGB images, 18 gesture classes
  • I used 2 gestures: palm (open hand) + fist (closed hand) for maximum pose diversity
  • Bboxes are normalized [x, y, w, h] in 0-1 range, converted to [x1, y1, x2, y2]
  • Citation: Kapitanov et al., "HaGRID -- HAnd Gesture Recognition Image Dataset", WACV 2024

COCO val2017 (negative samples)

  • 5,000 validation images from COCO 2017
  • 2,307 images without any "person" annotation (guaranteed no hands)
  • 594 images with person annotation but wrists not visible in keypoints (person on screen, hands occluded)
  • The person-no-wrists subset addresses a critical failure mode: the model associated "person in frame" with "hand must exist", producing false positives whenever a person was on camera without hands raised
  • Citation: Lin et al., "Microsoft COCO: Common Objects in Context", ECCV 2014

Dataset iteration The negative sampling went through four iterations, each addressing a failure mode observed in testing:

  1. No negatives — model always detected a hand, even on empty scenes
  2. HaGRID patches only — reduced empty-scene false positives, but still detected hands whenever a person was visible
  3. Added COCO no-person — improved diversity, but person-on-screen still triggered false positives
  4. Added COCO person-no-wrists — resolved the "person = hand" bias

Content order augmentation (positional bias)

During testing, I discovered that the Q8_0 quantization was sensitive to the order of items in the user content. With image before text, the model correctly returned [] when no hands were visible. With text before image, the same model produced false positive hand detections.

This is a form of positional bias. To mitigate this, I applied Counterfactual Data Augmentation (CDA): each training example has a 50% chance of having the image/text order swapped. This teaches the model that content order should not affect the response.

References:

  • AAAI 2024: "Enhance Modality Robustness in Text-Centric Multimodal Alignment"
  • ACL 2024: "Unveiling Selection Biases: Order and Token Sensitivity in LLMs"
  • arXiv 2404.01430: "Position-Aware PEFT for Reducing Positional Bias"

Seed: 3407 (reproducible augmentation)

HaGRID patches (negative samples)

  • 2,000 crops from HaGRID training images, regions outside hand bboxes
  • Various sizes (256-640px), same domain as positives

Data Format

{
  "conversations": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "path/to/image.jpg"},
        {"type": "text", "text": "Detect all hands in the image."}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "[{\"label\":\"hand\",\"bbox\":[0.23,0.45,0.67,0.89]}]"}
      ]
    }
  ]
}

Negative examples use "[]" as the assistant response.

Published Files

Model repo: luksamuk/LFM2.5-VL-450M-Hand-Tracking

File Description
adapter_config.json LoRA adapter config
adapter_model.safetensors LoRA weights (5M params, 24MB)
tokenizer.json + config Tokenizer files
README.md This model card

GGUF repo: luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF

File Size Description
LFM2.5-VL-450M-Hand-Tracking-BF16.gguf ~900MB Full precision (for re-quantization)
LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf 379MB Recommended (best quality)
LFM2.5-VL-450M-Hand-Tracking-Q4_K_M.gguf 229MB Edge / lightweight
mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf 190MB Vision encoder (required)

The BF16 file is the merged model (LoRA + base) in full precision. It allows others to create custom quantizations or use the model without quantization. Q8_0 is recommended for PC/laptop. Q4_K_M is recommended for edge devices (Raspberry Pi).

Usage

llama.cpp (GGUF)

llama-server \
  --model LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf \
  --mmproj mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf \
  --port 8001 \
  --temp 0.1 \
  --jinja

Python (transformers)

from unsloth import FastVisionModel
import torch

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="luksamuk/LFM2.5-VL-450M-Hand-Tracking",
    max_seq_length=4096,
    load_in_4bit=False,
)

Example Application

A demo harness (VLM Webcam Vision) is available at github.com/luksamuk/vlm-webcam-vision. It provides real-time webcam hand tracking with LERP smoothing, label sanitization, and camera controls. Supports any OpenAI-compatible endpoint (llama-swap, Ollama, llama-server) via environment variables.

The llama.cpp server also supports multi-model orchestration natively via model and mmproj flags, enabling hand tracking alongside other models without a proxy.

Output Format

[{"label": "hand", "bbox": [0.23, 0.45, 0.67, 0.89]}]

Coordinates are normalized 0-1. Empty array [] when no hands are visible.

Training Plots

Training Metrics

450M vs 1.6B Comparison

Evaluation Results

Category Q8_0 Q4_K_M
1 hand (recall) 100% 100%
2 hands (recall) 100% 100%
Negative HaGRID (specificity) 100% 100%
Negative COCO no-person 100% 100%
Negative COCO person-no-wrists 100% 100%
Overall accuracy 100% 100%

250 samples (50 per category). Zero false positives, zero false negatives.

Important: Val set accuracy reflects controlled conditions (curated dataset, ideal images). Empirical webcam testing shows more discrepant results — see below.

Empirical Webcam Testing

Structured protocol: ~10s setup + 5s phases (no hands / 1 hand / 2 hands / alternating) + 3s final.

Model Frames [] (empty) 1 hand 2 hands Latency Flicker
450M Q8_0 356 244 (69%) 100 (28%) 12 (3%) 130ms 20
450M Q4_K_M 305 126 (41%) 123 (40%) 56 (18%) 154ms 29
1.6B Q8_0 90 42 (47%) 48 (53%) 0 (0%) 462ms 2
1.6B Q4_K_M 113 53 (47%) 60 (53%) 0 (0%) 405ms 4

Key findings:

  • 2-hand detection is the main weakness across all models in real webcam conditions
  • 450M Q8_0 has the best specificity (69% empty frames) and lowest latency (130ms)
  • 1.6B models show zero 2-hand detection despite 100% val set accuracy — domain gap
  • Both 1.6B variants were trained with QLoRA (4-bit base) due to VRAM constraints

1.6B experiment (not published)

A 1.6B Extract variant was trained with the same dataset using QLoRA (4-bit base, r=16). Val set showed 100% accuracy, identical to 450M. However, empirical webcam testing revealed zero 2-hand detection and 3x higher latency (400-460ms vs 130-154ms).

The 1.6B is excluded from publication because:

  1. It does not improve 2-hand detection in real conditions (the main weakness)
  2. Latency is too high for real-time webcam use (target: <200ms)
  3. QLoRA may have degraded quality — full LoRA on more capable hardware is left as future work

The domain gap (HaGRID studio images vs webcam real-world) appears to be the primary bottleneck for 2-hand detection, not model size. This affects both 450M and 1.6B equally.

Limitations

  • Trained on 2 gestures only (palm + fist); may not generalize to all hand poses
  • 2-hand detection in webcam scenarios may flicker between 1 and 2 detections
  • Optimized for webcam-resolution images (640x480); may need adjustment for higher resolutions
  • Single-person training data; multi-person scenes not tested

What was tried and discarded

  • Horizontal flip augmentation: flipped images + bboxes. Introduced label noise (bbox-only flip without image flip in early versions). Corrected version showed no improvement. Discarded.
  • Distilling with Qwen3-VL (4B): used Qwen3-VL as teacher to generate hand detection labels on COCO images. IoU 0.805 vs HaGRID ground truth, but introduced patterns that degraded person-no-wrists specificity from 100% to 86%. Discarded.
  • Q6_K and Q5_K_M quants: inconsistent behavior across quantization levels. Q6_K produced broken JSON. Q5_K_M over-detected. Only Q8_0 and Q4_K_M published.

Attribution

References

Downloads last month
81
GGUF
Model size
0.4B params
Architecture
lfm2
Hardware compatibility
Log In to add your hardware

4-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF

Adapter
(4)
this model

Dataset used to train luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF

Collection including luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF

Papers for luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF