docs/walkthrough.md

SignBridge — technical walkthrough

Internal technical record of the build. Not a submission deliverable (Build-in-Public extra challenge was dropped on 2026-05-07). Kept around because it documents the AMD-specific engineering thinking and is useful if anyone later asks "why these design choices?".

What we built

A real-time webcam-based ASL → English speech translator. A deaf user signs into the webcam; the pipeline (MediaPipe Holistic → trained sign classifier → Llama-3.1-8B sentence composer → Coqui XTTS-v2) returns spoken English in under 2 seconds. Designed to fit Track 3 (Vision & Multimodal AI) with the entire model stack running concurrently on a single AMD Instinct MI300X.

Why AMD MI300X

• 192 GB HBM3 — the trained classifier (20 MB), Llama-3.1-8B (16 GB FP16), XTTS-v2 (2 GB), and (V2 stretch) Whisper-large-v3 (3 GB) all fit concurrently with margin for KV cache.
• 5.3 TB/s memory bandwidth — bandwidth-bound streaming workload (many small inferences per second on the classifier + TTS chunked decode + LLM next-token) is exactly what bandwidth wins.

Architecture

webcam frames → MediaPipe Holistic → trained classifier
                  (CPU-fast)            (TorchScript on MI300X)
                                              │
                                              ▼
                                  Llama-3.1-8B sentence composer
                                       (vLLM on MI300X)
                                              │
                                              ▼
                                          XTTS-v2 → audio
                                       (XTTS on MI300X)

Models

Component	Source	Notes
Pose extractor	MediaPipe Holistic (Google)	CPU-fast preprocessing — not GPU-bound
Sign classifier	trained from scratch on WLASL Top-100 + ASL fingerspelling alphabet	3-layer transformer encoder over 543-dim landmark sequences; published to HF Hub at lucas-loo/signbridge-classifier
Sentence composer	meta-llama/Llama-3.1-8B-Instruct	Pulled from HF Hub; served on MI300X via vLLM
Text-to-speech	coqui/XTTS-v2	Multilingual; we use English V1

Datasets

• WLASL Top-100 subset
• ASL fingerspelling alphabet (open dataset)

ROCm / AMD Developer Cloud experience

Filled in across Day 1–3.

Day 1 — environment + sanity

TODO

Day 2 — training the classifier

TODO

Day 3 — serving + latency tuning

TODO

What worked well

TODO

What we'd flag as friction

TODO

Why AMD MI300X — concretely

The pipeline (MediaPipe Holistic + Qwen3-VL-8B + Llama-3.1-8B + Coqui XTTS-v2) fits comfortably on a single MI300X with KV-cache headroom. The same workload on NVIDIA forces sharding once we add the V2 reasoner.

Component	Weights (FP16)	MI300X 1× (192 GB)	H100 80 GB	H200 141 GB
Qwen3-VL-8B (vision)	~16 GB	✅ fits	✅	✅
Llama-3.1-8B (composer)	~16 GB	✅ fits	✅	✅
Whisper-large-v3 (V2 reverse direction)	~3 GB	✅ fits	⚠ tight	✅
Coqui XTTS-v2 (TTS)	~2 GB	✅ fits	⚠ tight	✅
(V2) Llama-3.1-70B FP8 reasoner upgrade	~70 GB	✅ still fits	❌ doesn't fit at all	⚠ FP8 only, no headroom
Concurrent serving + KV cache	✅ comfortable	❌ requires sharding	⚠ tight	✅

The single-GPU concurrency story is the AMD pitch. V1 fits anywhere; the architecture has clear MI300X headroom for V2 model upgrades that NVIDIA H100 cannot match without sharding across multiple cards.

Deployment ethics

SignBridge is a substrate, not a finished product. We ship the open-source multi-modal pipeline so Deaf-led organisations — schools-for-the-Deaf, regional NGOs, ministries of social services — can deploy on their own AMD compute, fine-tune for their dialect, and own the deployment.

Three principles, drawn from the Deaf-led literature on sign-language AI:

1. ASL only V1 is a scope decision. Sign languages are not interchangeable — BSL, ISL, MSL, CSL, and 200+ other sign languages each deserve their own teams, training data, and Deaf community leadership. Bragg et al. "Systemic Biases in Sign Language AI Research" (2024, Deaf-led position paper) is direct on this point.

2. Deaf community engagement before deployment. Per the ACM ASSETS 2025 "Exploring Collaboration to Center the Deaf Community in Sign Language AI" the productive ML/Deaf collaboration question isn't "how do we build this?" but "should we build this, for whom, with whom?". Any deployment downstream of this code must answer that locally.

3. Privacy by default. SignBridge sessions are ephemeral — webcam frames and audio are processed in-memory and not persisted server-side beyond the request lifetime. In the spirit of Privacy-Aware Sign Language Translation at Scale (ACL 2024).

Future work — academic foundations we'd build on next

• SignCLIP (EMNLP 2024) — learned text↔sign embeddings; replaces the prompt-only composer with a CLIP-style alignment head for higher-quality sign-to-English mapping.
• SL-SLR (arXiv 2509.05188v1) — self-supervised representation learning with motion-aware data augmentation; the right path if we ever train a custom classifier on raw signer footage.
• Continuous SLT trained models (Swin-MSTP, Stack Transformer) — the current trained-from-scratch ceiling on WLASL is ~93.5% Top-1. The VLM zero-shot path we ship here is a deployment-cost play, not an accuracy-ceiling play; SignCLIP-style learned embeddings are the natural V2 step toward that ceiling.

Latency

Target: ≤ 2 s from end-of-sign to start of speech.

Measured on a single MI300X (Day 3):

• MediaPipe Holistic per frame: TODO ms
• Classifier per window: TODO ms
• Llama-3.1-8B sentence composition (≤ 30 tokens): TODO ms
• XTTS-v2 first-audio-chunk: TODO ms

MI300X vs NVIDIA H100 — the AMD pitch

Item	MI300X (1 GPU)	H100 (1 GPU)	H100 cluster needed
Llama-3.1-8B FP16 weights	✅ fits with margin	✅ fits with margin	1×
+ XTTS-v2 + Whisper-large-v3 + classifier	✅ all concurrent	⚠️ tight (~28 GB total + KV)	likely 1× but no headroom
+ 70B reasoner upgrade (V2)	✅ 70B FP8 ~70 GB still fits	❌ doesn't fit at all	≥3×

The single-GPU concurrency story is the AMD pitch. This V1 fits on H100; the architecture has clear headroom on MI300X for higher-quality V2 models.

License

MIT.