kataguru/Riva-Translate-4B-Instruct-v2-FP8

NVIDIA Riva-Translate-4B-Instruct-v2 (FP8 compressed-tensors)

This repository provides an optimized FP8 (compressed-tensors) release of NVIDIA's official nvidia/Riva-Translate-4B-Instruct-v2 translation model, specifically engineered for high-throughput, ultra-low-latency deployment with vLLM on modern NVIDIA GPUs (Blackwell RTX 5090, Hopper H100, Ada Lovelace RTX 4090 / L40S).

Key Features & Advantages

• FP8 Precision (e4m3fn): Linear layers are quantized to 8-bit floating point (FP8_DYNAMIC), reducing memory footprint by 50% (4.29 GB total vs. 8.38 GB BF16) while maintaining lossless accuracy (100% parity with unquantized BF16).
• Tied Embeddings Protected: Shared embed_tokens and lm_head are kept in native bfloat16 to prevent vocabulary degradation.
• Ultra-High Throughput: Delivers 315–330 tokens/sec on NVIDIA RTX 5090 with a Time-To-First-Token (TTFT) of ~12–20 ms.
• Universal Multilingual Support: Supports translation between English and 36 languages (both En→Any and Any→En), covering sentence-level and document-level translation tasks.

---

Benchmark & Performance Comparison

Empirical benchmarks run on 2x NVIDIA RTX 5090 (TP=2, vLLM v0.29.0):

Metric	BF16 (Original)	FP8 (This Release)	W4A16 (AWQ)
Model Size	8.38 GB	4.29 GB (-49%)	2.80 GB
VRAM Consumption	3.98 GiB / GPU	2.24 GiB / GPU	1.52 GiB / GPU
Throughput (tok/s)	229 tok/s	~318 tok/s (+39%)	330 tok/s
TTFT (Latency)	~25 ms	~13–20 ms	~11–16 ms
Accuracy Parity	100% (Baseline)	100% (Lossless)	Degraded on small models

---

Deployment with vLLM

You can serve this model directly using vLLM:

vllm serve kataguru/Riva-Translate-4B-Instruct-v2-FP8 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --host 0.0.0.0 \
  --port 8000

Python API Example

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none"
)

# Example: English to German
response = client.chat.completions.create(
    model="kataguru/Riva-Translate-4B-Instruct-v2-FP8",
    messages=[
        {"role": "system", "content": "en-de"},
        {"role": "user", "content": "Accelerated computing enables real-time natural language processing across distributed cloud clusters."}
    ],
    temperature=0.0
)

print(response.choices[0].message.content)

---

Supported Languages

English (en) + 36 languages: Arabic (ar), Bulgarian (bg), Chinese Simplified (zh/zh-cn), Chinese Traditional (zh-tw), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), Estonian (et), Finnish (fi), French (fr), German (de), Greek (el), Hindi (hi), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Korean (ko), Latvian (lv), Lithuanian (lt), Norwegian (no), Polish (pl), Portuguese (pt/pt-br/pt-pt), Romanian (ro), Russian (ru), Slovak (sk), Slovenian (sl), Spanish (es/es-es/es-us), Swedish (sv), Thai (th), Turkish (tr), Ukrainian (uk), Vietnamese (vi).

---

License & Attribution

• Base model: Developed by NVIDIA Corporation (nvidia/Riva-Translate-4B-Instruct-v2).
• License: Governed by the NVIDIA Open Model License Agreement.
• Quantization: Released by kataguru using llmcompressor (compressed-tensors).