nitsugua/speech-llm-translation

Speech-to-Text Translation with LLM Adapter

End-to-end speech-to-text translation pipeline designed for low-resource languages. Combines a frozen speech encoder (Whisper/MMS) with a frozen multilingual LLM (Aya-101/Aya-23) through a lightweight trainable adapter (1D CNN + QFormer + MLP).

Architecture

Audio → [Whisper/MMS Encoder] → [1D CNN Downsample] → [QFormer] → [MLP Projection] → [LLM (Aya-101/Aya-23)]
         (frozen)                  (trainable)           (trainable)    (trainable)      (frozen/LoRA)

Key Features:

• Modality Bridge: 1D CNN compresses time dimension, QFormer (cross-attention queries) compresses to fixed-size tokens (~64), MLP projects to LLM dimension
• Dual LLM Support: Works with both Seq2Seq (T5/Aya-101) and CausalLM (Aya-23/Llama) architectures
• Low-Resource Optimized: Curriculum training (ASR → Translation → Fine-tune) for data-efficient transfer
• PEFT Compatible: Optional LoRA on LLM for extra capacity with minimal compute

Quick Start

Installation

pip install -r requirements.txt

Training

Basic Single-Stage Training

python train.py \
  --speech-encoder-name openai/whisper-large-v3 \
  --llm-name CohereLabs/aya-101 \
  --data-path ./data/train.jsonl \
  --eval-data-path ./data/eval.jsonl \
  --output-dir ./outputs \
  --num-epochs 3 \
  --batch-size 4 \
  --gradient-accumulation-steps 4

Curriculum Training (Recommended for Low-Resource)

python train.py \
  --speech-encoder-name openai/whisper-large-v3 \
  --llm-name CohereLabs/aya-101 \
  --stage1-data ./data/asr_corpus.jsonl \
  --stage1-epochs 2 \
  --stage1-lr 5e-5 \
  --stage2-data ./data/translation_pairs.jsonl \
  --stage2-epochs 3 \
  --stage2-lr 3e-5 \
  --stage3-data ./data/target_language.jsonl \
  --stage3-epochs 2 \
  --stage3-lr 1e-5 \
  --output-dir ./outputs \
  --batch-size 4 \
  --use-lora

Inference

python inference.py \
  --model-path ./outputs/stage2_translation_final \
  --audio-path ./audio/sample.wav \
  --task translation \
  --source-lang sw \
  --target-lang en

Data Format

Your data should be a JSONL file where each line is:

{
  "audio": "/path/to/audio.wav",
  "text": "The transcription or translation text",
  "task": "asr",
  "source_lang": "sw",
  "target_lang": "sw"
}

For datasets with inline audio arrays (from HuggingFace):

{
  "audio": {"array": [0.0, 0.01, ...], "sampling_rate": 16000},
  "text": "The text",
  "task": "translation",
  "source_lang": "sw",
  "target_lang": "en"
}

Model Variants

Recommended Configs

Config	Encoder	LLM	Queries	CNN Layers	Stage	Best For
Standard	whisper-large-v3	Aya-101	64	2	Curriculum	General multilingual ST
Low-Resource	mms-1b-all	Aya-101	32	2	ASR-only	Languages with <10h data
Speed	whisper-turbo	Aya-23-8B	32	2	Single	Real-time inference
Quality	whisper-large-v3	Aya-23-35B	150	2	Full	Best translation quality

Research Foundation

This implementation is based on:

• MCAT (2025): QFormer+Pooling architecture for 70-language speech translation
• Soundwave (2025): Curriculum learning with CTC-based compression
• LLaST (2024): Dual-LoRA training with ASR augmentation
• Adapter Comparison (2024): Systematic benchmarking of 5 adapter types

Key insight from the literature: Speech encoder choice >> adapter design >> LLM size for ST quality. For low-resource languages, MMS supports 1000+ languages while Whisper has stronger representations. Use curriculum training (ASR → ST) for maximum transfer.

Project Structure

.
├── model.py           # SpeechLLM model definition
├── train.py           # Training script with curriculum stages
├── inference.py       # Inference/generation script
├── data_utils.py      # Dataset and collation utilities
├── data_prep.py       # Data preparation and pseudo-labeling
├── requirements.txt   # Python dependencies
└── README.md          # This file

Citation

@article{speechllm2025,
  title={Speech-to-Text Translation with LLM Adapters},
  year={2025}
}