rlabz/quantum-s2s

Moshika Swahili Speech-to-Speech

A full-duplex speech-to-speech model fine-tuned from Moshika for Swahili conversation, using kyutai-labs/moshi-finetune.

Overview

This model adapts Moshi's full-duplex spoken dialogue architecture to Swahili, enabling real-time, overlapping (not strictly turn-based) speech interaction directly in the audio domain — no intermediate text transcription step required for the conversational loop.

• Base model: kyutai/moshika-pytorch-bf16
• Fine-tuning framework: moshi-finetune
• Training data: rlabz/mwanamke_moshi
• Task: Full-duplex speech-to-speech (Swahili)

Training

Fine-tuning was done with LoRA via moshi-finetune, more details in this notebook.

Hardware requirements

Fine-tuning requires at least an A100 GPU. Moshika is a 7B-parameter model in bf16, which alone occupies close to the full memory capacity of smaller GPUs (e.g. T4) before accounting for LoRA adapters, gradient checkpointing overhead, and activations — an A100 (40GB+) is needed for a stable training run.

Inference

Once the model has been trained, inference can be run on the same Colab GPU used for training, and Gradio can be used to tunnel audio data from a local client to the notebook.

More details on how to set this up can be found in the moshi readme.

Dataset

Trained on rlabz/mwanamke_moshi, a Swahili speech dataset formatted for full-duplex conversational fine-tuning.

Limitations

• This is a fine-tuned adaptation of Moshika and inherits the base model's general limitations and biases.
• Full-duplex behavior and Swahili conversational quality depend on the coverage of the training data; performance may vary across dialects, speakers, and topics not well represented in the dataset.

Acknowledgements

• Kyutai Labs for Moshi/Moshika and the moshi-finetune toolkit.