XEUS: Cross-lingual E-Branchformer Universal Speech Representation (SafeTensors BF16)

This repository provides the SafeTensors (bfloat16) conversion and standalone pure-PyTorch implementation of XEUS, originally developed by Carnegie Mellon University's WAVLab.

• Original Hub Repository: espnet/xeus
• Paper: XEUS: A Cross-lingual Encoder for Universal Speech
• Architecture: E-Branchformer (19 layers, 577M parameters)
• Pretraining scale: 1,000,000+ hours across 4,000+ languages
• Input: 16 kHz raw audio waveform (batch_size, time_samples)
• Output: Frame representations (batch_size, time_frames, 1024) at 50 Hz frame rate (stride 320, 20ms per frame)
• Precision: bfloat16 (1.07 GB vs 2.15 GB in float32, 50% storage & memory savings)
• Weight format: model.safetensors (fast, zero arbitrary code execution, memory-mapped)

---

Architecture Specifications

Parameter	Value
Pretrained Objective	HuBERT-style masked prediction + acoustic noise & reverberation augmentation
Input Format	Raw mono waveform at 16,000 Hz
Frontend	7-layer Conv1d extractor (strides: 5, 2, 2, 2, 2, 2, 2; total stride: 320)
Feature Dimension ($d_{model}$)	1024
Intermediate FFN Dimension	4096
Number of E-Branchformer Layers	19
Attention Heads	8
CGMLP / CSGU Kernel Size	31 (depthwise)
Output Frame Rate	50.0 Hz
Default Precision	torch.bfloat16

---

Standalone Quickstart (No ESPnet installation required)

This repository includes a standalone modeling_xeus.py implementation that runs on pure PyTorch without requiring ESPnet or custom C++ dependencies.

Installation

pip install torch safetensors huggingface_hub

Inference Example

import torch
from huggingface_hub import hf_hub_download

# Download modeling code and load weights
# Or copy modeling_xeus.py directly to your codebase
from modeling_xeus import XeusTrunk

# Automatically downloads bfloat16 model.safetensors from this repository
model = XeusTrunk.from_pretrained("giangndm/xeus", dtype=torch.bfloat16)
model.eval()

# Input: 16 kHz waveform (batch_size, num_samples)
# Example: 1.0 second of dummy audio
audio = torch.randn(1, 16000)

with torch.no_grad():
    # Shape: [1, 49, 1024] (50 Hz frame rate)
    features = model(audio)

print("Acoustic representation shape:", features.shape, "dtype:", features.dtype)

---

Weight Verification & Conversion Details

The weights in model.safetensors were converted directly from model/xeus_checkpoint_new.pth (commit 02acdd3ac9445561eeb7e9e7f096e4482a32b3a7) of espnet/xeus to bfloat16 SafeTensors format.

• Tensors: All 763 tensors converted to contiguous bfloat16 format.
• Size: ~1.07 GB (compact footprint for efficient training and inference).

---

License

This model and its weights are distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC-BY-NC-SA 4.0) license, matching the original license established by the authors at CMU WAVLab / ESPnet. See LICENSE for complete legal terms.

---

Citation

@article{chen2024xeus,
  title={XEUS: A Cross-lingual Encoder for Universal Speech},
  author={Chen, Wan-Chi and Choi, Kwanghee and Kim, David and Watanabe, Shinji},
  journal={arXiv preprint arXiv:2406.11832},
  year={2024}
}