dolphinteam/OpenWhistle-JeanZay

OpenWhistle JeanZay

dolphinteam/OpenWhistle-JeanZay provides a Wav2Vec2 audio backbone pretrained from scratch on bottlenose dolphin whistle recordings using native Fairseq on sixteen NVIDIA V100 GPUs at Jean Zay.

It follows the repository layout and frontend geometry of OpenWhistle-Wav2Vec2.0. The weights come from the completed Jean Zay run at 400,000 optimizer updates.

Model Details

• Model type: wav2vec2
• Transformers architecture: Wav2Vec2ForPreTraining
• Hidden size: 768; Transformer layers: 12; attention heads: 8
• Seven convolutional layers, with strides [15, 2, 2, 2, 2, 2, 2]
• Total stride: 960 audio samples, approximately 21.77 ms at 44,100 Hz
• Sampling rate: 44,100 Hz; mono audio
• Weights: model.safetensors, exported in float32

The export maps every learned Fairseq tensor strictly and checks encoder embeddings against the native checkpoint. Conversion details and the checkpoint SHA-256 are recorded in export_provenance.json.

Training and Evaluation Data

Pretraining uses dolphinteam/OpenWhistle-Pretraining, pinned to revision eea37423c7841b0f99a7c0380f945999c0a2c489.

The manifests contain 28,410 training examples and 3,370 validation examples. Fairseq excludes two training examples shorter than five seconds. Audio was prepared as normalized mono 44.1 kHz input using the native Fairseq FLAC manifests.

Training completed on 12 October 2026 with native Fairseq 0.12.2, PyTorch 2.0.1/CUDA 11.7, FP16, seed 42, 32,000 warmup updates and a peak learning rate of 0.0005. Each GPU has a maximum audio budget of 882,000 samples per update, with accumulation 1 and crops capped at 20 seconds. Validation and checkpointing occur every 5,000 updates, at epoch boundaries and at the end of training.

The final native Fairseq validation loss is 0.950 at 400,000 updates. The best observed validation loss is 0.943 at 390,000 updates. This repository publishes the final 400,000-update checkpoint. These self-supervised losses do not measure downstream classification or detection performance.

The OpenWhistle supervised datasets are available for subsequent fine-tuning and evaluation:

• OpenWhistle-Classification-Finetuning
• OpenWhistle-Detection-Finetuning

Intended Use

Use the backbone to extract audio representations or fine-tune a downstream model for bottlenose dolphin whistle classification and detection. This checkpoint has no supervised classification or detection head.

Input Audio

Provide mono waveforms sampled at 44,100 Hz. The feature extractor normalizes the waveform and uses right padding with value 0.0 and return_attention_mask=False.

For comparisons with the unpadded native encoder, process waveforms individually or batch equal-length waveforms. Resample audio recorded at other sampling rates before processing.

Loading

import torch
from transformers import AutoFeatureExtractor, AutoModel

repo_id = "dolphinteam/OpenWhistle-JeanZay"
processor = AutoFeatureExtractor.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id).eval()

# waveform is a one-dimensional mono array sampled at 44,100 Hz.
inputs = processor(waveform, sampling_rate=44100, return_tensors="pt")
with torch.no_grad():
    embeddings = model(**inputs).last_hidden_state

To load the quantizer and projection heads as well:

from transformers import Wav2Vec2ForPreTraining
model = Wav2Vec2ForPreTraining.from_pretrained(repo_id)

Transformers pretraining uses its own masking and loss implementation. The original training recipe and metrics come from native Fairseq.

Files

• config.json: model architecture and Transformers configuration
• model.safetensors: converted encoder and pretraining-head weights
• preprocessor_config.json: audio preprocessing settings
• trainer_state.json: native Fairseq metrics reconstructed in the reference Trainer-state layout
• export_provenance.json: checkpoint identity, tensor mapping and embedding parity
• training_provenance.json: dataset revision, training settings and validation history

The native optimizer, scheduler and RNG states are retained in the Jean Zay Fairseq checkpoint and are not included in this inference repository. trainer_state.json is a metrics record, not a Transformers training-resume artifact; training FLOPs were not tracked.

Limitations

• Training focuses on bottlenose dolphin whistles; transfer to other species or recording settings requires evaluation.
• Downstream results for this checkpoint have not yet been established.
• Biological interpretations of embeddings require independent validation.

License

No model-weight license has been specified, following the reference repository. Contact the OpenWhistle maintainers for licensing terms.

Paper, collection, and authors

• Paper: OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations
• Hugging Face collection: NeurIPS 2026 OpenWhistle
• This repository: https://huggingface.co/dolphinteam/OpenWhistle-JeanZay

Authors: Faadil Mustun, Chiara Semenzin, Roberto Dessi, Pablo Robin Guerrero, Pierre Orhan, Alexis Emanuelli, Emanuele Rossi, Yair Lakretz, Gonzalo de Polavieja, and German Sumbre.

Citation

@misc{mustun2026openwhistle,
  title         = {OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations},
  author        = {Mustun, Faadil and Semenzin, Chiara and Dessi, Roberto and Robin Guerrero, Pablo and Orhan, Pierre and Emanuelli, Alexis and Rossi, Emanuele and Lakretz, Yair and de Polavieja, Gonzalo and Sumbre, German},
  year          = {2026},
  eprint        = {2609.34839},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.34839}
}