mc_wseg_cop_bro

mc_wseg_cop_bro is an unconditioned MC-WhisperSeg model for multi-channel zebra-finch vocal-segment detection and quality-label prediction. It predicts onset, cluster, and offset tokens from three synchronized audio channels ordered as:

1. target bird;
2. other bird;
3. shared microphone.

This repository contains the fixed-duration step-4,828 Hugging Face checkpoint. It is the baseline model only; it does not contain the experimental FDDT/oracle conditioning models.

Model details

• Architecture: Whisper-large-sized encoder-decoder
• Parameters: 1,542,024,960
• Encoder/decoder layers: 32 / 32
• Hidden size: 1,280
• Input representation: 80-bin, three-channel MC-WhisperSeg features
• Output: onset / quality-cluster / offset token sequences
• Species token: zebra finch
• Quality classes: clean, consolidated, crosstalk, orphan, unpair, plus supported uncertain variants

The model uses a custom MC-WhisperSeg feature layout and decoding procedure. It is not compatible with the standard Hugging Face ASR pipeline without the MC-WhisperSeg preprocessing and segmentation code.

A CTranslate2 export of the same step-4,828 checkpoint is included in the ct2_model/ directory.

Training data and protocol

The model was fine-tuned on 17 recording-disjoint stage02 and stage04 COP/BRO recording sessions using split seed 66100. The two original validation recordings remained completely excluded from gradient updates and were used only for the held-out evaluation reported below.

For stage02, paired target views were repaired so both use the canonical [target, other, microphone] channel order. Annotation seconds were rebuilt from the authoritative onsetBin and offsetBin columns using bin * hopLength / samplingRate.

Training used four fixed epochs (4,828 optimizer steps), batch size 4, learning rate 3e-6, 200 warm-up steps, linear decay, and weight decay 0.01. No evaluation or checkpoint selection occurred during training; the final step-4,828 checkpoint was frozen before the held-out validation evaluation.

Held-out validation results

All segment results use a ±10 ms onset/offset tolerance.

Metric	F1
Segment, cluster ignored	0.9078
Segment, cluster required	0.8708
Frame activity, cluster ignored	0.9646
Frame activity, cluster required	0.8850

These results are from the two original held-out validation recordings.

Usage

Download the checkpoint with huggingface_hub:

from huggingface_hub import snapshot_download

model_path = snapshot_download("songbirdini/mc_wseg_cop_bro")

Then load model_path with the MC-WhisperSeg WhisperSegmenterForEval implementation and provide a NumPy audio array shaped [3, samples] in the exact [target, other, microphone] order. The model configuration and tokenizer are included in this repository.

To load the included CTranslate2 export:

from pathlib import Path

import ctranslate2

ct2_path = Path(model_path) / "ct2_model"
ct2_model = ctranslate2.models.Whisper(str(ct2_path), device="cpu")

The same custom MC-WhisperSeg feature construction and segmentation decoding are required when using the CTranslate2 model.

Annotation reference

The activity and quality-label policy follows the accompanying published annotation guideline: DOI 10.5281/zenodo.20524686.