imprt

/

izanami-wav2vec2-large

like 5

Follow

Intelligent Media Processing Research Team 42

Thank you for releasing the Izanami models.

I believe the Hugging Face (transformers) configs of this repository do not
match the fairseq training configuration, which silently breaks all
transformer-layer features when the model is used via transformers as shown
in the README usage example.

Shipped HF configs (wav2vec2-base-style)

• config.json: "do_stable_layer_norm": false
• preprocessor_config.json: "do_normalize": false, "return_attention_mask": false

Training configuration embedded in fairseq/izanami-wav2vec2-large.pt

The pickled fairseq cfg inside the checkpoint contains:

• extractor_mode: "layer_norm"
• layer_norm_first: True
• normalize: True (task)

i.e. the standard wav2vec 2.0 large recipe. Note the shipped
config.json is also internally inconsistent: "feat_extract_norm": "layer"
(large-style CNN) combined with "do_stable_layer_norm": false (base-style
encoder) is a combination that does not exist in any fairseq recipe.

Observed impact

Layer-wise linear probe on a Japanese emotional-speech corpus (JVNV, 360
clips, 6-class, speaker-independent CV):

With the shipped configs, every transformer layer performs worse than the
CNN output and accuracy decays monotonically with depth — a symptom of
layer norms being applied in the wrong position. With both overrides the
model shows a normal mid-layer peak, comparable to
imprt/kushinada-hubert-large (whose HF configs are correct:
do_stable_layer_norm: true, do_normalize: true).

Suggested fix

In config.json:

"do_stable_layer_norm": true

In preprocessor_config.json:

"do_normalize": true,
"return_attention_mask": true

(izanami-wav2vec2-base may be unaffected if it was trained with the base
recipe, but it would be worth double-checking it the same way.)

Workaround for users until fixed:

from transformers import AutoFeatureExtractor, AutoModel

extractor = AutoFeatureExtractor.from_pretrained("imprt/izanami-wav2vec2-large")
extractor.do_normalize = True
model = AutoModel.from_pretrained("imprt/izanami-wav2vec2-large", do_stable_layer_norm=True)

Happy to provide the probe code or further details if useful.