Repository: embeddings-benchmark/mteb

Issue #2528: Error with BeijingOpera using wav2vec2
Created: 2025-04-09T17:39:46Z
@RahulSChand @silky1708 @anime-sh
Tagging you here. I tried running your w2v2 base model implementation with the BeijingOpera task and ran into some errors. Can you let me know how to proceed?
Code:
import mteb

model_name = "facebook/wav2vec2-base"
model = mteb.get_model(
    model_name, model_revision="0b5b8e868dd84f03fd87d01f9c4ff0f080fecfe8"
)
tasks = mteb.get_tasks(tasks=["BeijingOpera"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")

Error:
Error while evaluating BeijingOpera: The shape of the mask [1, 658, 768] at index 0 does not match the shape of the indexed tensor [32, 658, 768] at index 0
Traceback (most recent call last):
  File "/juice2/scr2/askhan1/mteb/mteb/run.py", line 21, in <module>
    results = evaluation.run(model, output_folder=f"results/{model_name}")
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/evaluation/MTEB.py", line 673, in run
    raise e
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/evaluation/MTEB.py", line 626, in run
    results, tick, tock = self._run_eval(
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/evaluation/MTEB.py", line 308, in _run_eval
    results = task.evaluate(
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/abstasks/Audio/AbsTaskAudioClassification.py", line 94, in evaluate
    scores[hf_subset] = self._evaluate_subset_cross_validation(
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/abstasks/Audio/AbsTaskAudioClassification.py", line 170, in _evaluate_subset_cross_validation
    scores_exp, test_cache = evaluator(model, test_cache=test_cache)
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/evaluation/evaluators/Audio/ClassificationEvaluator.py", line 111, in __call__
    X_train = model.get_audio_embeddings(
  File "/juice2/scr2/askhan1/mteb/mteb/mteb/models/wav2vec2_models.py", line 182, in get_audio_embeddings
    outputs = self.model(
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 2227, in forward
    outputs = self.wav2vec2(
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1822, in forward
    encoder_outputs = self.encoder(
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nlp/scr/askhan1/miniconda3/envs/mteb/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1023, in forward
    hidden_states[~expand_attention_mask] = 0
IndexError: The shape of the mask [1, 658, 768] at index 0 does not match the shape of the indexed tensor [32, 658, 768] at index 0

Duplicate issue #2553: MAEB Models Issues
Created: 2025-04-16T18:31:56Z
Tensor Shape Issues with Audio Models (wavlm-base, clap-htsat-fused, wav2vec2-base)
While working with several audio models to test my datasets, I encountered a few tensor shape issues across different models. Below are the details:

1. microsoft/wavlm-base
Error:
[item["array"]], IndexError: too many indices for tensor of dimension 2
2. laion/clap-htsat-fused
Error:
[item["array"]], IndexError: too many indices for tensor of dimension 2
3. facebook/wav2vec2-base
This model raised a dimension mismatch error, likely related to the attention_mask.
Fix:
outputs = self.model(
    inputs.input_values.squeeze(0),
    attention_mask=inputs.attention_mask.squeeze(0).unsqueeze(-1),
    output_hidden_states=True,
)

# Squeezing and unsqueezing the attention mask resolved the issue.

Closing comment (2025-04-17T12:08:47Z): Related to #2528