Repository: huggingface/lighteval
Issue: [BUG] Cannot evaluate facebook/seamless-m4t-v2-large
Root cause: Unimplemented Feature Gap

## Describe the bug
I get the following error when I want to evaluate seamless-m4t-v2-large:
```
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ubuntu/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/__main__.py", line 102, in <module>
    cli_evaluate()
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/__main__.py", line 62, in cli_evaluate
    main_accelerate(args)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/logging/hierarchical_logger.py", line 178, in wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/main_accelerate.py", line 78, in main
    pipeline = Pipeline(
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/pipeline.py", line 122, in __init__
    self.model = self._init_model(model_config, model)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/pipeline.py", line 166, in _init_model
    return load_model(config=model_config, env_config=self.pipeline_parameters.env_config)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/models/model_loader.py", line 96, in load_model
    return load_model_with_accelerate_or_default(config=config, env_config=env_config)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/models/model_loader.py", line 158, in load_model_with_accelerate_or_default
    model = BaseModel(config=config, env_config=env_config)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/models/base_model.py", line 85, in __init__
    self.model = self._create_auto_model(config, env_config)
  File "/home/ubuntu/LegalLLMEvaluation/lighteval/src/lighteval/models/base_model.py", line 260, in _create_auto_model
    model = AutoModelForCausalLM.from_pretrained(
  File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 567, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t_v2.configuration_seamless_m4t_v2.SeamlessM4Tv2Config'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, ElectraConfig, ErnieConfig, FalconConfig, FalconMambaConfig, FuyuConfig, GemmaConfig, Gemma2Config, GitConfig, GlmConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GraniteConfig, GraniteMoeConfig, JambaConfig, JetMoeConfig, LlamaConfig, MambaConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MllamaConfig, MoshiConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, OlmoConfig, OlmoeConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, PhimoeConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, ZambaConfig.
```

## To Reproduce
```
lighteval accelerate \
  --model_args pretrained=facebook/seamless-m4t-v2-large,trust_remote_code=True \
     --tasks "leaderboard|truthfulqa:mc|0|0" \
     --override_batch_size 1 \
     --output_dir="./evals/"
```

## Expected behavior
It should load the model and execute it.

What would you suggest to change to make it run?

## Version info
Ubuntu 22, lighteval 0.6.0


Hi!
We don't support multimodal models at the moment, and from what I can see from the doc [this model](https://huggingface.co/docs/transformers/en/model_doc/seamless_m4t) is audio to text - closing for now but it's on the roadmap for next year!