Model Card for SpeechLMM-v2.0-L-7B
SpeechLMM-v2.0-L-7B is a 7B instruction-tuned multimodal language model from the Meetween SpeechLMM 2.0 family, fine-tuned from Qwen2.5-Omni-7B for speech-centric virtual meetings. The SpeechLMM 2.0 family supports text, speech and video as input, and text and speech as output. The SpeechLMM 2.0 family is fine-tuned on a plethora of understanding and generation tasks, including: automatic speech recognition, speech and machine translation, spoken and text question answering, speech and text summarisation, spoken language understanding, audio chaptering, audio-visual speaker diarisation, and lip-reading, covering ten EU languages (Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, Swedish).
Model Details
Model Description
SpeechLMM-v2.0-L-7B is the 7B member of SpeechLMM 2.0, Meetween's second-generation multimodal foundation model for speech-centric virtual meetings. It is an auto-regressive instruction-tuned model built on Qwen2.5-Omni-7B, extended with a dedicated AutoAVSR visual-speech (lip-reading) pathway and trainable adapters that project audio, video, and lip-reading features into the Thinker representation space.
SpeechLMM 2.0 models take multilingual text, speech, and video as input and can produce text or speech. They are fine-tuned for meeting-oriented understanding and generation tasks, including automatic speech recognition, speech and machine translation, spoken and text question answering, speech and text summarisation, spoken language understanding (intent), audio chaptering, audio-visual speaker diarisation, lip-reading, text-to-speech, and speech-to-speech translation. Official support covers ten EU languages: Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, and Swedish.
This checkpoint follows a multi-stage training recipe: a lip-reading adapter stage, an audio-adapter stage, then multitask LoRA on the audio and video pathways as well as the language model.
- Developed by: Nicola Bertoldi, Lorenzo Pratesi, Thomas Passera, Marco Comunità (Translated); Giuseppe Tanzi, Mubashir Shah (Pi School); Francesco Giuliari (Fondazione Bruno Kessler)
- Funded by: Meetween (Horizon Europe, grant agreement No. 101135798, call HORIZON-CL4-2023-HUMAN-01-CNECT)
- Shared by: Meetween consortium
- Model type: Instruction-tuned multimodal large language model (text, speech, and video)
- Language(s): Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, Swedish
- License: CC-BY-NC-SA 4.0
- Finetuned from model: Qwen/Qwen2.5-Omni-7B
Model Sources
Uses
Direct Use
The model can be used as released, without further fine-tuning, as an instruction-tuned multimodal LLM for speech-centric meeting and speech–language tasks. Intended direct users are researchers and developers who prompt the model with speech, text, and/or video and read back text (or speech) for a single task at a time.
Typical direct uses include:
- transcribing speech (ASR) and translating speech or text (ST, MT) among the ten supported EU languages
- answering questions about spoken or written content (SQA, TQA) and classifying spoken intent (SLU-I)
- summarising speech or meeting text (SSUM, TSUM) and segmenting long audio into chapters (ACHAP)
- audio-visual speaker diarisation (AVSPEAKD) and lip-reading / visual speech recognition (LIPREAD)
- speech generation tasks supported by the SpeechLMM 2.0 family (TTS, S2ST)
Inference instructions are in the SpeechLMM inference repository. People whose meetings, recordings, or video are passed to the model are affected users: do not run it on personal data without an appropriate legal basis and consent.
Downstream Use
The intended downstream use is as a building block in a larger meeting-assistance system—such as Meetween's AI mediator—rather than as a stand-alone chat app. In that setting the model can be wired to a meeting client or batch pipeline to provide transcription, translation, minuting, chaptering, spoken QA, intent detection, diarisation, or lip-reading when audio is degraded.
Researchers and product teams may also further fine-tune (for example with LoRA) on in-domain meetings, additional languages, or a subset of tasks, using the SpeechLMM v2 training repository, and serve the result through the inference stack. Downstream deployers remain responsible for access control, logging, and how outputs are shown to meeting participants.
Training Details
Training Data
| TASK |
Dataset |
Language |
License |
| ACHAP |
AMI Meeting Corpus |
en → en |
CC-BY-4.0 |
|
YTSeg |
en → en |
CC-BY-NC-SA-4.0 |
| ASR |
FLEURS |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
CC-BY-4.0 |
|
Multilingual Spoken Topical-Chat |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
CDLA-Sharing-1.0 |
|
Spoken DGT-TM |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
EUPL |
|
VoxPopuli |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
CC0-1.0 |
|
EuroSpeech |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
Mixed Non-commercial |
| AVSPEAKD |
AMI Meeting Corpus |
en → en |
CC-BY-4.0 |
| LIPREAD |
LipCrops |
en → en |
- |
| MT |
Europarl-ST |
{de, en, es, fr, it, nl, pt} → {de, en, es, fr, it, nl, pt} |
CC-BY-NC-4.0 |
|
Multilingual Spoken Topical-Chat |
{cs, de, en, es, fr, hu, it, nl, pt, sv} → {cs, de, en, es, fr, hu, it, nl, pt, sv} |
CDLA-Sharing-1.0 |
|
Spoken DGT-TM |
{cs, de, en, es, fr, hu, it, nl, pt, sv} → {cs, de, en, es, fr, hu, it, nl, pt, sv} |
EUPL |
| SLU-I |
SLURP |
en → en |
CC-BY-NC-4.0 |
|
Speech-MASSIVE |
de → de, fr → fr |
CC-BY-NC-SA-4.0 |
| SQA-E |
Multilingual Spoken SQuAD |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
CC-BY-SA-4.0 |
| SSUM |
ICSI |
en → en |
CC-BY-4.0 |
| ST |
Europarl-ST |
{de, en, es, fr, it, nl, pt} → {de, en, es, fr, it, nl, pt} |
CC-BY-NC-4.0 |
|
Multilingual Spoken Topical-Chat |
{cs, de, en, es, fr, hu, it, nl, pt, sv} → {cs, de, en, es, fr, hu, it, nl, pt, sv} |
CDLA-Sharing-1.0 |
|
Spoken DGT-TM |
{cs, de, en, es, fr, hu, it, nl, pt, sv} → {cs, de, en, es, fr, hu, it, nl, pt, sv} |
EUPL |
| TQA-E |
Multilingual Spoken SQuAD |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
CC-BY-SA-4.0 |
| TQA-M |
Multilingual Text LibriSQA |
cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv |
- |
| TSUM |
AMI Meeting Corpus |
en → en |
CC-BY-4.0 |
|
ELITR Minuting Corpus |
cs → cs, en → en |
CC-BY-NC-SA-4.0 |
Training Procedure
Training is organized in four stages. Each stage starts from the previous checkpoint.
Stage 0A — Lip-reading adapter. Only the lip-reading adapter is trained, on English lip-crop video, with the lip-reading encoder frozen. This aligns visual speech features with the language model.
Stage 0B — Audio adapter. The audio adapter is fully fine-tuned on all audio tasks and languages. The audio encoder and language model stay frozen. This aligns audio speech features with the language model for the relevant domains and languages.
Stage 1 — Multitask LoRA. The main run continues full fine-tuning of the audio adapter and applies LoRA to the audio encoder and language model. Training covers all text, audio and video tasks, including audio-visual speaker diarisation. Lip-reading data is left out; the video path, lip-reading path, and Talker stay frozen.
Stage 2 — Talker LoRA. LoRA is applied to the Talker for text-to-speech and speech-to-speech translation. Encoders, adapters, and the language model stay frozen.
Evaluation
Results
ACHAP
| ICSI |
Source Language |
Target Language |
Metric |
|
en |
en |
|
ASR
| ACL6060 |
Source Language |
Target Language |
WER ↓ |
CER ↓ |
| Qwen2.5-Omni-7B |
en |
en |
12.40 |
7.02 |
| SpeechLMM-v2.0-L-7B |
en |
en |
13.77 |
7.39 |
| COVOST |
Source Language |
Target Language |
WER ↓ |
CER ↓ |
| Qwen2.5-Omni-7B |
de |
de |
6.01 |
2.48 |
| SpeechLMM-v2.0-L-7B |
de |
de |
5.32 |
2.13 |
| Qwen2.5-Omni-7B |
en |
en |
7.35 |
3.86 |
| SpeechLMM-v2.0-L-7B |
en |
en |
7.66 |
3.84 |
| Qwen2.5-Omni-7B |
es |
es |
4.69 |
2.05 |
| SpeechLMM-v2.0-L-7B |
es |
es |
3.92 |
1.54 |
| Qwen2.5-Omni-7B |
fr |
fr |
7.23 |
2.43 |
| SpeechLMM-v2.0-L-7B |
fr |
fr |
7.19 |
2.53 |
| Qwen2.5-Omni-7B |
it |
it |
4.55 |
1.55 |
| SpeechLMM-v2.0-L-7B |
it |
it |
4.51 |
1.38 |
| FLEURS |
Source Language |
Target Language |
WER ↓ |
CER ↓ |
| Qwen2.5-Omni-7B |
cs |
cs |
97.95 |
70.76 |
| SpeechLMM-v2.0-L-7B |
cs |
cs |
25.79 |
11.23 |
| Qwen2.5-Omni-7B |
de |
de |
6.68 |
3.85 |
| SpeechLMM-v2.0-L-7B |
de |
de |
4.99 |
1.95 |
| Qwen2.5-Omni-7B |
en |
en |
4.66 |
2.17 |
| SpeechLMM-v2.0-L-7B |
en |
en |
4.33 |
1.82 |
| Qwen2.5-Omni-7B |
es |
es |
5.44 |
2.95 |
| SpeechLMM-v2.0-L-7B |
es |
es |
3.11 |
1.27 |
| Qwen2.5-Omni-7B |
fr |
fr |
10.49 |
4.54 |
| SpeechLMM-v2.0-L-7B |
fr |
fr |
7.74 |
2.49 |
| Qwen2.5-Omni-7B |
hu |
hu |
249.65 |
222.32 |
| SpeechLMM-v2.0-L-7B |
hu |
hu |
40.42 |
20.74 |
| Qwen2.5-Omni-7B |
it |
it |
4.69 |
2.45 |
| SpeechLMM-v2.0-L-7B |
it |
it |
3.27 |
1.18 |
| Qwen2.5-Omni-7B |
nl |
nl |
11.06 |
5.09 |
| SpeechLMM-v2.0-L-7B |
nl |
nl |
5.80 |
2.21 |
| Qwen2.5-Omni-7B |
pt |
pt |
7.65 |
4.07 |
| SpeechLMM-v2.0-L-7B |
pt |
pt |
3.55 |
1.39 |
| Qwen2.5-Omni-7B |
sv |
sv |
88.11 |
56.23 |
| SpeechLMM-v2.0-L-7B |
sv |
sv |
19.52 |
8.04 |
| MLS |
Source Language |
Target Language |
WER ↓ |
CER ↓ |
| Qwen2.5-Omni-7B |
de |
de |
7.51 |
3.03 |
| SpeechLMM-v2.0-L-7B |
de |
de |
10.88 |
5.98 |
| Qwen2.5-Omni-7B |
en |
en |
5.72 |
2.85 |
| SpeechLMM-v2.0-L-7B |
en |
en |
6.41 |
3.21 |
| Qwen2.5-Omni-7B |
es |
es |
5.40 |
2.38 |
| SpeechLMM-v2.0-L-7B |
en |
en |
8.05 |
4.36 |
| Qwen2.5-Omni-7B |
fr |
fr |
5.16 |
2.26 |
| SpeechLMM-v2.0-L-7B |
fr |
fr |
6.84 |
3.59 |
| Qwen2.5-Omni-7B |
it |
it |
12.58 |
3.15 |
| SpeechLMM-v2.0-L-7B |
it |
it |
18.79 |
9.28 |
| Qwen2.5-Omni-7B |
nl |
nl |
12.26 |
4.12 |
| SpeechLMM-v2.0-L-7B |
nl |
nl |
11.32 |
4.39 |
| Qwen2.5-Omni-7B |
pt |
pt |
12.44 |
5.98 |
| SpeechLMM-v2.0-L-7B |
pt |
pt |
19.76 |
13.93 |
AVSPEAKD
| VOXMM |
Source Language |
Target Language |
Metric |
|
en |
en |
|
LIPREAD
| LRS3 |
Source Language |
Target Language |
Metric |
|
en |
en |
|
| VOXMM |
Source Language |
Target Language |
Metric |
|
en |
en |
|
MT
| ACL6060 |
Source Language |
Target Language |
BLEU ↑ |
BLEURT ↑ |
COMET ↑ |
| Qwen2.5-Omni-7B |
en |
de |
34.88 |
0.73 |
0.83 |
| SpeechLMM-v2.0-L-7B |
|
de |
37.17 |
0.75 |
0.84 |
| Qwen2.5-Omni-7B |
|
fr |
53.19 |
0.74 |
0.84 |
| SpeechLMM-v2.0-L-7B |
|
fr |
39.30 |
0.70 |
0.83 |
| Qwen2.5-Omni-7B |
|
nl |
43.22 |
0.77 |
0.86 |
| SpeechLMM-v2.0-L-7B |
|
nl |
33.97 |
0.75 |
0.85 |
| Qwen2.5-Omni-7B |
|
pt |
54.27 |
0.78 |
0.88 |
| SpeechLMM-v2.0-L-7B |
|
pt |
42.72 |
0.75 |
0.86 |
| FLORESPLUS |
Source Language |
Target Language |
BLEU ↑ |
BLEURT ↑ |
COMET ↑ |
| Qwen2.5-Omni-7B |
en |
cs |
21.75 |
0.74 |
0.86 |
| SpeechLMM-v2.0-L-7B |
|
cs |
20.73 |
0.74 |
0.85 |
| Qwen2.5-Omni-7B |
|
de |
34.33 |
0.76 |
0.87 |
| SpeechLMM-v2.0-L-7B |
|
de |
35.00 |
0.77 |
0.87 |
| Qwen2.5-Omni-7B |
|
es |
25.13 |
0.74 |
0.86 |
| SpeechLMM-v2.0-L-7B |
|
es |
25.23 |
0.74 |
0.85 |
| Qwen2.5-Omni-7B |
|
fr |
44.65 |
0.75 |
0.88 |
| SpeechLMM-v2.0-L-7B |
|
fr |
38.61 |
0.73 |
0.86 |
| Qwen2.5-Omni-7B |
|
hu |
11.93 |
0.74 |
0.77 |
| SpeechLMM-v2.0-L-7B |
|
hu |
11.39 |
0.76 |
0.78 |
| Qwen2.5-Omni-7B |
|
it |
25.62 |
0.76 |
0.87 |
| SpeechLMM-v2.0-L-7B |
|
it |
24.83 |
0.76 |
0.87 |
| Qwen2.5-Omni-7B |
|
nl |
23.23 |
0.74 |
0.86 |
| SpeechLMM-v2.0-L-7B |
|
nl |
23.32 |
0.74 |
0.86 |
| Qwen2.5-Omni-7B |
|
pt |
46.13 |
0.77 |
0.89 |
| SpeechLMM-v2.0-L-7B |
|
pt |
35.66 |
0.75 |
0.88 |
| Qwen2.5-Omni-7B |
|
sv |
30.70 |
0.74 |
0.86 |
| SpeechLMM-v2.0-L-7B |
|
sv |
27.67 |
0.76 |
0.87 |
| Qwen2.5-Omni-7B |
cs |
en |
38.43 |
0.78 |
0.88 |
| SpeechLMM-v2.0-L-7B |
cs |
|
34.97 |
0.77 |
0.87 |
| Qwen2.5-Omni-7B |
de |
|
43.94 |
0.80 |
0.89 |
| SpeechLMM-v2.0-L-7B |
de |
|
40.14 |
0.79 |
0.89 |
| Qwen2.5-Omni-7B |
es |
|
30.07 |
0.77 |
0.87 |
| SpeechLMM-v2.0-L-7B |
es |
|
28.31 |
0.77 |
0.87 |
| Qwen2.5-Omni-7B |
fr |
|
41.83 |
0.80 |
0.89 |
| SpeechLMM-v2.0-L-7B |
fr |
|
37.58 |
0.79 |
0.89 |
| Qwen2.5-Omni-7B |
hu |
|
30.17 |
0.75 |
0.87 |
| SpeechLMM-v2.0-L-7B |
hu |
|
27.05 |
0.74 |
0.86 |
| Qwen2.5-Omni-7B |
it |
|
31.89 |
0.77 |
0.88 |
| SpeechLMM-v2.0-L-7B |
it |
|
30.06 |
0.77 |
0.87 |
| Qwen2.5-Omni-7B |
nl |
|
30.58 |
0.77 |
0.87 |
| SpeechLMM-v2.0-L-7B |
nl |
|
28.77 |
0.76 |
0.87 |
| Qwen2.5-Omni-7B |
pt |
|
47.18 |
0.80 |
0.90 |
| SpeechLMM-v2.0-L-7B |
pt |
|
42.28 |
0.79 |
0.89 |
| Qwen2.5-Omni-7B |
sv |
|
45.97 |
0.79 |
0.89 |
| SpeechLMM-v2.0-L-7B |
sv |
|
41.64 |
0.78 |
0.89 |
SLU-I
| SLURP |
Source Language |
Target Language |
Intent accuracy ↑ |
| Qwen2.5-Omni-7B |
en |
en |
49.60 |
| SpeechLMM-v2.0-L-7B |
en |
en |
78.80 |
| SPEECHMASSIVE |
Source Language |
Target Language |
Intent accuracy ↑ |
| Qwen2.5-Omni-7B |
de |
de |
50.40 |
| SpeechLMM-v2.0-L-7B |
de |
de |
81.20 |
| Qwen2.5-Omni-7B |
es |
es |
56.00 |
| SpeechLMM-v2.0-L-7B |
es |
es |
85.20 |
| Qwen2.5-Omni-7B |
fr |
fr |
53.60 |
| SpeechLMM-v2.0-L-7B |
fr |
fr |
84.80 |
| Qwen2.5-Omni-7B |
hu |
hu |
5.20 |
| SpeechLMM-v2.0-L-7B |
hu |
hu |
49.60 |
| Qwen2.5-Omni-7B |
nl |
nl |
52.40 |
| SpeechLMM-v2.0-L-7B |
nl |
nl |
84.80 |
| Qwen2.5-Omni-7B |
pt |
pt |
48.00 |
| SpeechLMM-v2.0-L-7B |
pt |
pt |
79.20 |
SQA-E
| SPOKENSQUAD |
Source Language |
Target Language |
Exact Match ↑ |
Token F1 ↑ |
| Qwen2.5-Omni-7B |
en |
en |
60.80 |
74.10 |
| SpeechLMM-v2.0-L-7B |
en |
en |
79.20 |
86.85 |
SQA-A
| LIBRISQA |
Source Language |
Target Language |
Metric |
|
en |
en |
|
SSUM
| ICSI |
Source Language |
Target Language |
R-1 F1 ↑ |
R-2 F1 ↑ |
R-L F1 ↑ |
|
en |
en |
|
|
|
ST
| ACL6060 |
Source Language |
Target Language |
BLEU ↑ |
BLEURT ↑ |
COMET ↑ |
| Qwen2.5-Omni-7B |
en |
de |
32.30 |
0.67 |
0.77 |
| SpeechLMM-v2.0-L-7B |
|
de |
29.59 |
0.69 |
0.78 |
| Qwen2.5-Omni-7B |
|
fr |
41.27 |
0.63 |
0.77 |
| SpeechLMM-v2.0-L-7B |
|
fr |
35.96 |
0.64 |
0.77 |
| Qwen2.5-Omni-7B |
|
nl |
28.54 |
0.67 |
0.78 |
| SpeechLMM-v2.0-L-7B |
|
nl |
27.84 |
0.69 |
0.79 |
| Qwen2.5-Omni-7B |
|
pt |
35.86 |
0.66 |
0.80 |
| SpeechLMM-v2.0-L-7B |
|
pt |
37.00 |
0.68 |
0.80 |
| COVOST |
Source Language |
Target Language |
BLEU ↑ |
BLEURT ↑ |
COMET ↑ |
| Qwen2.5-Omni-7B |
en |
de |
31.83 |
0.71 |
0.84 |
| SpeechLMM-v2.0-L-7B |
|
de |
32.26 |
0.71 |
0.83 |
| Qwen2.5-Omni-7B |
de |
en |
39.91 |
0.75 |
0.80 |
| SpeechLMM-v2.0-L-7B |
de |
|
38.85 |
0.74 |
0.80 |
| Qwen2.5-Omni-7B |
es |
|
43.66 |
0.76 |
0.84 |
| SpeechLMM-v2.0-L-7B |
es |
|
41.49 |
0.75 |
0.84 |
| Qwen2.5-Omni-7B |
fr |
|
40.12 |
0.74 |
0.83 |
| SpeechLMM-v2.0-L-7B |
fr |
|
40.23 |
0.74 |
0.83 |
| Qwen2.5-Omni-7B |
it |
|
37.88 |
0.74 |
0.83 |
| SpeechLMM-v2.0-L-7B |
it |
|
36.25 |
0.73 |
0.83 |
| FLEURSPLUS |
Source Language |
Target Language |
BLEU ↑ |
BLEURT ↑ |
COMET ↑ |
| Qwen2.5-Omni-7B |
en |
cs |
15.68 |
0.68 |
0.79 |
| SpeechLMM-v2.0-L-7B |
|
cs |
18.24 |
0.71 |
0.82 |
| Qwen2.5-Omni-7B |
|
de |
29.78 |
0.75 |
0.82 |
| SpeechLMM-v2.0-L-7B |
|
de |
30.82 |
0.75 |
0.82 |
| Qwen2.5-Omni-7B |
|
es |
23.78 |
0.72 |
0.81 |
| SpeechLMM-v2.0-L-7B |
|
es |
24.27 |
0.73 |
0.82 |
| Qwen2.5-Omni-7B |
|
fr |
36.25 |
0.70 |
0.80 |
| SpeechLMM-v2.0-L-7B |
|
fr |
37.72 |
0.73 |
0.82 |
| Qwen2.5-Omni-7B |
|
hu |
7.52 |
0.74 |
0.72 |
| SpeechLMM-v2.0-L-7B |
|
hu |
9.02 |
0.77 |
0.74 |
| Qwen2.5-Omni-7B |
|
it |
22.82 |
0.74 |
0.82 |
| SpeechLMM-v2.0-L-7B |
|
it |
24.57 |
0.75 |
0.82 |
| Qwen2.5-Omni-7B |
|
nl |
21.26 |
0.71 |
0.79 |
| SpeechLMM-v2.0-L-7B |
|
nl |
22.17 |
0.73 |
0.81 |
| Qwen2.5-Omni-7B |
|
pt |
38.41 |
0.74 |
0.84 |
| SpeechLMM-v2.0-L-7B |
|
pt |
34.30 |
0.74 |
0.84 |
| Qwen2.5-Omni-7B |
|
sv |
27.57 |
0.69 |
0.79 |
| SpeechLMM-v2.0-L-7B |
|
sv |
29.43 |
0.73 |
0.81 |
| Qwen2.5-Omni-7B |
cs |
en |
4.25 |
0.41 |
0.57 |
| SpeechLMM-v2.0-L-7B |
cs |
|
21.90 |
0.66 |
0.76 |
| Qwen2.5-Omni-7B |
de |
|
37.42 |
0.78 |
0.84 |
| SpeechLMM-v2.0-L-7B |
de |
|
38.19 |
0.78 |
0.84 |
| Qwen2.5-Omni-7B |
es |
|
25.95 |
0.75 |
0.82 |
| SpeechLMM-v2.0-L-7B |
es |
|
28.32 |
0.76 |
0.83 |
| Qwen2.5-Omni-7B |
fr |
|
35.78 |
0.77 |
0.84 |
| SpeechLMM-v2.0-L-7B |
fr |
|
35.51 |
0.77 |
0.84 |
| Qwen2.5-Omni-7B |
hu |
|
0.34 |
0.28 |
0.45 |
| SpeechLMM-v2.0-L-7B |
hu |
|
12.46 |
0.51 |
0.66 |
| Qwen2.5-Omni-7B |
it |
|
27.99 |
0.76 |
0.83 |
| SpeechLMM-v2.0-L-7B |
it |
|
29.69 |
0.76 |
0.83 |
| Qwen2.5-Omni-7B |
nl |
|
27.16 |
0.74 |
0.82 |
| SpeechLMM-v2.0-L-7B |
nl |
|
28.28 |
0.74 |
0.82 |
| Qwen2.5-Omni-7B |
pt |
|
42.56 |
0.78 |
0.84 |
| SpeechLMM-v2.0-L-7B |
pt |
|
42.12 |
0.78 |
0.85 |
| Qwen2.5-Omni-7B |
sv |
|
6.88 |
0.38 |
0.54 |
| SpeechLMM-v2.0-L-7B |
sv |
|
29.31 |
0.68 |
0.78 |
S2ST
| CVSS |
Source Language |
Target Language |
BLEU ↑ |
|
de |
en |
|
|
es |
|
|
|
fr |
|
|
|
it |
|
|
|
nl |
|
|
|
pt |
|
|
|
sv |
|
|
| FLEURSPLUS |
Source Language |
Target Language |
BLEU ↑ |
|
en |
cs |
|
|
|
de |
|
|
|
es |
|
|
|
fr |
|
|
|
hu |
|
|
|
it |
|
|
|
nl |
|
|
|
pt |
|
|
|
sv |
|
|
cs |
en |
|
|
de |
|
|
|
es |
|
|
|
fr |
|
|
|
hu |
|
|
|
it |
|
|
|
nl |
|
|
|
pt |
|
|
|
sv |
|
|
TQA-A
TQA-E
TQA-M
TSUM
| AUTOMIN |
Source Language |
Target Language |
R-1 F1 ↑ |
R-2 F1 ↑ |
R-L F1 ↑ |
|
cs |
cs |
|
|
|
|
en |
en |
|
|
|
| ICSI |
Source Language |
Target Language |
R-1 F1 ↑ |
R-2 F1 ↑ |
R-L F1 ↑ |
| Qwen2.5-Omni-7B |
en |
en |
18.77 |
3.25 |
16.82 |
| SpeechLMM-v2.0-L-7B |
en |
en |
22.50 |
3.65 |
19.95 |
TTS
| LJSPEECH |
Source Language |
Target Language |
Metric |
|
en |
en |
|
Technical Specifications
Model Architecture
SpeechLMM-v2.0-L-7B uses Qwen2.5-Omni-7B as the multimodal backbone. Qwen2.5-Omni is a Thinker–Talker model that unifies perception and generation across text, image, audio, and video:
- Thinker handles multimodal reasoning and text generation (~7B parameters).
- Talker produces streaming speech tokens conditioned on hidden states from the Thinker (used for TTS and speech-to-speech translation).
- The audio encoder is initialized from Whisper-large-v3 and processes audio in blocks, so long inputs can be encoded without rebuilding alignment to the language model.
- The vision encoder is the ViT from Qwen2.5-VL. In SpeechLMM it is used for general video (for example audio-visual speaker diarisation), not for lip-reading.
The backbone already covers a wide set of written and spoken languages. SpeechLMM still trains for ten EU languages, including Czech, Hungarian, and Swedish. Updating the audio encoder and adapter (Stages 0B and 1) is how those languages are brought into the speech-understanding path.
On top of Qwen2.5-Omni, SpeechLMM v2 adds the pieces needed for meeting scenarios:
| Component |
Role |
| Backbone |
Qwen2.5-Omni-7B (Thinker + Talker + Whisper-based audio encoder + Qwen2.5-VL vision encoder) |
| Lip-reading encoder |
AutoAVSR for visual speech / lip crops |
| Adapters |
Trainable projections from audio, video, and lip-read encoder outputs into Thinker space |
The dedicated AutoAVSR pathway is the main architectural addition relative to Qwen2.5-Omni: it supplies visual speech features when audio is degraded or absent, separate from the Qwen2.5-VL vision tower used for general video.
