fanwufan/AuscuQA

AuscuTSLM — Patient-Level Multimodal QA from Multi-Site Auscultation Recordings

This repository hosts the final checkpoint corresponding to the results reported in "AuscuTSLM: Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings" (ETH Zurich, TU Eindhoven, University of St. Gallen, Stanford University).

AuscuTSLM aligns multi-site auscultation recordings with a frozen LLM embedding space via Flamingo-style gated cross-attention, enabling open-ended and binary clinical question answering grounded in cardiac/respiratory audio.

Code: https://github.com/OpenTSLM/AuscuTSLM

Files

• best_model.pt (3.4 GB) — full model state dict: audio encoder, MLP projector, Perceiver Resampler, and gated cross-attention adapters. The backbone LLM (meta-llama/Llama-3.2-1B) is frozen and not duplicated in this file; it is loaded separately from Hugging Face at inference time by the training/eval code.

Architecture / hyperparameters

Component	Value
Audio encoder	RawAudioTokenizer (scratch-trained, 40 ms raw-waveform patches)
Perceiver latents	64
Backbone LLM	meta-llama/Llama-3.2-1B (frozen)
Fusion	Gated cross-attention (Flamingo-style)
Max audio context	30 s
Trainable parameters	~1.4B (encoder + adapters; LLM frozen)

How to use

This checkpoint is meant to be loaded with the model classes from the project's GitHub repo (opentslm.model.encoder.RawAudioTokenizer, opentslm.model.projector.MLPProjector, opentslm.model.llm.AudioFlamingo), for example via the repo's own evaluation script:

hf download fanwufan/AuscuQA best_model.pt --local-dir .

python scripts/evaluate_audio_cot.py \
  --checkpoint best_model.pt \
  --llm_id meta-llama/Llama-3.2-1B \
  --num_latents 64

The checkpoint is a flat state_dict (loadable via model.load_state_dict(torch.load(...), strict=False)); see the repo's evaluate_audio_cot.py for the exact loading logic.

Training data

Trained on CaReSound, a patient-level clinical QA benchmark built from public cardiac and respiratory auscultation datasets (ICBHI, KAUH, CirCor, SPRSound, ZCHSound). Raw audio and source datasets are not redistributed in this repository — see the original dataset sources.

Citation

If you use this model, please cite:

@article{wu2026patient,
  title={Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings},
  author={Wu, Fan and Wang, Tsai-Ning and Zumarraga, Nicolas and Wang, Ning and Kreft, Markus and O'Sullivan, Kevin and Fleisch, Elgar and Aalami, Oliver and Schmiedmayer, Paul and Jakob, Robert and others},
  journal={arXiv preprint arXiv:2603.13362},
  year={2026}
}

License

MIT — see the project repository for the full license text.