Spaces:
Build error
Build error
File size: 2,287 Bytes
31a7207 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
# WhisperFusion
<h2 align="center">
<a href="https://www.youtube.com/watch?v=_PnaP0AQJnk"><img
src="https://img.youtube.com/vi/_PnaP0AQJnk/0.jpg" style="background-color:rgba(0,0,0,0);" height=300 alt="WhisperFusion"></a>
<br><br>Seamless conversations with AI (with ultra-low latency)<br><br>
</h2>
Welcome to WhisperFusion. WhisperFusion builds upon the capabilities of
the [WhisperLive](https://github.com/collabora/WhisperLive) and
[WhisperSpeech](https://github.com/collabora/WhisperSpeech) by
integrating Mistral, a Large Language Model (LLM), on top of the
real-time speech-to-text pipeline. Both LLM and
Whisper are optimized to run efficiently as TensorRT engines, maximizing
performance and real-time processing capabilities. While WhiperSpeech is
optimized with torch.compile.
## Features
- **Real-Time Speech-to-Text**: Utilizes OpenAI WhisperLive to convert
spoken language into text in real-time.
- **Large Language Model Integration**: Adds Mistral, a Large Language
Model, to enhance the understanding and context of the transcribed
text.
- **TensorRT Optimization**: Both LLM and Whisper are optimized to
run as TensorRT engines, ensuring high-performance and low-latency
processing.
- **torch.compile**: WhisperSpeech uses torch.compile to speed up
inference which makes PyTorch code run faster by JIT-compiling PyTorch
code into optimized kernels.
## Getting Started
- We provide a pre-built TensorRT-LLM docker container that has both whisper and
phi converted to TensorRT engines and WhisperSpeech model is pre-downloaded to
quickly start interacting with WhisperFusion.
```bash
docker run --gpus all --shm-size 64G -p 6006:6006 -p 8888:8888 -it ghcr.io/collabora/whisperfusion:latest
```
- Start Web GUI
```bash
cd examples/chatbot/html
python -m http.server
```
## Build Docker Image
- We provide the docker image for cuda-architecures 89 and 90. If you have a GPU
with a different cuda architecture. For e.g. to build for RTX 3090 with cuda-
architecture 86
```bash
bash build.sh 86-real
```
This should build the `ghcr.io/collabora/whisperfusion:latest` for RTX 3090.
## Contact Us
For questions or issues, please open an issue. Contact us at:
marcus.edel@collabora.com, jpc@collabora.com,
vineet.suryan@collabora.com
|