vad-detect 0.1.0
A Python library for detecting non-speech gaps in audio/video files using FSMN-VAD-ONNX with streaming processing
License: MIT
Requires Python: >=3.10

# vad-detect

A Python library for detecting non-speech gaps in audio/video files using FSMN-VAD-ONNX with streaming processing.

## Features

- **Streaming VAD detection**: Process large audio/video files in chunks without loading everything into memory
- **Non-speech gap detection**: Automatically detect and return all non-speech intervals
- **Format support**: Supports all audio/video formats that FFmpeg supports (MP3, WAV, FLAC, Opus, MP4, etc.)
- **Time range support**: Support start time and duration parameters for partial processing
- **Memory efficient**: Constant memory usage regardless of audio file duration

## Installation

```bash
pip install vad-detect
```

**Note**: This package requires:

- FFmpeg to be installed on your system and available in PATH
- FSMN-VAD-ONNX model files (see Model Setup below)

## Model Setup

This package requires FSMN-VAD-ONNX model files. You need to:

1. Obtain the FSMN-VAD model files (including `model_quant.onnx`, `config.yaml`, etc.)
2. Set the `MODEL_FSMN_VAD_DIR` environment variable to point to the model directory:

```bash
export MODEL_FSMN_VAD_DIR=/path/to/fsmn-vad-model
```

Alternatively, you can specify the model directory when initializing `VadDetector`:

```python
from vad_detect import VadDetector

detector = VadDetector(model_dir="/path/to/fsmn-vad-model")
```

## Quick Start

```python
from vad_detect import VadDetector

# Initialize detector (reads MODEL_FSMN_VAD_DIR from environment)
detector = VadDetector()

# Detect non-speech gaps in an audio file
gaps = detector.detect_gaps_streaming("audio.mp3")

# gaps is a list of dictionaries: [{"start_ms": 0, "end_ms": 500}, ...]
for gap in gaps:
    start_ms = gap["start_ms"]
    end_ms = gap["end_ms"]
    duration = end_ms - start_ms
    print(f"Non-speech gap: {start_ms}ms - {end_ms}ms (duration: {duration}ms)")
```

### Processing Specific Time Range

```python
# Process only the first 30 seconds
gaps = detector.detect_gaps_streaming(
    file_path="audio.mp3",
    start_ms=0,
    duration_ms=30000,
)

# Process from 10 seconds, duration 5 seconds
gaps = detector.detect_gaps_streaming(
    file_path="audio.mp3",
    start_ms=10000,
    duration_ms=5000,
)
```

### Custom Chunk Size

```python
# Use 1-minute chunks instead of default 20-minute chunks
gaps = detector.detect_gaps_streaming(
    file_path="audio.mp3",
    chunk_duration_sec=60,
)
```

## API Reference

### VadDetector

Main class for VAD detection. All methods are instance methods.

#### `VadDetector.__init__(model_dir=None)`

Initialize VAD detector.

**Parameters:**

- `model_dir` (str, optional): Path to the FSMN-VAD model directory. If None, reads from `MODEL_FSMN_VAD_DIR` environment variable.

**Note:** The FSMN-VAD model only has a quantized version, so `quantize=True` is always used internally.

**Raises:**

- `VadModelNotFoundError`: If model directory is not found or not set
- `VadModelInitializationError`: If model initialization fails

#### `VadDetector.detect_gaps_streaming(file_path, chunk_duration_sec=None, start_ms=None, duration_ms=None)`

Stream audio file and detect non-speech gaps.

**Parameters:**

- `file_path` (str): Path to the audio/video file (supports all FFmpeg formats)
- `chunk_duration_sec` (int, optional): Duration of each chunk in seconds. Defaults to 1200 (20 minutes). Must be > 0 if provided.
- `start_ms` (int, optional): Start position in milliseconds. None means from file beginning. If None but `duration_ms` is provided, defaults to 0.
- `duration_ms` (int, optional): Total duration to process in milliseconds. None means process until end. If specified, processing stops when this duration is reached.

**Returns:**

- `list[dict]`: Non-speech gap list, format: `[{"start_ms": ms, "end_ms": ms}, ...]`
  - Timestamps are relative to audio start (from 0)
  - Unit: milliseconds

**Raises:**

- `VadProcessingError`: If processing fails

## Exceptions

### `VadModelNotFoundError`

Raised when VAD model directory is not found or not set.

**Attributes:**

- `message`: Human-readable error message

### `VadModelInitializationError`

Raised when VAD model initialization fails.

**Attributes:**

- `message`: Primary error message
- `model_dir`: Path to the model directory that caused the error

### `VadProcessingError`

Raised when VAD processing fails.

**Attributes:**

- `message`: Primary error message
- `file_path`: Path to the file being processed
- `details`: Additional error details dictionary

## Requirements

- Python >= 3.10
- FFmpeg (must be installed separately)
- numpy >= 1.26.4
- funasr-onnx >= 0.4.1
- ffmpeg-audio >= 0.1.2

## License

MIT License