laser-asd 0.1.1
LASER ASD - Lip Landmark Assisted Speaker Detection for Active Speaker Detection
Requires Python: >=3.8
Keywords: active-speaker-detection, speaker-detection, audio-visual, lip-sync, deep-learning, pytorch, computer-vision

# LASER ASD - Lip Landmark Assisted Speaker Detection (PyPI Package)

[![PyPI version](https://badge.fury.io/py/laser-asd.svg)](https://badge.fury.io/py/laser-asd)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

> **Note**: This is a PyPI packaging wrapper for easy installation.
>
> **Original Repository**: https://github.com/plnguyen2908/LASER_ASD
>
> All credit for the LASER ASD model architecture and training goes to the original authors.

This package provides a simple Python API for the LASER ASD (Lip Landmark Assisted Speaker Detection) model, enabling easy installation via `pip` and straightforward inference.

## Original Work

This package is based on the LASER ASD implementation by Phat Lam Nguyen et al.:

- **Original Repository**: https://github.com/plnguyen2908/LASER_ASD
- **Paper**: LASER: Lip Landmark Assisted Speaker Detection

If you use this code, please cite the original work (see [Citation](#citation) section).

## Features

- Active Speaker Detection using audio-visual fusion
- Based on LoCoNet architecture with LASER landmark injection
- Simple Python API for inference
- GPU acceleration with CUDA support
- Compatible with PyTorch 1.10+

## Installation

```bash
pip install laser-asd
```

## Quick Start

```python
import numpy as np
from laser_asd import LaserASDModel

# Initialize model
model = LaserASDModel(device="cuda")

# Load pre-trained weights (download from original repo)
model.load_weights("/path/to/loconet_laser.model")

# Prepare inputs
# face_crops: numpy array of shape [T, H, W, C] or [T, H, W]
# audio_data: numpy array of shape [samples] at 16kHz
face_crops = np.random.rand(100, 112, 112, 3).astype(np.float32)
audio_data = np.random.rand(64000).astype(np.float32)  # 4 seconds at 16kHz

# Predict speaking scores
scores = model.predict(face_crops, audio_data, sample_rate=16000, fps=25.0)

# scores >= 0 indicates speaking
is_speaking = scores >= 0
```

## Model Weights

Pre-trained model weights must be downloaded from the original repository:

- **Download**: https://github.com/plnguyen2908/LASER_ASD

Set the model path via environment variable:
```bash
export LASER_ASD_MODEL_PATH=/path/to/loconet_laser.model
```

Or pass it directly:
```python
model = LaserASDModel(device="cuda", model_path="/path/to/loconet_laser.model")
```

## API Reference

### LaserASDModel

```python
class LaserASDModel:
    def __init__(
        self,
        device: str = "cuda",
        model_path: Optional[Path] = None,
        use_landmarks: bool = False,
    ):
        """
        Initialize LASER ASD model.

        Args:
            device: Device to run model on ('cuda' or 'cpu')
            model_path: Path to model weights file
            use_landmarks: Whether to use landmark features (False uses zeros)
        """

    def load_weights(self, model_path: Optional[str] = None):
        """Load model weights."""

    def predict(
        self,
        face_crops: np.ndarray,
        audio_data: np.ndarray,
        sample_rate: int = 16000,
        fps: float = 25.0,
    ) -> np.ndarray:
        """
        Predict speaking probability for each frame.

        Args:
            face_crops: Face crop images [T, H, W, C] or [T, H, W]
            audio_data: Audio waveform [samples]
            sample_rate: Audio sample rate (default: 16000)
            fps: Video frame rate (default: 25.0)

        Returns:
            Per-frame speaking scores [T] (>= 0 means speaking)
        """
```

### Factory Function

```python
def create_laser_model(
    device: str = "cuda",
    model_path: Optional[str] = None,
    **kwargs,
) -> LaserASDModel:
    """Factory function to create LASER ASD model."""
```

## Requirements

- Python >= 3.8
- PyTorch >= 1.10.0
- torchvision >= 0.11.0
- numpy >= 1.20.0
- opencv-python >= 4.5.0
- resampy >= 0.4.0
- python_speech_features >= 0.6

## Citation

If you use this code, please cite the original LASER ASD paper:

```bibtex
@inproceedings{nguyen2024laser,
  title={LASER: Lip Landmark Assisted Speaker Detection},
  author={Nguyen, Phat Lam and others},
  booktitle={Proceedings},
  year={2024}
}
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

The original LASER ASD code is also MIT licensed.

## Acknowledgments

- **Original LASER ASD**: https://github.com/plnguyen2908/LASER_ASD (Phat Lam Nguyen et al.)
- LoCoNet encoder architecture
- TalkNet audio-visual framework

## Differences from Original Repository

This PyPI package adds:
- `wrapper.py`: Simple inference API (`LaserASDModel` class)
- `pyproject.toml`: PyPI packaging configuration
- Modified imports for package compatibility

The core model architecture and weights are unchanged from the original.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

For issues related to the model architecture, please refer to the [original repository](https://github.com/plnguyen2908/LASER_ASD).