youtube-audio-dataset-collector 0.1.2
A comprehensive tool for downloading, processing, and transcribing audio from YouTube videos and playlists for machine learning datasets
License: MIT
Requires Python: >=3.8

# YouTube Audio Dataset Collector

A comprehensive Python library for downloading, processing, and transcribing audio from YouTube videos and playlists for machine learning datasets.

## Features

- **YouTube Audio Download**: Download audio from single videos or entire playlists using yt-dlp
- **Audio Processing**: Convert audio to 16kHz, 16-bit mono WAV format optimized for ML
- **Intelligent Segmentation**: Segment audio into 12-28 second chunks based on silence detection
- **Multi-API Transcription**: Transcribe audio using Gemini API with multiple key rotation for rate limit avoidance
- **Parallel Processing**: Multi-threaded processing for efficient handling of large datasets
- **CSV Export**: Export results in CSV format with audio_filepath and transcript columns

## Installation

### From PyPI (recommended)

```bash
pip install youtube-audio-dataset-collector
```

### From Source

```bash
git clone https://github.com/yourusername/youtube-audio-dataset-collector.git
cd youtube-audio-dataset-collector
pip install -e .
```

### System Dependencies

The library requires ffmpeg for audio processing:

**Ubuntu/Debian:**
```bash
sudo apt install ffmpeg
```

**macOS:**
```bash
brew install ffmpeg
```

**Windows:**
Download from [https://ffmpeg.org/download.html](https://ffmpeg.org/download.html)

## Quick Start

### Command Line Interface

```bash
# Basic usage
youtube-audio-collector --url "https://www.youtube.com/watch?v=VIDEO_ID" --output ./my_dataset

# With multiple API keys for better rate limits
youtube-audio-collector --url "https://www.youtube.com/playlist?list=PLAYLIST_ID" \
                       --output ./my_dataset \
                       --keys-file gemini_keys.txt \
                       --rotate-keys

# Advanced options
youtube-audio-collector --url "https://www.youtube.com/watch?v=VIDEO_ID" \
                       --output ./my_dataset \
                       --threads 8 \
                       --batch-size 6 \
                       --language en
```

### Python API

```python
from youtube_audio_collector import YouTubeAudioCollector

# Initialize with single API key
collector = YouTubeAudioCollector(
    output_dir="./my_dataset",
    api_key="your_gemini_api_key",
    threads=4
)

# Process a YouTube URL
output_csv = collector.process_url("https://www.youtube.com/watch?v=VIDEO_ID")
print(f"Dataset saved to: {output_csv}")

# Initialize with multiple API keys for better rate limits
collector = YouTubeAudioCollector(
    output_dir="./my_dataset",
    api_keys=["key1", "key2", "key3"],
    threads=8,
    batch_size=6
)

# Process a playlist
output_csv = collector.process_url(
    "https://www.youtube.com/playlist?list=PLAYLIST_ID",
    language="kn"  # Kannada
)
```

### Individual Components

```python
from youtube_audio_collector import (
    YouTubeDownloader, 
    AudioProcessor, 
    TranscriptionService,
    APIKeyManager
)

# Download audio
downloader = YouTubeDownloader(cookies_path="cookies.txt")
audio_files = downloader.download_audio(
    "https://www.youtube.com/watch?v=VIDEO_ID",
    "./downloads"
)

# Process audio
processor = AudioProcessor(sample_rate=16000)
segments = processor.segment_audio(
    audio_files[0],
    "./segments",
    min_length=12000,  # 12 seconds
    max_length=28000   # 28 seconds
)

# Transcribe with multiple API keys
api_manager = APIKeyManager(["key1", "key2", "key3"])
transcriber = TranscriptionService(api_key_manager=api_manager)

for segment in segments:
    transcript = transcriber.transcribe_audio(segment, language="en")
    print(f"{segment}: {transcript}")
```

## Configuration

### Environment Variables

Create a `.env` file in your project directory:

```env
# Single API key
GEMINI_API_KEY=your_gemini_api_key

# Multiple API keys (for rate limit avoidance)
GEMINI_API_KEY_1=your_first_api_key
GEMINI_API_KEY_2=your_second_api_key
GEMINI_API_KEY_3=your_third_api_key
```

### API Keys File

Create a text file with one API key per line:

```
AIzaSyD...key1
AIzaSyE...key2  
AIzaSyF...key3
```

### Cookies for Restricted Content

To access age-restricted or private content:

1. Install a browser extension like [Cookie Editor](https://chrome.google.com/webstore/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)
2. Export YouTube cookies in Netscape format
3. Save as `cookies.txt`
4. Pass the file path to the library

## Output Structure

The library organizes output in a structured format:

```
output_directory/
├── raw/                    # Downloaded audio files
│   ├── video1.wav
│   └── video2.wav
├── converted/              # Format-converted audio
│   ├── converted_video1.wav
│   └── converted_video2.wav  
├── segments/               # Segmented audio chunks
│   ├── video1_segment_001.wav
│   ├── video1_segment_002.wav
│   └── ...
└── transcriptions.csv      # Final dataset
```

The CSV contains two columns:
- `audio_filepath`: Path to audio segment
- `transcript`: Transcribed text

## API Reference

### YouTubeAudioCollector

Main class for the complete pipeline.

```python
collector = YouTubeAudioCollector(
    output_dir="./output",           # Output directory
    prefix="dataset",                # File prefix
    cookies_path="cookies.txt",      # Cookies file path
    api_keys=["key1", "key2"],       # Multiple API keys
    api_key="single_key",            # Single API key
    sample_rate=16000,               # Audio sample rate
    threads=4,                       # Processing threads
    batch_size=4,                    # Transcription batch size
    calls_per_key_per_minute=15,     # API rate limit
    key_cooldown=60                  # Key cooldown period
)
```

### APIKeyManager

Manages multiple API keys with intelligent rotation.

```python
manager = APIKeyManager(
    api_keys=["key1", "key2", "key3"],
    calls_per_key_per_minute=15,
    cooldown_period=60
)

# Get next available key
key = manager.get_next_available_key()

# Mark key as unhealthy (e.g., after rate limit)
manager.mark_key_unhealthy(key)
```

### YouTubeDownloader

Downloads audio from YouTube URLs.

```python
downloader = YouTubeDownloader(cookies_path="cookies.txt")
files = downloader.download_audio(url, output_dir, prefix="dataset")
```

### AudioProcessor

Processes and segments audio files.

```python
processor = AudioProcessor(sample_rate=16000)

# Convert format
converted = processor.convert_format(audio_path, output_dir)

# Segment audio
segments = processor.segment_audio(
    audio_path, 
    output_dir,
    min_length=12000,    # 12 seconds
    max_length=28000,    # 28 seconds
    min_silence_len=500, # 500ms
    silence_thresh=-35   # -35dB
)
```

### TranscriptionService

Transcribes audio using Gemini API.

```python
# With API key manager
transcriber = TranscriptionService(api_key_manager=manager)

# With single key
transcriber = TranscriptionService(api_key="your_key")

# Transcribe
transcript = transcriber.transcribe_audio(audio_path, language="en")
```

## Supported Languages

The library supports transcription in multiple languages:

- `en` - English
- `kn` - Kannada
- `hi` - Hindi
- `es` - Spanish
- `fr` - French
- And many more supported by Gemini API

## Performance Tips

1. **Use Multiple API Keys**: Significantly improves throughput by avoiding rate limits
2. **Optimize Thread Count**: Use 4-8 threads for most systems
3. **Batch Processing**: Use batch sizes of 4-6 for optimal API usage
4. **Storage**: Use SSD storage for better I/O performance with large datasets

## Legal Considerations

- Ensure you comply with YouTube's Terms of Service
- Only download content you have rights to use
- Respect content creators' intellectual property
- Consider fair use guidelines for research and educational purposes

## Contributing

Contributions are welcome! Please read our [contributing guidelines](CONTRIBUTING.md) and submit pull requests to our [GitHub repository](https://github.com/yourusername/youtube-audio-dataset-collector).

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Support

- **Documentation**: [https://youtube-audio-dataset-collector.readthedocs.io/](https://youtube-audio-dataset-collector.readthedocs.io/)
- **Issues**: [GitHub Issues](https://github.com/yourusername/youtube-audio-dataset-collector/issues)
- **Discussions**: [GitHub Discussions](https://github.com/yourusername/youtube-audio-dataset-collector/discussions)

## Changelog

### v1.0.0
- Initial release
- YouTube audio downloading with yt-dlp
- Intelligent audio segmentation
- Multi-API key support for Gemini transcription
- Parallel processing
- Command-line interface
- Python API