oneplanettechnologies/my-big-model

Voicecraft: The Art of Text-to-Speech

A complete, production-ready Text-to-Speech (TTS) system that crafts human-like voices from text. This system uses Tacotron2 for mel-spectrogram generation and HiFiGAN for high-quality neural vocoding.

🎯 Project Overview

Voicecraft is a comprehensive TTS system developed by students of Dhemaji Engineering College, Computer Science and Engineering Department, under the guidance of Barga Deori Sir.

Team Members:

• Abinash Dutta
• Arindom Bordoloi
• Bakhtiar Ahmed
• Chinmoy Mahanta
• Mumon Saikia
• Preety Rani Boruah

✨ Features

Core Capabilities

• Text Normalization: Handles numbers, abbreviations, and special symbols
• G2P Conversion: Grapheme-to-Phoneme with CMU dictionary and rule-based fallback
• Prosody Prediction: Predicts pitch, duration, rhythm, stress, and intonation
• Tacotron2: State-of-the-art mel-spectrogram generation with location-sensitive attention
• HiFiGAN Vocoder: High-quality neural vocoder for natural waveform synthesis
• Audio Preprocessing: Silence trimming, noise reduction, normalization
• Evaluation Metrics: MCD, Spectral Distortion, and MOS helper

User Interface

• Gradio Web Interface: Beautiful, responsive GUI
• Real-time Synthesis: Fast inference with GPU support
• Visualization: Mel-spectrogram display
• Phoneme Display: Shows G2P conversion results
• Audio Playback & Download: Listen and save generated audio

📁 Project Structure

voicecraft/
├── datasets/              # Dataset handling
│   ├── download_ljspeech.py
│   └── prepare_dataset.py
├── preprocessing/         # Text and audio preprocessing
│   ├── text_normalizer.py
│   └── audio_processor.py
├── g2p/                  # Grapheme-to-Phoneme conversion
│   └── g2p_converter.py
├── prosody/              # Prosody prediction
│   └── prosody_predictor.py
├── tacotron2/            # Tacotron2 acoustic model
│   ├── model.py
│   └── train.py
├── vocoder/              # Vocoders (HiFiGAN, WaveGlow)
│   ├── hifigan.py
│   └── train.py
├── utils/                # Utility functions
│   ├── audio_utils.py
│   ├── collate_fn.py
│   └── evaluation.py
├── deployment/           # Deployment files
│   ├── app.py            # Gradio GUI
│   ├── requirements.txt
│   └── colab_deploy.ipynb
├── future/               # Future expansion modules
│   ├── assamese_tts.py
│   ├── emotional_tts.py
│   ├── whisper_transcription.py
│   └── mobile_inference.py
└── README.md

🚀 Quick Start

Installation

1. Clone the repository (or download the code):

git clone <repository-url>
cd voicecraft

2. Install dependencies:

pip install -r deployment/requirements.txt

3. Download datasets (optional, for training):

python voicecraft/datasets/download_ljspeech.py
python voicecraft/datasets/prepare_dataset.py data/LJSpeech-1.1

Running the GUI

1. Start the Gradio interface:

python voicecraft/deployment/app.py

2. Open your browser to http://localhost:7860

3. Use the interface:

   • Enter text in the text box
   • Click "Paste Intro Paragraph" for demo text
   • Click "Generate Speech" to synthesize
   • Listen to the audio and view phonemes/spectrogram

📊 Datasets

Primary Dataset: LJSpeech 1.1

• 13,100 text-audio pairs
• 24 hours of clean speech
• 22.05 kHz WAV format
• Used for Tacotron2 + HiFiGAN training

Download and Prepare

# Download LJSpeech
python voicecraft/datasets/download_ljspeech.py

# Prepare dataset with train/val/test splits
python voicecraft/datasets/prepare_dataset.py data/LJSpeech-1.1

The dataset will be split into:

• 90% training (11,790 samples)
• 5% validation (655 samples)
• 5% test (655 samples)

Secondary Dataset (Optional)

For experimental purposes, a small custom dataset (100-500 samples) can be used for GlowTTS experimentation.

🏗️ System Architecture

Complete Pipeline

1. Text Preprocessing

   • Text normalization (numbers → words, abbreviation expansion)
   • Special symbol cleaning
   • Whitespace normalization

2. G2P Conversion

   • CMU dictionary lookup
   • Rule-based fallback for unknown words
   • Phoneme sequence generation

3. Prosody Prediction

   • Pitch prediction
   • Duration estimation
   • Rhythm patterns
   • Stress patterns
   • Intonation contours

4. Tacotron2 Acoustic Model

   • Encoder: Character embeddings + CNN + BiLSTM
   • Location-Sensitive Attention: Monotonic alignment
   • Decoder: Attention RNN + Decoder RNN + Postnet
   • Output: Mel-spectrogram

5. HiFiGAN Vocoder

   • Multi-period discriminator
   • Generator with residual blocks
   • Fast inference
   • High-quality waveform synthesis

6. Post-processing

   • Audio normalization
   • Silence trimming
   • Quality enhancement

🔬 Training

Tacotron2 Training

python voicecraft/tacotron2/train.py \
    --train_data data/processed/train \
    --val_data data/processed/val \
    --checkpoint_dir checkpoints/tacotron2 \
    --log_dir logs/tacotron2 \
    --num_epochs 100 \
    --batch_size 32 \
    --lr 1e-3 \
    --use_amp \
    --save_interval 1000

Training Parameters:

• Steps: 150k-200k steps recommended
• Optimizer: Adam (lr=1e-3, weight_decay=1e-6)
• Scheduler: ExponentialLR (gamma=0.999)
• Mixed Precision: AMP enabled
• Multi-GPU: Optional (DDP support)

Loss Functions:

• L1 loss (mel-spectrogram)
• MSE loss (mel-spectrogram)
• Binary cross-entropy (gate prediction)

HiFiGAN Training

python voicecraft/vocoder/train.py \
    --train_data data/processed/train \
    --checkpoint_dir checkpoints/hifigan \
    --log_dir logs/hifigan \
    --num_epochs 100 \
    --batch_size 16 \
    --lr_g 2e-4 \
    --lr_d 2e-4 \
    --use_amp \
    --save_interval 1000

Training Parameters:

• Steps: 250k steps recommended
• Optimizer: AdamW (betas=(0.8, 0.99))
• Losses: Generator loss, discriminator loss, feature matching loss

📈 Evaluation

Metrics

1. MCD (Mel Cepstral Distortion)

   • Lower is better
   • Typical range: 4-8 dB

2. Spectral Distortion

   • Lower is better
   • Measures mel-spectrogram quality

3. MOS Helper

   • Estimated quality indicators
   • SNR, spectral centroid, etc.

Usage

from voicecraft.utils.evaluation import compute_mcd, compute_spectral_distortion

mcd = compute_mcd(mel_pred, mel_target)
sd = compute_spectral_distortion(mel_pred, mel_target)

🌐 Deployment

HuggingFace Spaces

1. Create a new Space on HuggingFace

2. Upload files:

   • voicecraft/deployment/app.py
   • voicecraft/deployment/requirements.txt
   • All voicecraft/ source code
   • Model checkpoints (or use model loading from URLs)

3. Configure Space:

   • Hardware: CPU or GPU
   • Python SDK

4. The app will automatically:

   • Install dependencies
   • Load models on startup
   • Provide public URL

Google Colab

1. Upload the notebook:

   • voicecraft/deployment/colab_deploy.ipynb

2. Run cells:

   • Install dependencies
   • Load models
   • Start Gradio interface

3. Create public tunnel:

   from pyngrok import ngrok
   public_url = ngrok.connect(7860)
   print(f"Public URL: {public_url}")
   

4. Share the link for public access

Local Deployment

# Install dependencies
pip install -r voicecraft/deployment/requirements.txt

# Run the app
python voicecraft/deployment/app.py

The app will be available at http://localhost:7860

🎨 GUI Features

The Gradio interface includes:

• Text Input: Multi-line text box
• "Paste Intro Paragraph" Button: Automatically fills demo text
• "Generate Speech" Button: Synthesizes audio
• "Clear" Button: Resets all fields
• Audio Output: Play generated speech
• Phoneme Display: Shows G2P conversion
• Mel-Spectrogram Image: Visual representation
• Download Audio: Save generated files
• Presentation Mode: Toggle for enhanced visualization (stub)

Special Feature: Paste Intro Paragraph

Click the "📋 Paste Intro Paragraph" button to automatically fill the text box with:

Hello everyone, hope you are well. We are students of Dhemaji Engineering College, Computer Science and Engineering Department, and our names are: Abinash Dutta, Arindom Bordoloi, Bakhtiar Ahmed, Chinmoy Mahanta, Mumon Saikia, Preety Rani Boruah. We are developing the project "Voicecraft: The Art of Text-to-Speech – Crafting Human-Like Voices from Text" under the guidance of Barga Deori Sir. Thank you.

🔮 Future Expansions

Planned Features

1. Assamese TTS (future/assamese_tts.py)

   • Assamese language support
   • Assamese-specific preprocessing
   • Assamese G2P conversion

2. Emotional TTS (future/emotional_tts.py)

   • Happy, Sad, Angry emotions
   • Emotion-conditioned synthesis
   • Prosody modification

3. Whisper Transcription (future/whisper_transcription.py)

   • Speech-to-text integration
   • Automatic transcription
   • Multi-language support

4. Mobile Inference (future/mobile_inference.py)

   • Model quantization
   • ONNX/TFLite export
   • Lightweight inference

📚 Documentation

Module Documentation

• Dataset Handling: See voicecraft/datasets/
• Preprocessing: See voicecraft/preprocessing/
• G2P: See voicecraft/g2p/
• Models: See voicecraft/tacotron2/ and voicecraft/vocoder/
• Deployment: See voicecraft/deployment/

API Usage

from voicecraft.preprocessing.text_normalizer import normalize_text
from voicecraft.g2p.g2p_converter import text_to_phonemes
from voicecraft.tacotron2.model import Tacotron2
from voicecraft.vocoder.hifigan import HiFiGAN

# Normalize text
text = "Hello, it's 9:30 a.m. on Jan. 15, 2023."
normalized = normalize_text(text)

# Convert to phonemes
phonemes = text_to_phonemes(normalized)

# Load models
tacotron2 = Tacotron2()
hifigan = HiFiGAN()

# Generate speech
text_seq = text_to_sequence(normalized)
mel = tacotron2.infer(text_seq)
audio = hifigan.infer(mel)

🛠️ Development

Adding New Features

1. Text Processing: Extend preprocessing/text_normalizer.py
2. G2P: Add rules to g2p/g2p_converter.py
3. Models: Modify architectures in tacotron2/ or vocoder/
4. GUI: Update deployment/app.py

Testing

# Test individual modules
python voicecraft/preprocessing/text_normalizer.py
python voicecraft/g2p/g2p_converter.py

📝 License

[Add your license here]

👥 Credits

• Institution: Dhemaji Engineering College
• Department: Computer Science and Engineering
• Supervisor: Barga Deori Sir
• Developers: Voicecraft Team

🙏 Acknowledgments

• LJSpeech dataset creators
• Tacotron2 authors
• HiFiGAN authors
• CMU Pronouncing Dictionary
• Open source community

📞 Contact

For questions, issues, or contributions, please contact the development team.

---

Voicecraft: The Art of Text-to-Speech - Crafting Human-Like Voices from Text