lushy/fastspeech2-ljspeech

FastSpeech 2 — Python 3.12/3.13 and Colab

PyTorch implementation of FastSpeech 2: Fast and High-Quality End-to-End Text to Speech, based on ming024/FastSpeech2. This fork updates the training workflow for standard CPython 3.12 and 3.13 and provides a Google Colab notebook for GPU training and Hugging Face Hub publishing.

The implementation uses phoneme-level pitch and energy by default and includes a HiFi-GAN vocoder. It is a custom PyTorch project; load and synthesize with the repository scripts rather than the Transformers pipeline() API.

What this fork updates

• Declares standard CPython 3.12 and 3.13 support and refreshes the dependency/install metadata.
• Replaces machine-specific LJSpeech paths with portable project-relative defaults.
• Adds a Colab workflow that can reuse Drive features or fetch LJSpeech and the upstream MFA alignments, then preprocess in the runtime.
• Adds Hugging Face Hub publishing for the code, HiFi-GAN vocoder, and selected FastSpeech 2 checkpoint.
• Updates training for CUDA AMP, MPS/CPU fallback, configurable data workers, atomic checkpoint saves, and checkpoint retention.
• Adds an iSTFTNet vocoder integration while retaining HiFi-GAN as the LJSpeech default.

These are workflow and compatibility updates to the existing implementation; they do not claim a new FastSpeech 2 architecture or independently measured quality improvements.

Hugging Face model repository

Choose a Hugging Face model repo ID, then publish the repository code, the LJSpeech vocoder, and the newest local checkpoint. The helper creates the model repo if it does not exist:

python -m pip install -r requirements.txt
hf auth login
python scripts/push_to_hub.py YOUR_USER/fastspeech2-ljspeech

Pass --private to create a private repository, or --checkpoint PATH to choose a specific checkpoint. The helper excludes raw audio, preprocessed .npy features, logs, and local output folders. It uploads code, hifigan/generator_LJSpeech.bin and the selected checkpoint under checkpoints/<step>.bin. The Hub uses .bin filenames because the local .pth.tar names are intentionally ignored; the notebook restores the expected local filenames after download. The ignore rules intentionally hide checkpoints and vocoder weights from a generic folder upload; use this helper to add only the selected release weights. It also excludes the old demo audio, dataset manifests, credentials, caches, and the local virtual environment.

The model card describes the project and training recipe. Add measured evaluation results and sample audio after training before presenting a checkpoint as a finished release. Do not upload voice data unless you have the rights to redistribute it.

Colab training

Open notebook.ipynb in Colab, or use the Open in Colab button above. The notebook is named notebook.ipynb so the Hub's /colab link launches this custom training workflow instead of a generic generated notebook. Before running:

1. Set HF_REPO_ID in the notebook to the model repo you created and published above.
2. Choose a data path in the notebook. drive_preprocessed reuses MyDrive/FastSpeech2/preprocessed_data/LJSpeech; download_and_preprocess fetches the official LJSpeech 1.1 archive and the MFA alignments linked by the upstream FastSpeech 2 repo, then runs the preparation scripts. The latter downloads several GB and recomputes features, so it takes longer.
3. Select a GPU runtime. The notebook downloads code and model weights from the Hub, prepares or copies features to Colab's temporary disk for faster reads, resumes from step 200,000, and saves checkpoints/logs to Drive. At the end it uploads the newest checkpoint and updated code back to the same model repo.

For a fresh run, set START_STEP = 0; the notebook skips checkpoint download in that case. The default recipe resumes the available 200,000-step LJSpeech checkpoint.

Local setup

Use regular CPython (not the experimental free-threaded 3.13t build). Python 3.12 is the recommended default; 3.13 is supported by the project dependency range.

python -m venv .venv
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Linux/macOS: source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For CUDA, install the matching PyTorch build for your driver/runtime first using the official PyTorch selector. Then install this project's remaining requirements.

Dataset and preprocessing

Download and extract LJSpeech 1.1 under raw_data/LJSpeech-1.1/. The default config writes aligned audio/text to raw_data/LJSpeech/ and features to preprocessed_data/LJSpeech/.

MFA alignments are required before feature extraction. Prepare the alignment inputs, place the provided TextGrid files under preprocessed_data/LJSpeech/TextGrid/, then preprocess:

python prepare_align.py config/LJSpeech/preprocess.yaml
# Run Montreal Forced Aligner using the corpus-specific instructions if alignments are not supplied.
python preprocess.py config/LJSpeech/preprocess.yaml

Do not commit raw audio, TextGrids, or generated .npy features to the code/model repository.

Train and resume

python train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
python train.py --restore_step 200000 -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml

Training saves atomically every 10,000 steps and keeps the newest two checkpoints by default. Change step.save_step, step.keep_checkpoint_count, and optimizer.num_workers in config/LJSpeech/train.yaml to fit your storage and machine. Monitor training with:

tensorboard --logdir output/log/LJSpeech

Synthesis

Put a FastSpeech 2 checkpoint at output/ckpt/LJSpeech/<STEP>.pth.tar and the HiFi-GAN weights at hifigan/generator_LJSpeech.pth.tar, then run:

python synthesize.py --text "Hello from FastSpeech 2." --restore_step STEP --mode single -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml

Outputs are written under output/result/LJSpeech/.

Upstream work and attribution

This fork builds on, and retains attribution to, the following research and implementations:

• FastSpeech 2 research: Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu, FastSpeech 2: Fast and High-Quality End-to-End Text to Speech, ICLR 2021.
• Base PyTorch implementation: Chung-Ming Chien (Ming024)'s FastSpeech2 repository, which credits Zhengxi Liu (xcmyz)'s FastSpeech implementation as its starting point. This fork credits that upstream work and lists its own changes above.
• HiFi-GAN vocoder: Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, HiFi-GAN. Its source license is kept in hifigan/LICENSE.
• iSTFTNet vocoder: Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki, iSTFTNet; the bundled implementation is from rishikksh20/iSTFTNet-pytorch, under Apache-2.0. Its license is kept in istftnet/LICENSE.
• LJSpeech: dataset by Keith Ito, recordings by Linda Johnson for LibriVox, with alignment and annotation by Keith Ito. The dataset's official page identifies it as public domain: LJ Speech Dataset.

See THIRD_PARTY_NOTICES.md for bundled component scope and license notes.

License and data

The root FastSpeech 2 project code is MIT licensed under LICENSE. This repository also includes separately licensed third-party code, so the Hugging Face metadata is marked other; review the notices and each included license. The LJSpeech 1.1 dataset describes itself as public domain. Training checkpoints and pretrained vocoder weights should be reviewed and released under their applicable terms; uploading this repository does not change those terms. Raw audio and generated feature data are excluded from the upload workflow.

Citation

@inproceedings{ren2021fastspeech2,
  title={FastSpeech 2: Fast and High-Quality End-to-End Text to Speech},
  author={Ren, Yi and Hu, Chenxu and Tan, Xu and Qin, Tao and Zhao, Sheng and Zhao, Zhou and Liu, Tie-Yan},
  booktitle={International Conference on Learning Representations},
  year={2021},
  url={https://arxiv.org/abs/2006.04558}
}