vertox-ai/speech-text-data-aligners

Speech-Text Data Aligners

A library-first framework that uses a known, authoritative English transcript to estimate word timing in WAV audio. It performs forced alignment, not ASR transcription, and never replaces en_transcript with recognized text.

Features

• One backend-neutral Python contract for NeMo, MFA, and WhisperX.
• Reversible normalization and monotonic mapping back to original transcript words.
• Validated, non-fabricated timestamps with structured failures and provenance.
• Source-preserving, deterministic CSV annotation with atomic publication.
• Reusable annotation and cross-backend benchmark services independent of the CLI.
• Lazy, isolated optional backends; ordinary tests require no model, GPU, or network.

Supported Backends

Backend	Implementation	Empirical status
NVIDIA NeMo Forced Aligner (NFA)	Accepted	NVIDIA/GPU smoke and five-row validation deferred to an EC2 follow-up
Montreal Forced Aligner (MFA) 3.x	Accepted	Real five-row validation accepted: 5/5 successful
WhisperX 3.8.6	Accepted	Real five-row validation accepted: 5/5 successful; full annotation 36/36 successful

Implementation acceptance and empirical validation are separate. See Project Status for the evidence boundary.

Input and Output

Input metadata must contain:

line_id,audio_id,en_transcript,ru_translation

Audio is resolved as <audio_dir>/<audio_id>.wav. Annotation preserves all source columns, values, and row order, then appends exactly:

first_word_text,first_word_start_s,first_word_end_s,
last_word_text,last_word_start_s,last_word_end_s

en_transcript is authoritative. Backends may create internal reference representations, but independent ASR output cannot become the transcript.

Quick Start

Use the pinned environment definitions in environments/ and keep backend runtimes isolated. Detailed setup and execution paths are in Workflows.

Minimal Python API:

from pathlib import Path

from speech_text_data_aligners import AnnotationOptions, annotate_metadata
from speech_text_data_aligners.backends.mfa.backend import MfaBackend
from speech_text_data_aligners.backends.mfa.config import MfaConfig

run = annotate_metadata(
    metadata_csv=Path("metadata.csv"),
    audio_dir=Path("wavs"),
    output_csv=Path("annotated.csv"),
    backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
    options=AnnotationOptions(strict=False),
)
print(run.receipt.success_count, run.receipt.failure_count)

Minimal CLI:

PYTHONPATH=src python -m speech_text_data_aligners.cli.main annotate \
  metadata.csv wavs annotated.csv \
  --backend mfa \
  --backend-config-json '{"environment_name":"speech-aligners-mfa"}'

Use --strict to abort publication on the first failure. Without it, failed source rows are preserved with six blank alignment cells and a diagnostics JSON artifact.

Documentation

• Project status
• Architecture
• Workflows
• Aligner engineering contract
• Modular architecture contract
• Stage 5 final acceptance report

Current Validation Status

Stages 0–5 are accepted. MFA and WhisperX each succeeded on all five shared validation rows and matched all five supplied first-word text labels. First-word-end MAE was 0.0162 s for MFA and 0.0154 s for WhisperX. WhisperX also completed all 36 annotation rows.

Only first-word text and first-word end currently have manual gold. First-word start, last-word start, and last-word end have structural validation and pairwise comparison, but no manual ground-truth validation. NeMo's implementation is accepted; empirical NVIDIA/GPU validation remains pending.

License & Copyright

Licensed under the Apache License 2.0. The official license text is also available from the Apache Software Foundation.

Copyright (c) 2026 VertoX-AI. All rights reserved.

Author

Patrick Lumbantobing — VertoX-AI

Citation

@misc{vertoxai2026speechtextdataaligners,
  title        = {Speech-Text Data Aligners},
  author       = {Tobing, P. L. and VertoX-AI},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vertox-ai/speech-text-data-aligners}}
}

Acknowledgments

This project builds on NVIDIA NeMo and NeMo Forced Aligner, Montreal Forced Aligner, WhisperX, PyTorch, Torchaudio, and Hugging Face. WhisperX uses Torchaudio's selected WAV2VEC2_ASR_BASE_960H bundle. Acknowledgment does not imply endorsement by any upstream project or organization.