Speech AI Systems Engineer: Neural TTS & Voice Cloning

Speech AI for anything outside a handful of high resource languages runs into a wall fast. Pre-trained TTS models degrade sharply, and there is often no usable dataset to fine-tune on. The bottleneck isn't the model architecture. It's the absence of a clean, curated speech corpus, which most teams underestimate the cost of building.

Building a usable dataset starts with transcription. I use OpenAI Whisper, open source and used as-is, combined with Silero VAD to detect and segment speech from silence and background noise before any audio reaches the pipeline. From there, the dataset work is its own pipeline: audio segmentation, noise reduction, normalisation, metadata generation, and quality filtering, all done before any fine-tuning starts. On the synthesis side, I work with VITS and XTTS-v2 for natural speech generation. Where a target language is underserved by pretrained models, I apply LoRA and QLoRA fine-tuning to adapt their transformer backbones using the purpose-built dataset.

A FastAPI audio service with inference endpoints returning WAV or MP3 output at low latency, text normalisation for consistent pronunciation, and a documented, reproducible dataset pipeline rather than a one-off training run. This is the system I built end-to-end for Urdu: a 100+ hour curated speech corpus built with Silero VAD and Whisper transcription, LoRA and QLoRA fine-tuning of open-source TTS models for Urdu synthesis, and a FastAPI inference layer generating real-time audio output, for a language with limited existing speech AI tooling.