Mastering Generative Voice AI From Tokens to Agentic TTS

What you'll learn

---

Explain the physics, phonetics, and acoustic features that underlie human speech production
Compare traditional cascade TTS pipelines with modern Speech Language Model (SpeechLM) architectures
Build and apply neural audio codecs and semantic tokenization (EnCodec, HuBERT, wav2vec 2.0, RVQ) (
Implement autoregressive codec-based TTS with multi-stream token decoding strategies
Design unified speech-text models with cross-modal alignment and paralinguistic control
Apply latent diffusion and conditional flow matching to generate high-quality mel-spectrograms
Evaluate flow matching vs. diffusion trade-offs for speed, quality, and controllability
Deploy low-latency, streaming agentic TTS systems with real-time interruption handling
Requirements

---

A solid understanding of deep learning fundamentals (neural networks, backpropagation, and training basics)
Working knowledge of Python and a deep learning framework such as PyTorch
Basic familiarity with core NLP or LLM concepts (tokenization, transformers, attention) is helpful but not mandatory - key ideas are reviewed in the course
No prior audio signal processing experience needed - Module 1 builds this from first principles
Description

---

Generative voice AI has moved far beyond simple text-to-speech - and this course takes you from the physics of sound all the way to building production-grade, agentic voice systems.
Most TTS courses stop at basic vocoders or off-the-shelf APIs. This one goes deeper. You'll start with thefundamentals of human speech - acoustics, phonetics, and prosody - before diving into the architectures actually powering today's state-of-the-art voice models: self-supervised representation learning (wav2vec 2.0, HuBERT), neural audio codecs (EnCodec, SoundStream, DAC), and the tokenization strategies that let LLMs "speak."
From there, you'll master the two dominant modern paradigms -autoregressive codec-based TTS andlatent diffusion / conditional flow matching - understanding exactly when and why each is used in real systems. You'll also explore unified speech-text models, paralinguistic modeling (laughter, breathing, affect), and zero-shot voice cloning.
By the final module, you'll understand how to buildlow-latency, streaming, agentic voice pipelines - the same techniques behind real-time conversational AI agents - covering chunked inference, speculative decoding, WebSocket streaming, and turn-taking.
What you'll learn

---

The science of speech production and acoustic feature extraction
How neural audio codecs and semantic tokenization work
Autoregressive and diffusion/flow-based TTS architectures
Cross-modal speech-text alignment techniques
Building low-latency, interruption-aware conversational voice agents
Whether you're an ML engineer, researcher, or voice-tech founder, this course gives you thecomplete architectural picture - from tokens to agents.
Who this course is for

---

ML/AI engineers who want to move beyond calling TTS APIs and understand how state-of-the-art voice models actually work under the hood
Speech and NLP researchers looking to bridge classical signal processing with modern generative modeling (diffusion, flow matching, SpeechLMs)
Voice-tech founders and product engineers building conversational AI agents who need to make informed architecture decisions
Graduate students or self-taught ML practitioners seeking a rigorous, end-to-end curriculum on generative audio, from acoustic theory to agentic deployment
Homepage

---