NEXIDUB

MDX-Net (Music Demixing Network) decomposes the source audio track into two stems: the vocal foreground and the background accompaniment (music, ambient, SFX). The architecture uses a U-Net with band-split RNN processing, operating in the time-frequency domain via Short-Time Fourier Transform (STFT) with a hop size of 512 samples at 44.1 kHz.

The background stem is preserved through the pipeline and remixed at unity gain with the synthesized vocal track in the final output — preserving the ambient soundscape that defines the original content's identity. The isolated vocal track is passed to Whisper for transcription.

Whisper large-v3 is a sequence-to-sequence transformer encoder-decoder with 1.5 billion parameters, pre-trained on 680,000 hours of multilingual audio scraped from the web. The encoder processes 80-channel log-Mel spectrograms computed with a 25ms Hann window and 10ms hop size, yielding 100 frames per second of audio at the feature level.

Word-level timestamps are extracted via Dynamic Time Warping (DTW) alignment between the decoder's cross-attention weights and the encoder's time-frequency representations. This enables phoneme-level timing for the prosody transfer stage — the critical substrate that allows duration modeling to be linguistically grounded rather than estimated.

• WER targets — English <5%, Spanish/French/German <8%, CJK <12% (character error rate for logographic scripts)
• Word-level alignment — DTW on cross-attention weight matrices, yielding ±25ms timestamp accuracy for most words in conversational speech
• Long-form transcription — sliding window with 30s segments, 50% overlap for boundary reconciliation

Translation is handled by a sequence-to-sequence model with attention, injected with the 5-Layer Universal Semantic Representation (USR) framework as a structured system prompt. This forces the model to go beyond lexical equivalence and preserve the full pragmatic-prosodic intent of each utterance across language boundaries.

Context window: 512 tokens with 128-token sliding overlap between consecutive utterances, enabling paragraph-level coherence — critical for preserving pronoun coreference chains, discourse markers (however, therefore, moreover), and topic continuity across sentence boundaries in long-form content.

XTTS v2 (Coqui TTS) is a three-component generative stack that achieves state-of-the-art voice cloning with cross-lingual capability — meaning the cloned voice identity transfers across language boundaries, not just the acoustic texture of the source language.

• GPT-2 autoregressive text-to-latent decoder — 774M parameters, conditioned on a 256-dimensional d-vector speaker embedding. Generates discrete VQ-VAE latent codes from input phoneme sequences and speaker conditioning.
• Latent diffusion model — mel-spectrogram refinement via DDPM (Denoising Diffusion Probabilistic Model) with 50-step inference, guided by the discrete latent codes from stage one. Adds fine spectral texture, breathiness, and speaker-specific timbral characteristics.
• HiFi-GAN neural vocoder — 14M parameter generator with multi-scale, multi-period discriminators. Converts mel-spectrograms to 24kHz PCM waveforms. Trained on multi-speaker corpora; achieves MOS >4.2 (5-point mean opinion score scale) for voice-cloned outputs.

The 256-dim d-vector is computed by a speaker verification encoder (similar to GE2E loss architecture) applied to a 3-10 second reference audio clip of the target speaker. This embedding conditions all three synthesis stages, ensuring speaker identity coherence through the diffusion and vocoder stages.

Prosody transfer is the technical differentiator that separates NexiDub from every commercial dubbing API. It ensures the dubbed audio preserves not just the speaker's voice but their emotional cadence — the rises and falls that signal emphasis, surprise, sarcasm, and tenderness — even after language substitution.

• F0 extraction via CREPE — Convolutional Representation for Pitch Estimation. CREPE is a fully convolutional model trained on a diverse multi-speaker corpus, achieving pitch estimation at 1.95 cents resolution (approximately 0.11% frequency accuracy). Substantially more accurate than autocorrelation or RAPT on voiced fricatives and glottalized sounds.
• Duration modeling — forced alignment on source → phoneme-level timing extraction → cross-lingual DTW to map source phoneme durations to target phoneme sequences. Accounts for the syllabic rate differences between languages (e.g., Spanish ~7.8 syllables/sec vs. German ~5.7 syllables/sec).
• Energy contour mapping — frame-level RMS energy extracted at 10ms hops, normalized per utterance, then remapped to target language energy contour via linear interpolation. Language-specific normalization prevents over-amplification of naturally lower-intensity languages (e.g., Japanese) or clipping of high-intensity languages (e.g., Arabic).
• Pause placement — source prosodic phrase boundaries detected via F0 reset patterns and energy valleys, preserved as structural constraints in the target TTS conditioning.

Wav2Lip achieves accurate lip synchronization by learning the mapping from mel-spectrogram features to lip region pixel-level predictions, conditioned on the identity of the target face. Pre-trained on the LRS2 (Lip Reading Sentences 2) dataset, which contains 224×224 face crops from BBC broadcast footage with aligned audio transcripts.

• Viseme generation — the model produces target viseme sequences (the 14 canonical mouth shapes of English, extended to ~22 for cross-lingual application) driven directly by the mel-spectrogram of the dubbed audio, not from phoneme lookup tables
• Face detector — S3FD multi-scale face detector runs at 25fps on the source video; detected bounding box crops are fed to the lip synthesis network at 96×96 pixel resolution
• Temporal alignment accuracy — <50ms audio-to-lip offset, which is below the approximately 150ms threshold for perceptual detection of lip-sync errors by untrained viewers
• Blending — synthesized lip region is blended back into source frame using Gaussian feathering at the facial boundary mask, preserving skin tone consistency