TaoMate-H3: Low-Latency Streaming Audio-Video Generation

Generating a complete audio-video sequence with a large diffusion model can take longer than the final clip itself. That latency makes interactive applications—live avatars, virtual presenters, real-time storytelling, and adaptive video generation—difficult to build.

TaoMate-H3 addresses this problem with a streaming inference runtime built on MiniMax H3. Instead of waiting for an entire request to finish, it generates synchronized audio and video in small chunks, allowing the first playable output to arrive much earlier while preserving continuity across longer sequences.

The project is developed by the Alibaba TaoLive AIGC Team and released under the MiniMax H3 Community License Agreement.

What makes TaoMate-H3 different?

TaoMate-H3 is designed around streaming rather than one-shot generation. Its core generation path uses a three-step LoRA strategy: each small chunk passes through three Stage3 denoising intervals. This reduces the amount of work required before a chunk reaches its final latent state.

The runtime also generates speech, environmental sound, and video on a shared timeline. This joint audio-video approach is important for applications where lip movement, sound effects, and visual events must remain synchronized rather than being assembled independently after generation.

Key capabilities include:

• Three-step LoRA streaming generation for faster chunk completion.
• Joint audio-video generation with speech, sound, and visuals aligned on one timeline.
• Low chunk latency, so the first usable latent arrives well before a full MiniMax H3 request completes.
• Long-form continuity using clean KV-cache updates and integrated audio guidance.
• Visual, voice, and motion consistency across prompt boundaries.
• Portrait and landscape output at 480p, 768p, and aligned 1080p resolutions.
• Single-node inference on either four GPUs or eight GPUs using TP2 and Ulysses sequence parallelism.

Hardware and software requirements

The validated setup is demanding and targets NVIDIA Hopper hardware:

• Linux
• Python 3.10 or 3.11
• NVIDIA Hopper or SM90 GPUs
• Validated configuration: 8 × NVIDIA H20 96 GB
• CUDA 12.8
• PyTorch 2.8
• FFmpeg with H.264 and AAC support

Create a dedicated environment before installing the runtime:

conda create -n taomate-h3 python=3.10 -y
conda activate taomate-h3

pip install torch==2.8.0 torchvision==0.23.0 \\
  --index-url https://download.pytorch.org/whl/cu128

pip install triton==3.4.0 vllm==0.11.1

TaoMate-H3 depends on Hopper-compatible FlashAttention. Install the Hopper implementation without build isolation:

git clone https://github.com/Dao-AILab/flash-attention.git
pip install --no-build-isolation ./flash-attention/hopper

Then clone and install TaoMate-H3 in editable mode:

git clone https://github.com/TaoLiveAIGC/TaoMate-H3.git
cd TaoMate-H3
pip install -e .

Install FFmpeg on Ubuntu or Debian with:

sudo apt-get update
sudo apt-get install -y ffmpeg

Download the required models

The runtime uses the MiniMax H3 FL2VA model. Download the required files from Hugging Face into the local models/MiniMax-H3 directory:

hf download MiniMaxAI/MiniMax-H3 \\
  --include "model_index.json" "FL2VA/*" \\
  --local-dir models/MiniMax-H3

TaoMate-H3 also uses a LoRA adapter. The current release is the step-3000 generator EMA adapter with rank 128 and alpha 128. If no local adapter is supplied, inference downloads the official adapter automatically into models/TaoMate-H3 on first use.

To download it in advance:

hf download TaoLiveAIGC/TaoMate-H3 \\
  --include "config.json" "adapter_config.json" "adapter_model.safetensors" \\
  --local-dir models/TaoMate-H3

The resulting directory should contain:

models/TaoMate-H3/
├── config.json
├── adapter_config.json
└── adapter_model.safetensors

For private or gated Hugging Face repositories, authenticate first:

hf auth login

Alternatively, provide access through the standard HF_TOKEN environment variable.

Prompting in five-second blocks

TaoMate-H3 supports either a single prompt reused throughout the sequence or a JSON file containing one prompt for each five-second interval. Block-based prompting is useful for long-form generation because it lets you change the scene, action, dialogue, or camera direction while the runtime maintains continuity through its KV-cache and audio guidance mechanisms.

Example prompt file:

{
  "prompts": [
    "prompt for seconds 0-5",
    "prompt for seconds 5-10"
  ],
  "seeds": [8301, 8301]
}

Using the same seed across blocks can help maintain a stable authored request configuration. The --prompt and --prompt-json options are mutually exclusive.

Running a 10-second generation

The following command generates a 10-second portrait video at 768p across eight GPUs:

python -m taomate_h3 \\
  --model-root models/MiniMax-H3 \\
  --prompt-json examples/prompts_10s.json \\
  --duration 10 \\
  --resolution 768x1376 \\
  --gpus 8 \\
  --devices 0,1,2,3,4,5,6,7 \\
  --seed 8301 \\
  --output outputs/demo_10s

The complete pipeline writes the final result to:

outputs/demo_10s/video.mp4

The launcher creates its own local distributed workers, so an external torchrun command is not required. To use a different adapter, specify it explicitly:

--adapter /path/to/adapter

The output directory must be new or empty. The total duration must be a multiple of five seconds unless the duration is inferred from the prompt JSON configuration.

Supported resolutions

TaoMate-H3 supports both portrait and landscape layouts:

Target	Portrait	Landscape
480p	480x864	864x480
768p	768x1376	1376x768
1080p	1088x1920	1920x1088

The 1080p dimensions use a 1088-pixel edge so that both dimensions remain divisible by 32. If delivery requires an exact 1080-pixel edge, crop the generated output after inference.

Important command-line options

Option	Purpose	Default
--model-root	MiniMax H3 directory containing FL2VA/	Required
--adapter	Local TaoMate-H3 LoRA directory	Automatic download to models/TaoMate-H3
--prompt	Reuse one prompt for every five-second block	—
--prompt-json	Supply one prompt per five-second block	—
--duration	Total duration in seconds	5, or inferred from JSON
--resolution	WIDTHxHEIGHT; short edge must be 480, 768, or 1088	768x1376
--gpus	Local GPU count; supported values are 4 or 8	8
--devices	Comma-separated CUDA device IDs	0 to gpus-1
--seed	Authored request seed	8301
--output	New or empty output directory	Required

Both resolution dimensions must be divisible by 32. The runtime supports four-GPU and eight-GPU configurations; the benchmarked results use eight GPUs with TP2 × Ulysses4.

Benchmark results

The project reports measurements from one node containing 8 × NVIDIA H20 96 GB GPUs, using a 480×864 canvas, a 10-second output, and seed 8301.

Metric	TaoMate-H3	MiniMax H3	Improvement
Pure DiT time	14.810 s	169.572 s	11.45× faster
First final chunk latent	6.148 s	170.052 s	27.66× faster
First playable video	17.287 s	183.313 s	10.60× faster
Peak DiT memory allocated	31.37 GiB	32.03 GiB	—

These numbers distinguish several different latency measurements. Pure DiT time excludes model loading, text encoding, VAE decoding, and media encoding. First playable video includes Video VAE decoding and H.264 publication in the matched first-chunk publication benchmark. The benchmark covers the Stage3 generation path and excludes the command's internal audio preparation.

A 10-second TaoMate-H3 run contains 24 generation forwards and eight clean-KV updates. The most notable result is not merely the end-to-end speedup: the first final chunk latent arrives in 6.148 seconds, creating a path toward applications that can begin playback while later chunks are still being generated.

When to use TaoMate-H3

TaoMate-H3 is particularly relevant when time-to-first-video matters as much as total generation time. Potential applications include:

• Interactive digital humans and virtual presenters
• Live or semi-live storytelling systems
• Long-form AI video with scene changes between prompt blocks
• Audio-reactive video generation
• Prototyping synchronized dialogue, sound, and visuals
• Streaming creative tools that cannot wait for an entire clip to render

It is not a lightweight consumer inference package. The Hopper GPU requirement, validated eight-GPU configuration, and specialized CUDA/PyTorch stack mean that deployment is best suited to dedicated servers or research infrastructure.

License and acknowledgements

TaoMate-H3 is released under the MiniMax H3 Community License Agreement. Users must follow the terms of that license for use and distribution.

The project acknowledges the teams behind MiniMax H3, Qwen3-VL, PyTorch, FlashAttention, Triton, and vLLM.

For implementation details and the latest examples, visit the TaoMate-H3 GitHub repository.

Source

TaoLiveAIGC/TaoMate-H3