FastVideo: Accelerating Video Generation from Training to Inference
Explore FastVideo, an open framework combining sparse attention, distillation, scalable training, and real-time video generation across GPUs and Apple Silicon.
FastVideo: Accelerating Video Generation from Training to Inference
Video diffusion models are powerful, but they are expensive at nearly every stage of the workflow. Training requires distributed systems and large datasets, while inference can involve dozens of denoising steps over high-dimensional spatiotemporal representations. Even a short generated clip may require significant GPU memory and compute.
FastVideo addresses this problem as a unified framework for both video-model post-training and accelerated inference. Instead of treating optimization as a separate deployment concern, the project brings together fine-tuning, distillation, sparse attention, distributed training, and user-facing generation APIs in one codebase.
The result is a practical foundation for researchers and engineers working with open video diffusion models, especially when latency, memory usage, or hardware flexibility matters.
What FastVideo Provides
FastVideo supports end-to-end post-training workflows for both bidirectional and autoregressive video models. Developers can perform full fine-tuning or LoRA fine-tuning on state-of-the-art open video DiTs, prepare video, image, and text datasets, and use multiple distillation strategies to reduce inference cost.
The major training capabilities include:
- Distribution Matching Distillation (DMD2): Reduces the number of denoising steps required to generate a video.
- Video Sparse Attention (VSA): Reduces the attention workload by exploiting the structure of video data.
- Sparse distillation: Combines model training and sparse computation to achieve more than 50x denoising speedup in supported workflows.
- FSDP2 and sequence parallelism: Distributes large training jobs across devices.
- Selective activation checkpointing: Trades recomputation for lower memory consumption.
- Self-Forcing causal distillation: Supports causal generation workflows for autoregressive or streaming scenarios.
This combination is important because a faster attention kernel alone does not solve the full problem. A model may still be too slow because it requires too many denoising steps, or it may fail to fit into memory during training. FastVideo makes these optimization layers available together.
Inference Optimizations
For inference, FastVideo provides sequence parallelism for distributed generation and multiple attention backends. The framework exposes these features through both a command-line interface and a Python API, allowing it to serve as either a research environment or an application component.
The supported hardware and operating systems are broad for an acceleration-focused project. FastVideo supports NVIDIA H100, A100, and RTX 4090 GPUs, along with Linux, Windows, and macOS configurations. Apple Silicon support is provided through an MLX runtime and the FastMetal-QAD model family.
The project also includes integrations and deployment paths for real-time applications. Dreamverse, located under apps/dreamverse/, is FastVideo's real-time video generation and editing platform. It supports streaming generation and what the project calls vibe directing: interactively guiding a video while it is being produced. Dreamverse can run on a local GPU, a self-hosted B200 server over SSH, Docker, or serverless Modal.
Installation with uv
The project recommends using uv to create an isolated Python environment. This generally provides faster and more stable installation than an existing Conda environment.
For an NVIDIA system using CUDA 12, the basic setup is:
uv venv --python 3.12 --seed
source .venv/bin/activate
UV_TORCH_BACKEND=cu126 uv pip install fastvideo
For CUDA 13, use UV_TORCH_BACKEND=cu130 instead. The platform-specific installation guides are important because FastVideo depends on compiled kernels and different attention backends may have different compatibility requirements.
On Apple Silicon, install the MLX extras and use a FastMetal-QAD checkpoint:
uv pip install -e '.[mlx]'
The available FastMetal-QAD models include 1.3B, 5B, and 14B variants optimized for Mac hardware. After installation, download FastVideo/FastMetal-1.3B-QAD or another supported checkpoint and follow the Apple Silicon guide.
NVIDIA DGX Spark systems require a different approach. Because there is no prebuilt ARM wheel for the FastVideo CUDA kernel, the package must be installed from source:
UV_TORCH_BACKEND=cu130 uv pip install -e .
This compiles the kernel locally for the ARM64 environment. A compatible prebuilt ARM64 FlashAttention wheel is available separately.
Running a First Generation
Once the environment and VSA kernels are installed, a minimal Python example can generate a video with the VideoGenerator API:
import os
from fastvideo import VideoGenerator
def main():
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "VIDEO_SPARSE_ATTN"
generator = VideoGenerator.from_pretrained(
"FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
num_gpus=1,
)
prompt = (
"A curious raccoon peers through a vibrant field of yellow sunflowers, "
"its eyes wide with interest."
)
generator.generate_video(
prompt,
output_path="my_videos/",
save_video=True,
)
if __name__ == "__main__":
main()
Run it with:
python example.py
The FASTVIDEO_ATTENTION_BACKEND setting selects the video sparse attention backend. The num_gpus parameter can be adjusted for the available hardware, while output_path controls where the generated files are written. For production use, the inference quick start and model-family cookbook provide more detailed configuration options.
Distillation and Performance Examples
FastVideo's optimization work is particularly relevant when a model must produce short clips with low latency. The project reports that FastWan-QAD can generate five seconds of video in 1.8 seconds end to end. Another showcased workflow creates a five-second 1080p video in 4.5 seconds on a single GPU.
The repository includes sparse-distillation recipes for multiple model families. For example, the FastWan2.1-T2V-1.3B recipe uses the FastVideo Synthetic Wan2.1 480P dataset, while the FastWan2.2-TI2V-5B recipe uses the FastVideo Synthetic Wan2.2 720P dataset.
These examples illustrate the framework's central design principle: latency improvements come from combining model-level and systems-level techniques. Stepwise distillation lowers the number of denoising iterations, sparse attention reduces the cost of each iteration, and distributed execution makes larger workloads practical.
FastVideo in the Broader Ecosystem
The framework has also become a base for related research and production projects. SGLang's diffusion inference functionality is based on a FastVideo fork from September 24, 2025. Other projects built on or related to the framework include DanceGRPO, SRPO, DCM, HY-WorldPlay, Hunyuan Video 1.5, Kandinsky-5.0, and LongCat Video.
This ecosystem matters for developers choosing an inference stack. FastVideo is not limited to a single checkpoint or a single application. Its abstractions cover training, evaluation, inference, hardware-specific kernels, and application deployment, making it suitable for experimenting with new optimization methods as well as building video-generation products.
When to Use It
FastVideo is a strong candidate when you need to fine-tune an open video model, reduce a diffusion model's step count, deploy sparse attention, or build a low-latency generation service. It is especially useful for teams that want to keep research and deployment code close together.
The main prerequisites are access to compatible hardware, familiarity with PyTorch and diffusion models, and willingness to follow the platform-specific installation instructions. Compiled CUDA kernels and model-specific assumptions mean that a generic package installation may not be enough on every machine.
For developers working on real-time video generation, the combination of DMD2, VSA, sequence parallelism, and hardware-specific runtimes makes FastVideo more than a model demo. It is an engineering toolkit for moving video generation from an offline experiment toward an interactive system.