Train a 65M VLM from Scratch in 2 Hours: MiniMind-V Deep Dive
Learn how to train a 65M-parameter Vision-Language Model from scratch in just 2 hours for under $3, with full code and dataset details.
Building a Vision-Language Model (VLM) from scratch sounds like a daunting task reserved for large teams with massive compute budgets. But what if you could train one in just two hours, for less than the cost of a coffee? That's exactly what the MiniMind-V project delivers: a fully functional 65M-parameter VLM that you can train on a single NVIDIA 3090 GPU.
This isn't just a toy. MiniMind-V is a complete, open-source implementation that covers the entire pipeline: data preparation, pretraining, supervised fine-tuning (SFT), and inference. It's designed to be a learning resource for anyone who wants to understand how modern VLMs like LLaVA and Qwen-VL actually work under the hood.
Why MiniMind-V Matters
The project's tagline — "大道至简" (Great truths are simple) — captures its philosophy. Most VLM tutorials either gloss over implementation details or require resources that individual developers don't have. MiniMind-V strips away the complexity, showing that the core of a VLM is surprisingly straightforward.
At just 65M parameters (roughly 1/2600th the size of GPT-3), MiniMind-V is small enough to run on consumer hardware. The entire SFT phase takes about 2 hours on a single RTX 3090, with GPU rental costs around $3. This makes it accessible for experimentation, learning, and even practical applications where a tiny model is sufficient.
Architecture: The Minimal VLM
MiniMind-V follows the same architecture as LLaVA-1.5, with three main components:
- Vision Encoder: SigLIP2 (siglip2-base-p32-256-ve) — a ViT-B/32 model with ~95M parameters, kept frozen during training.
- Projection Module: A 2-layer MLP (Linear → GELU → Linear) with LayerNorm, mapping visual features to the LLM's hidden dimension.
- Language Model: MiniMind's 64M-parameter LLM (768 hidden size, 8 layers), with only the first and last layers fine-tuned.
The key insight is that the vision encoder acts as a "foreign language dictionary" for images. It translates pixel data into tokens that the LLM can understand. The projection module then aligns these visual tokens with the text token embedding space.
Token Flow
When you input an image, the SigLIP2 encoder processes a 256×256 image into 64 patch tokens (8×8 grid with patch_size=32). These pass through the MLP projector to become 64 visual tokens, which replace the <|image_pad|> placeholders in the text prompt. The combined sequence then flows through the LLM as usual.
# Example prompt structure
"<|image_pad|>" * 64 + "\nWhat is in this image?"
Training Pipeline
MiniMind-V uses a two-stage training process, though you can skip the first stage if you're short on time.
Stage 1: Pretraining (Optional)
- Data: ~1.27M image-caption pairs from ALLaVA-4V (LAION + VFLAN subsets)
- Objective: Align visual tokens with language tokens
- Freeze Strategy: Freeze both LLM and Vision Encoder, train only the Projector (
--freeze_llm 2) - Learning Rate: 4e-4
- Max Sequence Length: 450 tokens
This stage is optional because the SFT dataset already includes all pretraining samples. However, running it first helps the projector converge more cleanly before fine-tuning the LLM.
Stage 2: Supervised Fine-Tuning (SFT)
- Data: ~2.9M mixed samples (image instruct, captions, pure text)
- Objective: Teach the model to answer questions about images
- Freeze Strategy: Train Projector + first and last LLM layers (
--freeze_llm 1) - Learning Rate: 5e-6
- Max Sequence Length: 768 tokens
The freeze strategy is critical. With only 64M parameters, fully unfreezing the LLM would cause catastrophic forgetting of language capabilities. By keeping the middle layers frozen, the model retains its pretrained knowledge while adapting to multimodal inputs.
# Start SFT training
python train_sft_vlm.py --epochs 2 --from_weight llm
Dataset Details
All training data comes from the ALLaVA-4V collection, which provides high-quality, bilingual (Chinese + English) image-text pairs. The SFT dataset includes:
- Image Instruct: ~1.4M reasoning QA pairs
- Image Captions: ~1.27M (merged from pretrain)
- Pure Text: ~230K dialogues (with black placeholder images to maintain language ability)
Images are resized to 256×256 and stored as JPEG in Parquet format for efficient loading. The Parquet format eliminates the need for slow image file extraction — everything is in one compressed file.
# Dataset format (parquet)
# columns: conversations (json string), image_bytes (binary)
Evaluation Results
The model was tested on 6 diverse images (dog, umbrella, bike, car, superhero, racecar). Both the dense (65M) and MoE (200M-A65M) versions correctly identified the main subject in all 6 cases. The MoE version produced richer scene descriptions, while the dense version was more concise with fewer repetitions.
However, both models exhibit common small-model issues: occasional repetition and detail hallucination. The author notes that this is expected — the visual signal is like a "foreign language" to the LLM, and the quality of understanding is fundamentally limited by the LLM's capacity. Scaling the backbone to a few billion parameters would dramatically improve accuracy.
Getting Started
To run MiniMind-V yourself:
# Clone the repo
git clone --depth 1 https://github.com/jingyaogong/minimind-v
cd minimind-v
# Install dependencies
pip install -r requirements.txt
# Download vision encoder and LLM weights
modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
modelscope download --model gongjy/minimind-3v-pytorch llm_768.pth --local_dir ./out
# Download SFT data
wget https://hf-mirror.com/datasets/jingyaogong/minimind-v_dataset/resolve/main/sft_i2t.parquet -P ./dataset
# Train (SFT only, ~2 hours on RTX 3090)
python train_sft_vlm.py --epochs 2 --from_weight llm
Future Improvements
The project outlines several promising directions:
- Dynamic resolution and tile-based encoding (like LLaVA-NeXT)
- Stronger vision encoders for finer-grained features
- Multi-image understanding, video understanding, and visual grounding
Conclusion
MiniMind-V is a remarkable achievement in democratizing VLM research. It proves that you don't need a massive cluster or a huge budget to build a functional multimodal model. The code is clean, well-documented, and serves as an excellent starting point for anyone wanting to understand or experiment with vision-language models.
Whether you're a student, researcher, or hobbyist, this project gives you the tools to join the VLM revolution. As the author puts it: "Building a plane with LEGOs is far more exciting than flying first class."