AIBit-Discover Open Source Projects AIBit-Discover Open Source Projects

WeMM-Embedding: Universal Multimodal Retrieval from Tencent

Tencent’s WeMM-Embedding models unify text, images, video, and visual documents in one embedding space, with strong benchmark results and flexible dimensions.

WeMM-Embedding: Tencent’s Universal Multimodal Embedding Models

Multimodal retrieval systems often become complicated because every data type needs its own encoder, preprocessing pipeline, and similarity strategy. Tencent’s WeChat Vision team takes a different approach with WeMM-Embedding, a family of embedding models designed to represent text, images, videos, visual documents, and interleaved multimodal inputs in a unified vector space.

That makes the models useful for applications such as image-and-text search, video retrieval, visual document discovery, multimodal recommendation, and retrieval-augmented generation over mixed media. The project is open source under the Apache License 2.0 for Tencent-authored code and includes inference examples, serving scripts, and the evaluation pipeline used for its reported results.

Model family and supported inputs

The repository provides three model sizes:

Model Supported Matryoshka dimensions
WeMM-Embedding-2B 64, 128, 256, 512, 1024, 2048
WeMM-Embedding-4B 64, 128, 256, 512, 1024, 2560
WeMM-Embedding-9B 64, 128, 256, 512, 1024, 2048, 4096

All three models support text, images, videos, visual documents, and interleaved multimodal inputs. Audio is not currently supported. This limitation is important when interpreting broad multimodal benchmark results: audio tasks are assigned a score of zero in the MMEB-v3 aggregate evaluation.

Embeddings are extracted from the last-layer hidden state at the model’s dedicated <embedding> token position. The resulting vector is then L2-normalized, making cosine similarity or dot-product search straightforward to use in vector databases and retrieval pipelines.

Installation and inference

Clone the repository and install its dependencies:

pip install -r requirements.txt

The project recommends transformers==5.2.0 for inference and reproducibility. Newer versions may alter preprocessing behavior, so pinning the version is sensible when reproducing the repository’s examples or benchmark numbers.

Transformers inference

The Transformers example accepts text, image, and video inputs independently:

python examples/transformers_inference.py \
  --model /path/to/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

The command produces separate embeddings for the supplied text, image, and video inputs. If --dimension is omitted, the model emits its full embedding dimension. For the 2B model, that full size is 2048 dimensions; the 4B model uses 2560, and the 9B model uses 4096.

Sentence Transformers

The project also provides a Sentence Transformers integration:

python examples/sentence_transformers_inference.py \
  --model /path/to/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

Unlike the local-path example above, SentenceTransformer can load a Hugging Face model ID directly. For example:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("tencent/WeMM-Embedding-2B")

Text, image, and video inputs are passed through SentenceTransformer.encode(). The --dimension option selects the desired Matryoshka representation, allowing applications to trade vector size and search cost against retrieval quality.

Matryoshka embeddings: smaller vectors without retraining

WeMM-Embedding supports Matryoshka Representation Learning, or MRL. A single model can produce useful embeddings at several dimensions instead of requiring a separate model for each vector size.

To create a lower-dimensional representation, truncate the full vector and normalize the result again:

import torch

embedding = torch.nn.functional.normalize(
    embedding[..., :d],
    dim=-1,
)

The second normalization step matters. Truncation changes the vector’s norm, so renormalizing keeps similarity calculations consistent.

The repository reports that on MMEB-v2, the 2B model at 256 dimensions retains 98.7% of its full-dimensional image and video performance. This can substantially reduce storage, memory bandwidth, and approximate-nearest-neighbor index costs. A practical deployment can use 256-dimensional vectors for a first-pass search, while retaining the option to use a larger dimension when higher recall is necessary.

Serving with vLLM or SGLang

The models can be exposed as pooling endpoints through vLLM or SGLang. The repository tested vLLM 0.27.0 and SGLang 0.5.9.

For vLLM:

MODEL_PATH=/path/to/WeMM-Embedding-2B

vllm serve "$MODEL_PATH" \
  --runner pooling \
  --chat-template "$MODEL_PATH/embedding_chat_template.jinja"

For SGLang, first apply the project’s video-processing patch, then launch the embedding server:

MODEL_PATH=/path/to/WeMM-Embedding-2B

python scripts/patch_sglang_video.py
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --is-embedding \
  --enable-precise-embedding-interpolation

Equivalent wrappers are included in scripts/serve_vllm.sh and scripts/serve_sglang.sh. The explicit video patch and precise interpolation option are especially relevant for video workloads, where frame sampling and temporal preprocessing can affect both latency and retrieval quality.

Benchmark performance

On MMEB-v2, which covers 78 datasets, image and video tasks are measured with Hit@1 and visual-document tasks with NDCG@5. The 2B WeMM-Embedding model reports an average score of 77.9, with scores of 79.6 for image, 70.8 for video, and 80.7 for visual documents. The 4B model reaches an average of 79.2, while the 9B model reaches 80.6, with image, video, and visual-document scores of 81.9, 74.3, and 83.3, respectively.

For context, Qwen3-VL-Embedding reports MMEB-v2 averages of 73.2 at 2B and 77.8 at 8B. The repository also lists DME-Small at 74.8 and DME-Medium at 78.4; these are marked as closed-source leaderboard submissions without publicly released weights or a public inference endpoint.

On the broader MMEB-v3 evaluation, which contains 190 tasks spanning the 78 MMEB-v2 tasks, 53 text tasks, 47 agent tasks, 11 audio tasks, and MCMR, WeMM-Embedding reports:

  • 2B: V3-All 56.0, Text 45.3, Agent 45.1, MCMR 42.5, Audio 0.0
  • 4B: V3-All 58.2, Text 47.9, Agent 49.0, MCMR 41.9, Audio 0.0
  • 9B: V3-All 59.5, Text 48.8, Agent 51.0, MCMR 49.3, Audio 0.0

Text results use NDCG@5, while agent, MCMR, and audio results use Hit@1. The 9B model’s audio score is zero because audio input is unsupported, not because the model was evaluated as an audio encoder.

Reproducing the evaluation

The mmeb_v3_eval/ directory contains the evaluation code used for the reported numbers. It is based on the official TIGER-AI-Lab/VLM2Vec pipeline, with changes for multi-node, multi-GPU inference, a WeMM-specific backbone, batched preprocessing, aligned dataset instructions, and 64-frame video sampling.

A typical evaluation setup is:

cd mmeb_v3_eval

DATA_ROOT=/path/to/MMEB-V3 \
  bash scripts/download_data.sh

MODEL_PATH=/path/to/WeMM-Embedding-2B \
DATA_BASEDIR=/path/to/MMEB-V3 \
OUTPUT_DIR=exps/wemm_embedding \
  bash scripts/run_eval.sh

The evaluation tools also document single-node and multi-node execution using torchrun --nnodes=N, which is useful when testing larger checkpoints or reproducing the full benchmark.

When WeMM-Embedding fits

WeMM-Embedding is most compelling when a system needs one retrieval interface across several media types. A product catalog could search text descriptions against product images, a knowledge base could retrieve screenshots and scanned documents alongside prose, and a video archive could match natural-language queries to sampled clips.

The main engineering choices are model size, embedding dimension, and serving backend. Smaller checkpoints and Matryoshka dimensions reduce infrastructure cost, while the 9B model offers the highest reported benchmark scores in the repository. Teams should also account for video preprocessing, GPU memory, vector-index requirements, and the lack of audio support before selecting the model for production.

The repository includes only 14 commits and no published releases or packages, so version pinning and validating the exact preprocessing stack are particularly important. Third-party components retain their own licenses and copyright notices, which should be reviewed before deployment.

Citation

If you use the project in research or production experiments, the authors request a citation:

@article{wemm-embedding,
  title={WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report},
  author={Junjie Zhou and Ke Mei and Lei Li and Tianyi Wang and Fengyun Rao and Jing Lyu},
  year={2026},
  eprint={2608.24053},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.24053}
}

Source

Tencent/WeMM-Embedding: WeMM-Embedding is a family of universal multimodal embedding models by the WeChat Vision Team at Tencent, supporting multimodal understanding and retrieval.