alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 · Hugging Face

What’s new in 2.0

Everything else is carried over from v1: control_in_dim = 49 (latent + masked latent + masks, so the identical department does management and inpaint), control_apply_audio = false, guidance-distilled (guidance_scale = 1.0), and the identical zero-gated skip-add into the primary department.

Loading a v1 config towards this checkpoint is a silent failure. With minimax_h3_control.yaml (5 blocks) the mannequin builds solely half the management department; load_state_dict(strict=False) drops control_blocks.5~9 as surprising keys and misplaces the remainder, producing unsuitable outputs. Always use minimax_h3_control_inpaint_post_norm.yaml.

Model Card

Model Features

• Union management over 8 situations: one checkpoint handles Canny, Depth, HED, MLSD, Pose, Scribble, Layout and Gray management movies for video-to-video technology — no per-condition checkpoint switching.
• Denser management injection: the management department attaches to 10 of the 50 transformer blocks (layers 0, 5, 10, 15, 20, 25, 30, 35, 40, 45); each management skip is added to the primary department via a zero-gated projection. This is roughly 2× the injection factors of v1 and offers tighter structural adherence.
• Guidance-distilled: run with guidance_scale = 1.0, one ahead cross per step, no classifier-free steering wanted.
• Inpainting is supported, with the post_norm recipe: the management enter is widened to control_in_dim = 49 (latent + masked latent + masks channels). Unlike v1, the masked pixels are zeroed after the ImageNet normalization (holes sit at 0 / mid-gray) fairly than earlier than it (holes landed close to −2 / excessive darkish), which improves how crammed areas mix with stored areas. Use examples/minimax_h3_fun/predict_v2v_control_inpaint.py.
• control_context_scale scales each management skip earlier than it’s added to the primary department: 1.0 provides the strongest management (used for all outcomes beneath), values beneath 1.0 weaken the steering of the management video, 0.0 switches the management department off.
• The technology follows the management video: the body depend snaps right down to the biggest 17 * n + 5 the video VAE can decode (period capped at 15 seconds), the canvas retains the management video’s personal facet ratio on the top * width pixel finances (each multiples of 32), at a hard and fast 24 fps.
• Detailed prompts give higher stability; we advocate describing the scene, the topic and the digital camera within the immediate.

Supported management situations

The Layout management movies comply with the format technology recipe of Wan2.1-VACE: per-subject bounding containers (detected/tracked or given straight) are rendered as color-coded containers on a white background, producing an abnormal RGB video that situations the mannequin. You can reuse the VACE-Annotators preprocessing instruments (e.g. vace_preproccess.py --task layout_track ...) to provide format movies from a reference video or a pair of bboxes.

Results

All samples beneath are generated with num_inference_steps = 40, guidance_scale = 1.0, control_context_scale = 1.00, seed 43, canvas mode management at a 704×1280 pixel finances, 24 fps. In every pair the highest row is the management video, the underside row is the output.

Canny	Depth	HED	MLSD
Pose	Scribble ✨	Layout ✨	Gray ✨

Inpainting (post_norm)

A masked area of the supply video is re-drawn from the immediate whereas the remainder of the body is preserved. The masks video is white the place the content material ought to be re-generated and black the place it ought to be stored.

Source video	Mask	Inpaint output

Inference

Go to the VideoX-Fun repository for extra particulars.

Please clone the VideoX-Fun repository and create the required directories:

git clone 


cd VideoX-Fun


mkdir -p fashions/Diffusion_Transformer

Then obtain the bottom MiniMax-H3 mannequin and this checkpoint into fashions/Diffusion_Transformer.

📦 fashions/
├──  Diffusion_Transformer/
│   ├── 📂 MiniMax-H3/
│   └──  MiniMax-H3-Fun-Controlnet-Union-2.0/
│       └──  MiniMax-H3-Fun-Controlnet-Union-2.0.safetensors

Then edit the settings on the high of examples/minimax_h3_fun/predict_v2v_control.py (or predict_v2v_control_inpaint.py for inpainting) and run it.

model_name          = "models/Diffusion_Transformer/MiniMax-H3"
config_path         = "config/minimax_h3/minimax_h3_control_inpaint_post_norm.yaml"
transformer_path    = "models/Diffusion_Transformer/MiniMax-H3-Fun-Controlnet-Union-2.0/MiniMax-H3-Fun-Controlnet-Union-2.0.safetensors"
control_video       = "your_control_video.mp4"
immediate              = "your prompt"

python examples/minimax_h3_fun/predict_v2v_control.py

Notes:

• config_path should be config/minimax_h3/minimax_h3_control_inpaint_post_norm.yaml. It builds the management department precisely because the checkpoint expects (control_blocks_places: [0, 5, 10, 15, 20, 25, 30, 35, 40, 45], control_in_dim: 49, control_apply_audio: false, inpaint_masked_pixel_mode: post_norm); the v1 minimax_h3_control.yaml (5 blocks) will silently drop half the management weights.
• For pure management (no inpaint enter) the pipeline zero-pads the masks channels, so this inpaint checkpoint nonetheless runs plain Canny/Depth/… management appropriately.
• The checkpoint is guidance-distilled: hold guidance_scale = 1.0; a worth above 1 applies steering twice and degrades the output.
• The management checkpoint carries solely the management department; the bottom MiniMax-H3 weights should be current in model_name.
• For the Layout situation, generate the management video with the Wan2.1-VACE format pipeline (see Supported control conditions); different control-video codecs are unchanged from v1.
• Memory: the transformer (about 62 GB) plus the Qwen3-VL textual content encoder (about 62 GB) don’t match one 80 GB GPU totally loaded; use model_group_offload (quickest) or model_cpu_offload_and_qfloat8 on a single 80 GB GPU.

License

This mannequin is a by-product of MiniMax-H3 and is launched beneath the MiniMax H3 Community License Agreement. Please learn the license rigorously, particularly the territorial restrictions and the Acceptable Use Policy, earlier than use.