HOMIE Operational

HOMIE is an open-source digital human video generation framework from the Hong Kong University of Science and Technology, built on the Wan2.1-T2V-14B backbone and integrated with the Qwen3-VL multimodal large language model. The framework uniformly handles four elements—digital humans + products + logos + text—precisely addressing four major challenges: human-object interaction, brand logo placement, OCR text fidelity, and multi-view consistency. Model training requires only 5.5K steps / approximately 10K A100 hours. OCR accuracy improves by 38.7% over the strongest open-source model, and the framework supports inference from 480P on a single GPU to 720P on multiple GPUs, significantly lowering the barrier to video production for e-commerce livestreaming, virtual hosts, and other scenarios.

Personalized human-object interaction videos: Supports multiple people and various objects appearing in the same video while maintaining the appearance consistency of each subject and enabling natural, reasonable interactions, such as holding, displaying, and handing over products.
Precise logo / abstract concept attachment: Using the semantic reasoning capabilities of a multimodal large language model (MLLM), it automatically attaches abstract concepts such as brand logos to the objects most semantically relevant, without requiring explicit descriptions of positional relationships in the prompt.
High-fidelity OCR text generation: With OCR reference images, it ensures that text on product packaging and labels remains clear and readable in generated videos, without blurring or character mixing, addressing the common problem of “text turning into a blur” in AI video generation.
Multi-view-consistent generation: Given multi-view reference images of the same object, it maintains a consistent appearance and details when the object rotates or moves in the video, making it suitable for displaying figurines, 3D models, dolls, and other items.

Follow WeChat and reply “开源” to join the AI open-source project discussion group
Overall architecture: Uses the DiT of Alibaba's Wan2.1-T2V-14B as the backbone, introduces Qwen3-VL-2B-Thinking as the visual understanding module, and combines it with the UmT5 text encoder and VAE visual encoder to form a three-way parallel generation architecture incorporating text + multimodal semantics + vision.
MLLM integration strategy: Without disrupting the original text-to-video alignment or incurring expensive realignment costs, it injects the multimodal model's semantic reasoning capabilities into the DiT, enabling the model to automatically understand latent relationships between reference images.
Global multimodal self-attention guidance (GMG): In the DiT self-attention layers, it globally aligns semantic features extracted by the MLLM with VAE visual tokens. Through projection, pooling, and affine transformation, every video token during generation can “see” the global semantic information of the reference images.
Modality-reference embedding (MRE): Assigns modality embeddings separately to MLLM feature tokens and VAE visual tokens, assigns independent reference embeddings to different reference images, and associates multi-view/OCR reference image tokens of the same subject to avoid information confusion.
Three-stage progressive training: Single-subject 480P → multi-subject 480P → high-definition 720P, with training completed in only 5.5K steps, far fewer than the tens of thousands of steps required by similar methods.

Environment setup: Clone the HOMIE code repository from GitHub to your local machine and install the required dependencies.
Download weights: Download the officially released pretrained weights from HuggingFace to the designated model directory.
Prepare materials: Prepare reference images, including photos of people, product images, logo images, OCR text images, and other multi-subject materials.
Write prompts: Write text prompts that accurately describe the interactions and scene between people and objects in the target video.
Run inference: Run the inference script. HOMIE will automatically fuse the multimodal reference information and generate a personalized video.
Select resolution: A single GPU can directly generate videos at 832×480 resolution, while multi-GPU FSDP can scale to 720P high-definition output.

Unified framework: One framework handles four major tasks—human-object interaction, logo placement, OCR text fidelity, and multi-view consistency—without switching between different models for different scenarios.
Efficient training: Training can be completed with only 5.5K steps / approximately 10K A100 hours, far below the 30–40K steps typically required by similar methods.
Lossless MLLM integration: Injects the multimodal model's semantic reasoning capabilities into the video generation pipeline without sacrificing the controllability of the text encoder or incurring expensive realignment costs.
Leading quality: OCR accuracy improves by 38.7%, multi-view consistency improves by 8.2%, and in a user study with 40 participants, 64.7% voted the overall quality as the best.
Flexible inference: Supports inference from 480P on a single GPU to 720P on multiple GPUs, and can output 1280×720 portrait videos, directly supporting livestream commerce and short-video production.

Official website: https://yiyangcai.github.io/homie-page.github.io
GitHub repository: https://github.com/YIYANGCAI/HOMIE
HuggingFace model repository: https://huggingface.co/yychai/homie-r2v-wan2.1
arXiv technical paper: https://arxiv.org/pdf/2607.18217

E-commerce livestream commerce: Generate promotional videos featuring a specified digital human holding a specified product and performing specified actions, with 1280×720 portrait output that directly supports short-video platforms.
Brand advertising production: Automatically attach a brand logo to semantically relevant products without explicitly specifying positional relationships in the prompt.
Virtual hosts / digital humans: Maintain the identity consistency of a digital human while enabling natural interactions with various products.
Multi-view product display: Given multi-view reference images of a figurine, 3D model, or doll, generate a rotating showcase video with a consistent appearance from every angle.
Packaging text fidelity: With OCR reference images, ensure that text on product packaging and labels remains clear and readable in the video, suitable for foods, cosmetics, health products, and other scenarios requiring the display of ingredients or instructions.