What is DreamOmni2?

DreamOmni2 is an open-source artificial intelligence model designed to handle two complex tasks: multimodal instruction-based image editing and generation. This means the model can understand instructions given through both text and images, allowing users to edit existing images or create entirely new ones based on specific guidance. The model goes beyond traditional approaches by supporting both concrete objects (like people, animals, and physical items) and abstract concepts (such as textures, artistic styles, materials, makeup looks, and design patterns).

Released in October 2025 by researchers from leading institutions, DreamOmni2 represents a significant step forward in making AI image generation and editing more intuitive and accessible. The model was developed to address common limitations found in existing tools, where users often struggled to communicate their exact intentions using text alone. By accepting reference images alongside text instructions, DreamOmni2 bridges the gap between what users envision and what they can express in words.

The model is built on a unified framework, meaning it can handle both editing and generation tasks without requiring separate tools or workflows. This integration makes DreamOmni2 particularly valuable for creative professionals, researchers, and hobbyists who need flexible, powerful tools for visual content creation. The project is released under the Apache 2.0 license, making it freely available for research and commercial use in most regions.

Overview of DreamOmni2

Understanding the Two Core Tasks

Multimodal Instruction-based Generation

The generation task focuses on creating new images from scratch based on user instructions. Traditional subject-driven generation systems could only work with concrete objects, such as placing a specific person or product into a new scene. DreamOmni2 expands this capability significantly by supporting abstract attributes as well.

For concrete objects, the model excels at maintaining identity and pose consistency. If you provide a reference image of a person and ask the model to place them in a different setting, DreamOmni2 will preserve their facial features, body proportions, and general appearance while adapting them to the new context. This capability is particularly useful for applications like virtual try-ons, character design, and product visualization.

The real innovation comes with abstract attributes. Users can now reference elements like material textures (such as velvet, marble, or wood grain), artistic styles (impressionism, watercolor, or sketch), makeup looks, hairstyles, design patterns, and even lighting conditions. For example, you could show the model an image of a fabric texture and ask it to apply that texture to a piece of furniture, or reference a painting style and request that style be applied to a portrait. This flexibility opens up countless creative applications that were previously difficult or impossible to achieve with text-only instructions.

Multimodal Instruction-based Editing

Image editing presents different challenges than generation. When editing an existing image, the model must preserve all areas that should remain unchanged while making precise modifications to specific regions. Traditional instruction-based editing relied purely on text descriptions, which often proved inadequate for conveying specific visual details.

DreamOmni2 addresses this limitation by accepting reference images alongside text instructions. This multimodal approach allows for much more precise editing. For instance, if you want to change someone's outfit in a photo, you can provide a reference image showing the exact clothing item you want, rather than trying to describe it in words. The model will then replace the outfit while maintaining the original pose, lighting, and background.

The editing capabilities extend to both concrete and abstract concepts. You can replace objects, change materials, modify styles, adjust lighting, alter facial expressions, transform hairstyles, and apply various artistic effects. The model handles these edits while maintaining strict consistency in non-edited areas, which is crucial for professional applications where precision matters.

DreamOmni2 Visual Demo

Background Replace

Face Expression Editing

Font Replacement

Hair Style Editing

In-Context Generation

Lighting Rendering

Object Replacement

Pattern Imitation

Pose Imitation

Style Transfer

Technical Approach

The development of DreamOmni2 involved addressing two primary challenges: creating appropriate training data and designing a model framework capable of handling multimodal inputs effectively.

Data Synthesis Pipeline

The research team developed a three-step data synthesis pipeline to generate training examples. First, they used a feature mixing method to create extraction data for both abstract and concrete concepts. This process involved carefully separating visual attributes from source images to build a comprehensive dataset of extractable features.

Second, they generated multimodal instruction-based editing training data using the editing and extraction models. This step created examples showing how images should be modified based on various combinations of text and image instructions. The dataset includes diverse scenarios covering object replacement, style transfer, material changes, and other editing operations.

Third, they applied the extraction model to create additional training data for multimodal instruction-based generation. This process ensured the model could learn to create new images from scratch while following complex multimodal guidance.

Model Framework Design

To handle multiple image inputs effectively, the team implemented an index encoding and position encoding shift scheme. This mechanism helps the model distinguish between different input images and prevents confusion when processing multiple visual references simultaneously. Each image receives unique positional information that guides the model in understanding which elements to extract from which source.

The framework also includes joint training with a Vision Language Model. This integration improves the model's ability to parse complex instructions that involve both visual and textual information. The VLM component helps translate user intent into actionable processing steps, making the system more responsive to natural language requests.

Applications and Use Cases

Understanding Editing vs Generation

DreamOmni2 treats editing and generation as distinct tasks, each with specific requirements and outputs. Understanding this distinction helps users choose the appropriate mode for their needs.

Editing mode requires strict consistency in preserving non-edited areas of the source image. When you edit an image, everything except the specifically modified regions should remain exactly as it was. This precision is crucial for professional workflows where maintaining the original context and composition matters. The editing process focuses on making targeted changes while respecting the integrity of the source material.

Generation mode, in contrast, only needs to retain identity, intellectual property, or specific attributes from the reference images as specified in the instructions. The entire output image can be regenerated from scratch, with emphasis placed on creating an aesthetically pleasing result that incorporates the referenced elements naturally. Generation mode offers more creative freedom since it is not constrained by preserving specific pixels from a source image.

The research team separated these tasks because they found that instructions for editing and generation are often similar in form but different in intent. By explicitly distinguishing between the two modes, DreamOmni2 makes it easier for users to communicate their exact needs and get appropriate results. This separation also allows the model to optimize its processing for each specific task type.

Advantages and Limitations

Frequently Asked Questions