Use Generative AI For Content material Creation With ComfyUI

ComfyUI — an open-source, node-based graphical interface for working and constructing generative AI workflows for content material creation — revealed main updates this previous month, together with as much as 40% efficiency enhancements for NVIDIA RTX GPUs, and help for brand new AI fashions together with Wan 2.2, Qwen-Image, FLUX.1 Krea [dev] and Hunyuan3D 2.1.

NVIDIA additionally launched NVIDIA TensorRT-optimized variations of fashionable diffusion fashions like Stable Diffusion 3.5 and FLUX.1 Kontext as NVIDIA NIM microservices, permitting customers to run these fashions in ComfyUI as much as 3x quicker and with 50% much less VRAM.

Plus, an replace to NVIDIA RTX Remix — a platform that lets modders remaster basic video games — launched at the moment, including a sophisticated path-traced particle system that delivers beautiful visuals to breathe new life into basic titles.

ComfyUI v3.57 Gets a Performance Boost With RTX

NVIDIA has collaborated with ComfyUI to spice up AI mannequin efficiency by as much as 40%. To put this in perspective, GPU era upgrades usually solely ship a 20-30% efficiency enhance.

Developers concerned about optimizing efficiency and effectivity of diffusion fashions on their apps can learn extra on how NVIDIA helps speed up these workloads within the developer forum.

State-of-the-Art AI Models, Accelerated by RTX

Incredible fashions for AI content material creation have been launched within the final weeks, all of which are actually accessible in ComfyUI.

Wan 2.2 is a brand new video mannequin that gives unbelievable high quality and management for video era on PCs. It’s the most recent mannequin from Wan AI — a artistic AI platform providing a powerful lineup of AI fashions, together with Text to Image, Text to Video, Image to Video and Speech to Video. GeForce RTX and NVIDIA RTX PRO GPUs are the one GPUs able to working Wan 2.2 14B fashions in ComfyUI with out main delays in output. Check out the instance beneath created with the one immediate: “A robot is cracking an egg, but accidentally hits it outside the bowl.”

Qwen-Image is a brand new picture era basis mannequin from Alibaba that achieves important advances in complicated textual content rendering and exact picture modifying. It excels at rendering complicated textual content, dealing with intricate modifying and sustaining each semantic and visible accuracy in generated pictures. The mannequin runs 7x quicker on a GeForce RTX 5090 vs. Apple M3 Ultra.

Black Forest Labs’ new FLUX.1 Krea [dev] AI mannequin is the open-weight model of Krea 1, providing sturdy efficiency and skilled to generate extra life like, various pictures that don’t include oversaturated textures. Black Forest Labs calls the mannequin “opinionated,” because it provides all kinds of various, visually attention-grabbing pictures. This mannequin runs 8x quicker on a GeForce RTX 5090 vs. Apple M3 Ultra.

Hunyuan3D 2.1 is a completely open-source, production-ready 3D generative system that transforms enter pictures or textual content into high-fidelity 3D belongings enriched with bodily based mostly rendering supplies. Core parts embody a 3.3-billion-parameter mannequin for form era and a 2-billion-parameter mannequin for texture evaluation to shortly generate extra life like supplies. It all runs quicker on Blackwell RTX GPUs.

Get Started With Advanced Visual Generative Techniques

Visual generative AI is a robust device, however getting began might be tough even for technical specialists and studying to make use of extra superior methods usually takes months.

ComfyUI makes it simple to get began with superior workflows by offering templates or preset nodes that obtain a particular process, like protecting a personality fixed all through totally different generations, adjusting the sunshine of a picture or loading a fine-tuning. This permits even non-technical artists to simply use superior AI workflows.

These are 10 key methods to get began with generative AI:

• Guide video generation by defining the start frame and end frame: Upload a begin and finish body, in addition to how a video clip ought to start and finish. Wan 2.2 can then generate a easy, animated transition, filling the in-between frames to create a coherent animation. It’s splendid for making animations, scene shifts or defining poses.
• Edit images with natural language: Use FLUX.1 [dev] KONTEXT to edit particular textual content sections of a picture.
• Upscale images or videos: Take picture or video at a decrease decision and improve its decision and element high quality by including life like, high-frequency particulars.
• Control area composition: Assume extra granular management over picture era by controlling the association and structure of visible components inside particular areas of a picture.
• Restyle pictures: Use FLUX Redux to create totally different variations of pictures whereas preserving core visible components and particulars.
• Tap image-to-multiview-to-3D models: Use a number of pictures of an object captured from totally different angles to create a high-fidelity, textured 3D mannequin.
• Transform sound to video: Create video clips or animations instantly from audio inputs corresponding to speech, music or environmental sounds.
• Control video trajectory: Automatically information the movement of objects, digicam or scenes inside movies.
• Edit images with inpainting: Fill in or alter lacking or undesirable elements of a digital picture in a visually seamless, contextually constant approach with surrounding areas.
• Expand the canvas with outpainting: Generate new picture content material to increase the boundaries of an present picture or video footage, add element to cropped sections or full a scene.

Follow ComfyUI on X for updates to artistic templates and workflows.

Expand Your ComfyUI Zone

ComfyUI plug-ins allow customers so as to add generative AI workflows into their present purposes. The ComfyUI neighborhood has began constructing plug-ins for a few of the high fashionable artistic purposes.

The Adobe Photoshop plug-in enhances Photoshop’s native Firefly fashions by permitting customers to run their very own flows and choose specialised fashions for particular duties. Local inference additionally allows limitless generative fill with low latency.

The Blender plug-in — featured within the NVIDIA AI Blueprint for 3D-guided generative AI — permits customers to attach 2D and 3D workflows. Artists can use 3D scenes to manage picture generations or create textures in ComfyUI and apply them to separate 3D belongings.

The Foundry Nuke plug-in — just like the Blender — allows a connection between 2D and 3D workflows so customers don’t should alt-tab and swap between purposes.

The Unreal Engine plug-in allows ComfyUI nodes instantly within the Unreal Engine person interface to shortly create and refine textures for scenes utilizing generative diffusion fashions. See the instance beneath.

Run Hyper-Optimized Models for NVIDIA RTX GPUs in ComfyUI

The finest approach to make use of NVIDIA RTX GPUs is with the TensorRT library — a high-performance deep studying inference engine designed to squeeze most velocity out of the Tensor Cores in NVIDIA RTX GPUs.

NVIDIA has collaborated with the highest AI labs to combine TensorRT of their fashions, corresponding to Black Forest Labs’ models and Stability AI’s models. These fashions are additionally accessible quantized — a compressed model of the community that makes use of 50-70% much less VRAM and provides as much as 2x quicker inference whereas sustaining comparable high quality.

TensorRT-optimized fashions might be run instantly in ComfyUI by way of the TensorRT node, which at present helps SDXL, SD3 and SD3.5, in addition to FLUX.1-dev and FLUX.1-schnell fashions. The node converts the AI mannequin right into a TensorRT-optimized mannequin, after which generates a TensorRT-optimized engine for the person’s GPU — a map of find out how to execute that mannequin with optimum effectivity for his or her explicit {hardware} — offering important speedups.

Quantizing fashions, nevertheless, takes a bit extra work. For customers concerned about working quantized and TensorRT-optimized fashions, NVIDIA provides preconfigured recordsdata in a easy container referred to as a NIM microservice. Users can use the NIM node in ComfyUI to load these containers and use quantized variations of fashions like FLUX.1-dev, FLUX.1-schnell, FLUX.1 Kontext, SD3.5 Large and Microsoft TRELLIS.

Remix Update Adds a Path-Traced Particle System

A brand new RTX Remix replace launched at the moment by way of the NVIDIA app, including a sophisticated particle system that allows modders to boost conventional hearth and smoke results, in addition to extra fantastical results, like these within the online game Portal.

With RTX Remix, legacy particles from basic video games might be interpreted as path-traced, enabling them to forged life like gentle to boost the looks of many scenes. But finally, these particles have been nonetheless over 20 years outdated, missing element, aptitude and fluid animations.

RTX Remix’s new particles have bodily correct properties and might work together with a sport’s lighting and different results. This allows particles to collide, precisely transfer in response to wind and different forces, be mirrored on surfaces, forged shadows and assign their very own shadows.

For an entire breakdown of the brand new particle system, learn the GeForce article.

Each week, the RTX AI Garage weblog collection options community-driven AI improvements and content material for these seeking to study extra about NVIDIA NIM microservices and AI Blueprints, in addition to constructing AI agents, artistic workflows, productiveness apps and extra on AI PCs and workstations. 

Plug in to NVIDIA AI PC on Facebook, Instagram, TikTok and X — and keep knowledgeable by subscribing to the RTX AI PC newsletter. Join NVIDIA’s Discord server to attach with neighborhood builders and AI fans for discussions on what’s attainable with RTX AI.

Follow NVIDIA Workstation on LinkedIn and X. 

See notice concerning software program product data.