Feature/qwen eligen support

[nolan4]

Pull Request: Add Entity-Level Image Generation (EliGen) for Qwen Image

Summary

This update implements Entity-Level Image Generation (EliGen) for the Qwen Image model, allowing region-specific prompts through spatial masks. The feature provides fine-grained control over image generation by applying separate attention masks for each entity.

Key Features
• Spatial attention masking with isolated entity prompts
• Automatic mask resizing to match latent dimensions
• RoPE embedding implementation aligned with DiffSynth Studio
• Support for batch_size > 1
• Backward compatible with standard Qwen Image workflows

[krigeta]

Hey @nolan4 thank you so much for this amazing work! I am trying to test this out but not able to run it, I am using this PR and using the Eligen Lora provided by Diffsynth Studio and using this workflow, please check.

[krigeta]

hey @nolan4 may you please reply?

[nolan4]

Hi @krigeta — here’s a screenshot of my workflow, which is based on the Qwen text-to-image template. I’m also using the same EliGen LoRA from Diffsynth Studio that you linked. Hope this helps you get it running!

[krigeta]

Hey @nolan4 thank you so much for this, what if this branch is not merged then is it possible to create a custom node of this?

and yeah i will test this and share the results for sure.

[krigeta]

hey @nolan4 it is not working in my case, please check.

[nolan4]

Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications:

[Kosinkadink]

test locally and works, comfy will do a code review to see if anything else needs to be changed!

[krigeta]

Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications:

Will gonna test it asap and this time I am gonna test the official example as shown in the official Eligen lora implementation and one more thing I want to ask:

In the Diffsynth studio repo, they said it is important to make those colored masks with text overlays. Is that true?

As in their official example, the masks are overlaid to achieve smooth results.

[Amazon90]

How can I get this node?

[krigeta]

How can I get this node?

you have to install this PR and not the actual comfyUI setup to use this node, as of now it is not the part of the main repo.

[krigeta]

Hey @nolan4, I guess this implementation is missing the colour-coded masks that help the lora to differentiate between the regions when they overlap. Please look into it.

[geroldmeisinger]

btw something similar was implemented in the Inspire Pack called "regional conditioning by color masks" in case you need inspiration or code
https://github.com/ltdrdata/ComfyUI-extension-tutorials/blob/Main/ComfyUI-Inspire-Pack/workflow/RegionalCFG-RegionalConditioning.png

https://github.com/ltdrdata/ComfyUI-Inspire-Pack

[geroldmeisinger]

How can I get this node?

comfy-cli --workspace ./ComfyUI_eligen install --pr "#10473"
comfy-cli --workspace ./ComfyUI_eligen launch

[geroldmeisinger]

Looks like you have multiple loras, try just one for testing purposes. I tried your prompt with a few minor modifications

I can confirm it works: 8step lora -> eligen lora -> ksampler

I guess this implementation is missing the colour-coded masks that help the lora to differentiate between the regions when they overlap. Please look into it.

I can confirm it works

eligen_colorspheres.json

note i use euler+beta, cfg=1, 1328x1328 but that shouldn't make much difference

[krigeta]

Hey @geroldmeisinger , thank you so much for the share, is there any other social media where we can chat?

[geroldmeisinger]

here are the original masks https://www.modelscope.cn/datasets/DiffSynth-Studio/examples_in_diffsynth/files data/examples/eligen/entity_control
of the eligen demo page https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen

[krigeta]

here are the original masks https://www.modelscope.cn/datasets/DiffSynth-Studio/examples_in_diffsynth/files data/examples/eligen/entity_control of the eligen demo page https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen

These are the ones I got to know about Eligen. In my case, the overlapped masking is not working properly, as I want to create two characters in front of each other, and the view is from the back. Or I would say I use one entity for a character and another for another character, and with ControlNet? Does it work?

And if possible, may you share your Discord or other social app?

[Kosinkadink]

Apologies for the delay, I got comfy's review:

1. The eligen_attention_mask is already passed as a parameter into the block as the attention_mask - is the use of transformer_options to store that necessary? it seems redundant. if that is the case, could you remove it from being inserted into the transformer_options? Let us know if there is something that prevents this from working!
2. For logging, our pattern is to import logging and then just do logging.info(...), logging.debug(...) instead of initializing a logger. Could you change the code here to do that too? You can take a look at other ComfyUI code for clarity.
3. Could you restore the removed whitespace from model_base.py?

I opened up comments in the relevant parts of code!

[geroldmeisinger]

[krigeta]

Haha indeed, they are needed btw, I want to ask, are you able to get the result shown here?

[nolan4]

@geroldmeisinger @krigeta

Latest version supports 8 entities! Below is a screenshot for the referenced DiffSynth example.

[geroldmeisinger]

awesome!
see https://docs.comfy.org/custom-nodes/backend/more_on_inputs#dynamically-created-inputs in case you want to make it dynamic

@krigeta krigeta/eligen_test#1

[Kosinkadink]

There will be a native growing input type soon - any custom javascript to do this will not be allowed in core outside of a 'general' implementation! That will allow an indefinite amount of inputs into the node.