SDXL + SDXL Turbo UNetLoader, LoRA, FB Cache, ControlNet support #680
+2,472
−109
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the nodes 'NunchakuSDXLUNetLoader', 'NunchakuSDXLLoraLoader' and 'NunchakuSDXLLoraStack'. Does not modify the base nunchaku library.
The file models/sdxl.py converts the state dictionary from the existing nunchaku SDXL checkpoint on huggingface into one that is compatible with ComfyUI's UNetModel, which is distinct from the diffusers 2DUNetConditionModel.
Compare this image generated with the new node (should be able to reproduce my workflow by dragging it into ComfyUI):

With one copied from https://comfyanonymous.github.io/ComfyUI_examples/sdturbo/:

Update 25/10/29:
LoRA nodes implemented. Includes "NunchakuSDXLLoraLoader" and "NunchakuSDXLLoraStack". Because I converted everything to work with ComfyUI's UNetModel (and not diffusers 2DUNetConditionModel used in the base nunchaku repo), my LoRA implementation does not require any update to nunchaku library.
LoRA and LyCORIS should work. DoRA is not supported right now, I'm not sure how to apply the magnitude_vector DoRA layer to the quantized qweight Nunchaku layer.
From my testing, most LoRAs work but some don't. For example both the Lightning and LCM LoRAs cause horrible image quality.
Additionally, it appears ControlNet sort of works (can use standard ComfyUI Controlnet nodes, and sometimes get okay results), but see Known Issues at bottom.
Update 25/10/30:
First-block cache implemented.
Known issues: