As is widely known, the ControlNet model for SD1.5 is not compatible with SDXL. So, what should we do if we need to use ControlNet with SDXL? Don't worry, this tutorial will show you how to use ControlNet with SDXL using the canny method as an example.
Updating Our ControlNet
First, we need to update our ControlNet. In the qiud叶子 integration package, click on Version Management – Extensions – One-click Update.
Alternatively, you can also update in the WebUI – Extensions – Installed – Check for Updates – Click Apply – Restart WebUI.
Downloading SDXL Control Models
You can download the ControlNet models suitable for SDXL via the following link.
lllyasviel/sd_control_collection at main (huggingface.co)
You can place the models in
stable-diffusion-webui\extensions\sd-webui-controlnet\models
or
stable-diffusion-webui\models\ControlNet
However, with so many models to choose from when downloading, how do we decide?
Canny Model
Taking the Canny model as an example, we don't need to download all of them. I have tested the generation effects and speeds of these models.
Speed Test
Model Size
Although diffusers_xl_canny_full has excellent effects, unfortunately, it is the largest (2.5 gigabytes) and takes longer to generate. If you are only concerned about quality and not time, and your computer has high specifications, you can consider this.
The kohya_controllllite control models are very small. Given their small size, they perform exceptionally well.
Recommendations for Canny SDXL
If you are satisfied with its large size and lower speed, use diffusers_xl_canny_full.
If you need smaller, faster models and can accept slight changes in style, use kohya_controllllite_xl_canny.
Use sai_xl_canny_128lora for a reasonable file size while minimizing changes in style.
Control weight parameters are crucial for generating good images. Most models require it to be below 1.
-------------------------------------------------------------------------------------------
To learn more about AI tutorials, please visit our tutorial website:
