FLUX-Controlnet-Inpainting, this image restoration tool based on ControlNet and FLUX.1-dev, is redefining our understanding of image restoration.
This tool not only inherits the high-quality image generation capabilities of the FLUX.1-dev model but also cleverly integrates the strengths of ControlNet. It can accurately repair images based on information such as edges, line drawings, and depth maps, generating content in specified areas that harmoniously blends with the surrounding environment, bringing new life to damaged or missing parts of images.
Recently, FLUX's Inpainting ControlNet has achieved a breakthrough in inference capabilities through integration with ComfyUI. This means users can now perform complex image restoration tasks in an intuitive interface, enjoying a smooth workflow. However, this powerful functionality comes with corresponding hardware requirements. When using t5xxl-FP16 and flux1-dev-fp8 models for 28-step inference, approximately 27GB of GPU memory is required. Nevertheless, the inference speed remains satisfactory: at cfg=3.5, it takes only 27 seconds; if cfg is reduced to 1, it can be shortened to 15 seconds.
For users seeking faster speeds, Hyper-FLUX-lora offers an excellent option, significantly enhancing inference efficiency. Additionally, by fine-tuning key parameters such as control-strength, control-end-percent, and cfg, users can further optimize the restoration effect. For example, setting control-strength to 0.9, control-end-percent to 1.0, and cfg to 3.5 often achieves an ideal balance.
The FLUX model is trained on a large dataset, including 12M laion2B and other internal image sources. For optimal results, a resolution of 768x768 is recommended for inference. The adjustment ratio for the control net should be maintained between 0.9 and 0.95, allowing for sufficient creative freedom while maintaining control.
It is worth mentioning that the current version is only an Alpha test version. The development team promises more powerful updates in the future, which undoubtedly makes us look forward to the future of FLUX-Controlnet-Inpainting. With continuous technological advancements, we have reason to believe that the field of image restoration will see more astonishing breakthroughs.
Project link: https://github.com/alimama-creative/FLUX-Controlnet-Inpainting