AI GitHub
ControlNet for conditioning diffusion with spatial cues
Guide supported image generation models with inputs such as edges, poses or depth.
Key Takeaways
- ControlNet adds a trainable conditioning path to a diffusion architecture.
- Select a compatible control checkpoint and the preprocessor for its expected input.
- Generate with and without one edge map using the same prompt and seed.
- Control checkpoints need compatible base architectures and input types.
How it works
ControlNet adds a trainable conditioning path to a diffusion architecture. A preprocessor turns an input into a signal appropriate for the particular control checkpoint.
What you need
Select a compatible control checkpoint and the preprocessor for its expected input. Save the prepared conditioning image so later changes can be traced to the right stage.
What to try first
Generate with and without one edge map using the same prompt and seed. Vary control strength and inspect how closely composition follows the input.
Test a control input with deliberately incomplete edges or an awkward pose. Inspect whether the resulting image ignores the cue or copies its errors.
Where to look for failures
Control checkpoints need compatible base architectures and input types. Preprocessing errors can carry directly into the generation.
Related Franklin pages
Match the pipeline and device to a specific model variant. Compare a controlled prompt on known assets before testing a new checkpoint, conditioning input or export.
- Explore Black Forest Labs' official inference code and model references. FLUX reference code for image generation
- Convert supported diffusion models and run them through Apple's Core ML stack. Apple Stable Diffusion for Core ML deployment
Franklin’s image tests give you examples of outputs to compare across visual models. Franklin’s image tests
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗