The real consistency problem is usually upstream
When a Stable Diffusion portrait keeps changing the head shape, the issue is often not the prompt wording. The issue is that the model is inventing the angle, structure, and light all at once.
If the conditioning image is vague, the portrait will drift.
Start with a face reference instead of a finished image
FaceRef works well as the first step because it lets you set the head angle, choose a structure mode, and keep the facial proportions grounded before you ask the model to stylize anything.
This is different from using a polished portrait as the input. A clean structure-first guide gives ControlNet less noise and more intent.
A simple workflow
- set the camera angle in FaceRef
- choose whether you need a Loomis-style sketch or stronger planar guidance
- adjust the light until the large shadow pattern matches the mood
- export the reference that best suits your ControlNet mode
- iterate the prompt while keeping the same conditioning image
If you want the product-level overview first, visit ControlNet Face Reference.
When to use depth and when to use lines
Depth-style guidance is helpful when you care about volume and need the face to stay dimensional across multiple prompt passes.
Line or edge-heavy guidance is useful when you want sharper structural control or a more illustration-driven result.
The important part is not the label. It is whether the exported guide preserves the information your prompt should not be allowed to reinvent.
Keep lighting deliberate
Portrait consistency is not just about silhouette. If the light direction drifts, the face will often feel like a different person even when the angle is similar.
Set the key light intentionally before you export. FaceRef gives you a faster way to test the shadow family before Stable Diffusion starts inventing dramatic but inconsistent lighting.
Why FaceRef helps more than random reference packs
Photo packs are useful, but they are usually not built around the exact angle and light combination your scene needs. A controllable 3D head lets you build the control image for the shot instead of compromising with something close.
That is especially useful when you need multiple variants of the same portrait idea.
A practical checkpoint before every run
Ask four questions before you hit generate:
- Is the pitch correct?
- Is the yaw correct?
- Is the light doing one clear thing?
- Is the facial structure simplified enough to guide, but not noisy enough to confuse?
If those answers are clear, ControlNet usually has a much better foundation to work from.