Mind the RefGAP: Correcting Reference Attention in Diffusion-Based Visual Editing

Yanan Wang1,2  ·  Shengcai Liao3  ·  Guangyi Liu1,2  ·  Xiaodan Liang1
1Mohamed bin Zayed University of Artificial Intelligence   2Institute of Foundation Models (IFM), MBZUAI   3United Arab Emirates University
Paper (arXiv)

Diffusion editors often barely attend to the reference. RefGAP measures that attention on the fly and boosts it inside the edit, suppresses it outside.

Training-free

0 weight updates · plug-in logit offset at inference

Cross-model

7 image & video editors · 3 never seen in calibration

Cross-task

4 tasks · one config tuned only on head swap

Same settings, every model, every task

Each example shows Reference · Input · Base · RefGAP. Pick a task and an editor — the configuration never changes. Videos loop together; click any clip to pause or resume all.

Task
Editor

Examples are drawn from test clips where RefGAP improves the task metric (and, for identity tasks, keypoint error does not worsen by more than 0.01).

The problem

Editors see the reference — but hardly attend to it

In-context editors put reference tokens in the same attention sequence as the content being edited. Yet when we measure the share of attention that edit-region queries give to the reference, it is often far below what even uniform attention would give.

<1%

of edit-region attention mass goes to the reference in LoomVideo.

Edit-region reference attention vs. uniform

uedit = measured share ÷ uniform share · head-swap validation · line = 1.0
Method

Measure, then correct — two-sided and online

At every selected layer and denoising step, RefGAP reads the reference-attention mass for the edit region (E) and the keep region (K), and adds a single scalar to the reference-key logits of each query.

RefGAP pipeline: measure reference attention per region, compute dynamic bias, correct reference logits
1

Measure

Average each query's attention on reference keys, separately over edit and keep regions.

ρ̄edit, ρ̄keep
2

Set the bias

Low reference attention in the edit → larger boost. Keep region gets a suppressing offset so the change stays local.

bedit = +γedit·log(1/ρ̄edit)
bkeep = −γkeep·log(1/ρ̄keep)
3

Correct the logits

Shift reference logits only; relative weights inside each key group are preserved. A fixed rule picks all layers or the second half.

sij ← sij + bg,  j ∈ R

Shared everywhere: γedit=0.6, γkeep=0.8, τB=2.7, τb=3.7. A new editor only needs one profiling run to pick its layer set.

Results

Consistent gains without per-model tuning

Base → RefGAP on full test sets. Held-out editors were never used for calibration.

+0.199
mean identity gain, head swap (7 editors)
+0.237
mean identity gain, face swap (7 editors)
≥86%
head-swap replacement rate on every applicable editor
7 / 7
editors with better try-on garment fidelity
BaseRefGAPRefGAP (lower)

Human evaluation — reference fidelity

54 raters · 219 items · judgments: RefGAP better / tie / Base better
RefGAP betterTieBase better

vs. hand-tuned constant bias

A Diptych-style constant edit bias, swept separately per editor. Head-swap ID, 1,040 test clips.

RefGAP matches per-model tuned constants (Δ ≤ 0.006) — without any strength sweep.