Preference Optimization: an Under-Dosed Magnet and an Unexpectedly Stable Architecture

The Test (H3): BLiMP minimal pairs ARE preference pairs (good ≻ bad sentence) — no reward model, no judge. Train DPO on three linguistic phenomena, evaluate on two unseen ones. The magnetic arm (MPO) is the same optimizer code path with a proximal pull toward the frozen base; arms differ only in the magnet strength τ.

The magnet result — honest PARTIAL: at pre-registered τ ∈ {1e-3, 1e-2}, MPO's held-out accuracy equals DPO's exactly (all seeds, both base models) and the KL reduction is insignificant. A pre-registered dose-response rider refuted its own prediction: even τ = 10 — a thousand times the registered maximum — reduces KL drift only 1.2% with accuracy unchanged. The magnetic pull is second-order to the DPO gradient across four orders of magnitude at this budget. (Consistent with pretraining finding F12: the magnet's natural home is stability, not preference-phase drift control.)

Secondary finding A — DPO damages unseen phenomena: on the explicit base model, trained-phenomena accuracy climbs 0.646 → 0.877 while unseen phenomena drop 0.740 → 0.612, at ~1.24 nats/token KL drift from the base. The reward-hacking worry, observed directly in the linguistic domain.

Secondary finding B — the equilibrium architecture barely moves: under the identical loss, learning rate, and steps, EqLM shifts its train accuracy only 0.493 → 0.516 with 1655× less KL drift (0.00075 vs 1.24 nats/token). Adversarial review excluded the trivial explanation (its base accuracy is low, so the preferences were not already satisfied): the cause is damped gradients through the 12-iteration weight-tied solve. Double-edged and reported as such — stability against preference-induced drift, and insensitivity to preference fine-tuning at matched learning rate. This is a property of the architecture, not of the magnetic term.