Rethinking Noise Shaping in Differentially Private Federated LoRA: Why Isotropic Perturbation Wins.

Differentially private federated fine-tuning with
LoRA typically perturbs adapter updates as homogeneous vectors, overlooking the asymmetric effects of its low-rank factors.
Because the effective update is (α/r)BA, perturbations in A
and B are amplified according to the norm of the complementary factor. We formalize this asymmetry through a block-wise
Gaussian mechanism with separate clipping thresholds and noise
scales, subject to ∑︁g Cg2/s2 g = 1/σ2, which preserves the privacy
guarantee of isotropic noise. Although the proposed shaping reduces modeled noise energy, experiments show that it consistently
degrades utility. We identify three causes: degenerate single round error modeling under standard LoRA initialization, crossround cancellation between perturbation and gradient repair,
and update-direction distortion introduced by per-block clipping.
Across twelve paired experiments at identical (ϵ, δ), isotropic
noise consistently outperforms shaping, with gains of 3.02 and
7.28 percentage points on attention and feed-forward projections,
respectively. These results support isotropic perturbation as the
stronger baseline for private federated LoRA.