Hierarchical Alignment：块级 LoRA-DPO 用于大语言模型偏好微调

View PDF HTML (experimental)

Abstract:Preference tuning typically distributes adapter updates across every Transformer layer. We study how the placement of a fixed LoRA budget affects Direct Preference Optimization (DPO). Hierarchical Alignment partitions decoder depth into Local, Intermediate, and Global regions and restricts LoRA updates to a chosen contiguous block. Across Qwen and Llama models, three-seed sliding-window sweeps reveal broad low-loss basins around intermediate depth. Under exact trainableparameter matching, Middle achieves lower held-out DPO loss than an all-depth allocation on Qwen2.5-7B and Llama-3.1-8B. Generationlevel evidence is more variable. LLM judges and a blinded human study leave Selected versus All unresolved on Qwen; on Llama, two pairwise judges favor All while the human interval includes zero. Trained-adapter interventions show that learned depth components affect the optimization objective; removing the Llama Selected window also lowers generation quality. Across Qwen2.5 at 3B, 7B, and 14B, Middle versus All remains unresolved under Qwen3-Max; at 3B and 14B, both improve over Base. In our implementation, Middle also uses less peak memory and takes shorter steps than a parameter-matched all-depth allocation. Together, the results support intermediate depth as a practical starting point for fixed-budget preference tuning, though the best window and its behavioral value still need per-setting validation.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2510.12044 [cs.CL]
	(or arXiv:2510.12044v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2510.12044 arXiv-issued DOI via DataCite

Submission history

From: Yukun Zhang [view email]
[v1] Tue, 14 Oct 2025 00:58:34 UTC (2,443 KB)
[v2] Sat, 24 Oct 2026 15:54:38 UTC (137 KB)