LLM 对齐的排序感知差分隐私列表式偏好发布方法
Rank-Aware Differentially Private Release of Listwise Preferences for LLM Alignment
AI 导读
arXiv 论文提出一种排序感知的指数机制，用于在本地差分隐私（LDP）下发布 LLM 对齐所需的列表式偏好数据。该机制利用排序信息采样固定大小的二分划分，将噪声集中在边界项附近，而非像 randomized response（RR）那样均匀扰动所有项。在相同隐私预算下，该机制在下游 LLM 对齐任务中持续优于现有 LDP 基线，代码已开源。
正文 · 原文
Abstract:Listwise preference feedback offers richer supervision for large language model (LLMs) alignment than pairwise comparisons, but a full ranking also reveals more sensitive user preferences. Existing privacy-preserving alignment methods focus mainly on pairwise feedback, while listwise alignment under local differential privacy (LDP) remains unexplored. Extending LDP to this listwise domain raises a critical challenge: the high sensitivity of the full ranking space demands excessive noise, neutralizing the benefits of listwise supervision. To address this challenge, we analyze the interaction between supervision granularity and privacy noise in downstream training, and introduce a rank-aware exponential mechanism that privatizes listwise preference data into a low-sensitivity granularity suitable for downstream alignment. The mechanism leverages ranking information to sample a fixed-size binary partition, concentrating the noise near borderline items instead of perturbing all items uniformly as randomized response (RR) does. Empirical evaluations on downstream LLM alignment tasks show that our mechanism consistently outperforms existing LDP baselines at matched privacy budgets. The code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37682 [cs.CL] |
| (or arXiv:2610.37682v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37682 arXiv-issued DOI via DataCite |
Submission history
From: Junwei Chen [view email]
[v1]
Tue, 13 Oct 2026 04:17:15 UTC (508 KB)
来源：arXiv cs.AI · arxiv.org