跳到正文
arXiv cs.AI· Hua Yang, Aditya Bhardwaj, Shidong Pan, Bowen Xu·· 4 小时前AI 评分41

PrivacyPR:面向隐私感知代码审查的真实隐私修复基准数据集

PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review

AI 导读

研究者发布 PrivacyPR 数据集,包含来自 1,469 个 GitHub 仓库的 4,131 个已合并 PR,其中 1,169 个高置信度隐私修复,并附 Verified(250 修复 + 250 非修复)和 Small(100 + 100)人工校验子集,每个 PR 均带 LLM 生成的隐私相关性、修复类型与 GDPR 原则标签。

正文 · 原文

View PDF HTML (experimental)

Abstract:Coding agents increasingly review code, yet whether they catch the privacy defects that laws such as the GDPR require software to avoid remains largely untested, partly because few datasets connect real privacy defects to their accepted fixes and review context. We present PrivacyPR, a dataset of 4,131 merged GitHub pull requests (PRs) from 1,469 repositories, created between June 2018 and January 2024. Each PR includes its commits, diffs, review discussion, and LLM-generated labels for privacy relevance, fix type, and GDPR principles. PrivacyPR provides 1,169 high-confidence privacy fixes and 1,643 non-fix PRs, together with Verified, a manually checked split of 250 privacy fixes and 250 non-fix PRs, and Small, a 100 + 100 subset for evaluation. On Verified, both annotators endorsed 72% of the LLM's GDPR-principle assignments. Beyond supporting empirical study of how privacy defects arise and are fixed, PrivacyPR provides a benchmark for privacy-aware code review on real, developer-accepted fixes. Its evaluation harness reverts each fix to reintroduce the defect. On Small, two coding agents detect only 42-51% of defects under a generic review prompt, but 88-91% when explicitly asked to check privacy, while flagging 11-15% of non-fix PRs. These results show that privacy defects can be detected by coding agents, but that detection depends strongly on review intent and comes with false alarms. Data, code, reviews, judge verdicts, and an extensible evaluation harness are publicly available at this https URL.
Comments: 4 pages
Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Cite as: arXiv:2611.18293 [cs.CR]
  (or arXiv:2611.18293v1 [cs.CR] for this version)
  https://doi.org/10.48550/arXiv.2611.18293

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Hua Yang [view email]
[v1] Fri, 13 Nov 2026 15:18:33 UTC (86 KB)

来源:arXiv cs.AI · arxiv.org