arXiv cs.AI· Shengtao Wen, Yunying Yang, Xiang Chen, Lingbing Guo, Yu Tian, Sheng-Jun Huang·· 5 小时前AI 评分27
SPARK：面向 LLM 持续学习的任务后自蒸馏回放方法，解耦知识与隐私
Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning
AI 导读
论文提出 SPARK，将隐私保护持续学习拆分为保留与校正两步：先用 Self-Distillation Replay 在学习当前任务的同时蒸馏先前任务行为并冻结任务后分布，再用 Post-Task Privacy Correction 在稳定参考上选择性降低已标注 PII 的生成概率，同时锚定当前与旧任务的非 PII 行为。
正文 · 原文
Abstract:Privacy-preserving continual learning (PPCL) must reduce the reproduction of sensitive content while retaining useful knowledge across sequential tasks. Formal privacy guarantees characterize randomized mechanisms, whereas operational output control concerns whether a trained model selectively reduces the likelihood of sensitive content in its outputs. In this work, we investigate the latter together with continual-learning utility under realistic task evolution. Retention and privacy correction operate at different granularities: task acquisition requires broad preservation of current- and old-task behavior, whereas privacy correction targets sparse annotated positions. Joint optimization leaves the current-task preservation target continually changing. We propose SPARK, a retention-correction decomposition that first freezes the learned post-task distribution and then applies selective correction around this stable reference. Self-Distillation Replay learns the current task while distilling behavior from previous tasks, and Post-Task Privacy Correction reduces annotated-PII likelihood while anchoring current- and old-task non-PII behavior to the resulting checkpoint. Extensive evaluations demonstrate that SPARK achieves effective selective PII suppression while preserving strong continual-learning utility and knowledge retention across diverse settings. Code and data will be released upon publication.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.29711 [cs.LG] |
| (or arXiv:2609.29711v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.29711 arXiv-issued DOI via DataCite |
Submission history
From: Shengtao Wen [view email]
[v1]
Mon, 31 Aug 2026 13:01:36 UTC (1,178 KB)
[v2]
Wed, 4 Nov 2026 08:23:54 UTC (1,938 KB)
来源：arXiv cs.AI · arxiv.org