跳到正文
arXiv cs.AI· Yunxiao Shi, Shuo Yang, Yixin Su, Rui Zhang, Min Xu·· 2 小时前AI 评分39

跨轮次偏差分析:ACC-SGD-IE 修正 SGD 影响估计的累积误差

Cross-Epoch Bias in SGD-based Leave-One-Out Influence: Analysis and Accumulative Correction

AI 导读

论文提出累积修正估计器 ACC-SGD-IE,解决多轮次 SGD-IE 在近似有限删除时忽略早前删除对后续删除影响的交互误差,利用样本后续出现处的 Hessian 修正参数变化估计。

正文 · 原文

View PDF HTML (experimental)

Abstract:Training-data influence estimates support data selection and cleansing by predicting the effects of removing individual examples. The standard multi-epoch SGD-Influence Estimator (SGD-IE) aggregates contributions from a sample's repeated appearances along the recorded trajectory. When used to approximate finite deletion, it omits an interaction: later removals act on the parameter displacement accumulated from earlier removals. We introduce the Accumulative SGD-Influence Estimator (ACC-SGD-IE) to account for how earlier removals change the effect of removing the same sample again. It uses the sample's Hessian at subsequent appearances to correct the estimated parameter change. Our non-convex analysis shows that this correction removes SGD-IE's additional interaction contribution from the finite-horizon error bound: the omitted interaction produces a local error linear in the accumulated parameter change, while the remaining Taylor remainder is bounded quadratically under Lipschitz-Hessian assumptions. The bounds characterize how training propagates the remaining error. In non-convex experiments across clean and noisy tabular, text, and vision datasets, ACC-SGD-IE reduces validation-loss-change RMSE by 10.8% and improves Jaccard@10% by 15.2% on average relative to SGD-IE. These gains also translate into lower test misclassification rates after data cleansing on MNIST and CIFAR10 at the evaluated removal budgets. On noisy industrial datasets, it reduces aggregate parameter RMS error with a mean relative reduction of 53.4%. We further discuss computational tradeoffs, provide complementary theoretical analysis and experimental evaluation for strongly convex objectives, and explore potential extensions of the accumulative correction mechanism to other influence estimators.
Comments: 24 pages, including appendices. Updated theoretical analysis and experiments. Code: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2510.26185 [cs.LG]
  (or arXiv:2510.26185v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2510.26185

arXiv-issued DOI via DataCite

Submission history

From: Yunxiao Shi [view email]
[v1] Thu, 30 Oct 2025 06:45:22 UTC (5,015 KB)
[v2] Mon, 16 Nov 2026 23:52:47 UTC (5,641 KB)

来源:arXiv cs.AI · arxiv.org