分布偏移下的反事实模型测试方法研究
Counterfactual Model Testing Under Distribution Shift
AI 导读
一项 arXiv 研究提出了分布偏移下的在线模型测试方法，通过在已知标签的受控基线上复现偏移来估计模型当前性能。该方法区分总体变化与数据管道完整性问题两类机制，并估计修复数据完整性问题可带来的性能增益。真实数据集上模拟故障的实验表明，分离两类来源能改善估计效果，流失预测案例展示了其在监控与修复排序中的支持作用。
正文 · 原文
Abstract:Deployed machine learning (ML) models can degrade as the distribution of their input data shifts. Such shifts can result from changes in the population or from data integrity issues in the pipeline that materializes model features, two sources that operate through different mechanisms. Because labels are often absent or delayed, it is common practice to track shifts in the data instead, but a detected shift does not indicate how it affects the model. We formulate this problem as online model testing under distribution shift. Testing requires reproducing the shift on a controlled baseline with known labels. Since the same shift can come from either source, doing so requires knowing which mechanism changed. We propose a methodology that reproduces observed changes in both mechanisms to estimate current performance and the potential gain from repairing observed data integrity issues. Experiments on real-world datasets with simulated faults show that separating the two sources generally improves both estimates, supporting this formulation. A churn prediction case illustrates how these tests can support monitoring and repair prioritization
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33594 [cs.AI] |
| (or arXiv:2610.33594v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33594 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joran Leest [view email]
[v1]
Sat, 24 Oct 2026 10:42:31 UTC (59 KB)
来源：arXiv cs.AI · arxiv.org