深度学习性能的实证研究：跨架构与任务的性能分布分析
Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks
AI 导读
一项发表于 Sci. Rep. 的实证研究通过 186 组实验、每组以不同随机种子运行 100 次、共 18,600 次运行，分析深度学习在图像分类与时间序列预测上的性能分布。结果显示性能分布常呈非高斯，时间序列模型的低性能离群值约为图像分类的近三倍，尾部风险更高；平均性能不能预测稳健性。研究建议模型选择不能只看均值，须评估分布的离散度、对称性与尾部风险。
正文 · 原文
Abstract:Context: Non-determinism in deep learning algorithm design and implementation leads to performance variation, meaning model performance is not a single value, but rather a distribution. These model performance distributions are underexplored despite their impact on robustness. Objective: We investigate the robustness of deep learning performance to sources of non-determinism, specifically focusing on how performance distributions differ across various architectures and tasks. Method: We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds to generate performance distributions, resulting in 18,600 runs. Robustness was quantified using metrics for spread, symmetry, and tail risk. Results: Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness -- larger image classification models show fewer outliers but not lower spread, while smaller time series models show lower spread but more extreme underperformers. Training duration does not scale linearly; early stopping effectively balances performance and robustness. Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none. Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. Conclusions: Tail risk poses serious concerns for Trustworthy AI in high-stakes applications. Models performing well on average may exhibit long tails and extreme outliers revealed only through distributional analysis. Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38037 [cs.LG] |
| (or arXiv:2610.38037v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38037 arXiv-issued DOI via DataCite |
|
| Journal reference: | Sci. Rep. 16, 20474 (2026) |
| Related DOI: | https://doi.org/10.1038/s41598-026-49656-z
DOI(s) linking to related resources |
Submission history
From: Kevin Coakley [view email]
[v1]
Tue, 27 Oct 2026 22:47:10 UTC (720 KB)
来源：arXiv cs.AI · arxiv.org