arXiv 论文审计分布强化学习的不确定性可信度：40-95% 风险主张被证伪

View PDF HTML (experimental)

Abstract:Many learned predictive models represent their uncertainty as a probability distribution, and planners and policies use this distribution to judge risk. However, the fact that a model outputs a distribution does not tell us whether that distribution captures the uncertainty in the environment which is relevant for a decision. In this work we study this question in distributional reinforcement learning, where the learned return distribution P_{\theta}(Z | s,a) can be compared directly with an environment-grounded distribution P_env(Z | s,a) estimated by snapshot-restart Monte Carlo. Our audit first screens for the claims that a risk-sensitive user would act upon, using the excess Wasserstein gap between the top two actions (which is equal to the mass by which first-order stochastic dominance is violated). Each claim is then tested against ground truth using permutation nulls, bootstrap refutation and FDR control. Across QR-DQN, C51 and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, their placement is statistically indistinguishable from a truth-blind placement, and almost none of them can be confirmed. A similar pattern is seen for a pretrained full-Atari QR-DQN. On positive controls of known magnitude the same audit confirms 96-100% of real claims (correlation 0.89-0.92), so it is able to detect real risk when it is present. Acting on the learned CVaR advice at the most-flagged states gives results ranging from beneficial to significantly worse than chance. Ensembling, risk-sensitive training and recalibration do not fix the problem, and the recalibration experiment indicates that the learned spread has no usable information about the true risk. Based on these results, we suggest that uncertainty estimates, including those of learned world models, should be checked against realized outcomes at the decisions where they are used..

Comments:	25 pages, 8 figures, 3 tables (main text); includes supplementary material
Subjects:	Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML)
ACM classes:	I.2.6; G.3
Cite as:	arXiv:2607.11607 [cs.AI]
	(or arXiv:2607.11607v2 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2607.11607 arXiv-issued DOI via DataCite

Submission history

From: Hari Prasad [view email]
[v1] Mon, 13 Jul 2026 14:30:01 UTC (1,894 KB)
[v2] Thu, 12 Nov 2026 08:34:24 UTC (1,892 KB)