ReBRAC-v2：经典行为正则化 actor-critic 的系统性升级回归

View PDF HTML (experimental)

Abstract:Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value guidance. We ask how far systematic modernization of a conventional behavior-regularized actor-critic can go while preserving algorithmic simplicity. ReBRAC-v2 directly trains an exact-likelihood normalizing-flow actor with likelihood, MSE, and MAE behavior regularization, classification-based residual critics, staged optimization, and multi-sample inference. We develop one shared configuration through roughly 600 Bayesian proposals on six OGBench tasks, then adapt only two behavior-regularization coefficients using a 16-point grid for the main comparisons. Across ten common OGBench categories, ReBRAC-v2 averages 74.8 versus 52.3 for the next-best method and ranks first in eight categories. Without structural changes, it also obtains the strongest compared averages on D4RL AntMaze (90.2) and Adroit (33.6). Motivated by these results, we extend evaluation to all 235 state-based single-task problems across the 47 dataset variants in the original OGBench suite. To our knowledge, this is the first complete evaluation in this setting; no compared baseline reports results across the full suite. Per-task scores and configurations establish ReBRAC-v2 as a reference baseline for future work beyond the common task subset. The complete cloning objective retains the highest average after coefficient retuning of its alternatives, while further ablations characterize training, capacity, and inference choices. These results demonstrate the strength of systematic engineering within a minimalist offline RL foundation.

Submission history

From: Denis Tarasov [view email]
[v1] Sun, 2 Aug 2026 12:42:20 UTC (242 KB)
[v2] Fri, 30 Oct 2026 11:58:18 UTC (243 KB)