当智能体也在学习：基于不可观测奖励的激励学习研究
When the Agent Learns Too: Learning to Incentivize with Unobserved Rewards
AI 导读
一篇 arXiv 论文研究委托人无法观测智能体真实奖励时的重复逆向选择博弈：智能体用多臂老虎机（MAB）最大化期望奖励加激励，委托人则并行训练算法，在估计智能体未知奖励与通过自适应激励引导其决策之间权衡。论文针对非参数模型提出仅依赖激励历史与智能体选择历史的估计器，在不限制智能体算法类型的前提下证明了估计器的有限样本一致性和委托人的严格遗憾界，并以绿色能源聚合商合同的仿真验证了框架适用性。
正文 · 原文
Abstract:In practice, incentive providers (i.e., principals) often cannot observe the reward realizations of incentivized agents, which is in contrast to many principal-agent models that have been previously studied. This information asymmetry challenges the principal to consistently estimate the agent's unknown rewards by solely watching the agent's decisions, which becomes even more challenging when the agent has to learn its own rewards. This complex setting is observed in various real-life scenarios ranging from renewable energy storage contracts to personalized healthcare incentives. Hence, it offers not only interesting theoretical questions but also wide practical relevance. This paper explores a repeated adverse selection game between a self-interested learning agent and a learning principal. The agent tackles a multi-armed bandit (MAB) problem to maximize their expected reward plus incentive. On top of the agent's learning, the principal trains a parallel algorithm and faces a trade-off between consistently estimating the agent's unknown rewards and maximizing their own utility by offering adaptive incentives to lead the agent. For a non-parametric model, we introduce an estimator whose only input is the history of principal's incentives and agent's choices. We unite this estimator with a proposed data-driven incentive policy within a MAB framework. Without restricting the type of the agent's algorithm, we prove finite-sample consistency of the estimator and a rigorous regret bound for the principal by considering the sequential externality imposed by the agent. Lastly, our theoretical results are reinforced by simulations justifying applicability of our framework to green energy aggregator contracts.
| Comments: | 72 pages, 6 figures. Companion paper to arXiv:2304.07407; the overlapping text covers the shared model setup |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT); Machine Learning (stat.ML) |
| Cite as: | arXiv:2308.06717 [cs.LG] |
| (or arXiv:2308.06717v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2308.06717 arXiv-issued DOI via DataCite |
Submission history
From: Ilgin Dogan [view email]
[v1]
Sun, 13 Aug 2023 08:12:01 UTC (251 KB)
[v2]
Wed, 4 Nov 2026 08:41:35 UTC (226 KB)
来源：arXiv cs.AI · arxiv.org