跳到正文
arXiv cs.AI· Bohe Chen·· 2 小时前AI 评分49

提示词优化器审计研究:用演示条件提示搜索(DCPS)揭示简单生成-选择方法的潜力

Simplicity Goes Far: Auditing Prompt Optimizers with Demonstration-Conditioned Prompt Search

AI 导读

一项 arXiv 研究提出 Demonstration-Conditioned Prompt Search(DCPS),将生成-选择步骤作为控制机制,审计自动提示词优化器的真实收益。

正文 · 原文

View PDF HTML (experimental)

Abstract:Automatic prompt optimization is growing more elaborate. Recent methods stack policy-gradient updates, trace reflection, and evolutionary search. These methods share a generate-and-select step; without controlling for it, reported gains may conflate the contributions of optimizer mechanisms and larger sampling budgets. We isolate this step as Demonstration-Conditioned Prompt Search (DCPS), repurposing the Automatic Prompt Engineer (APE) recipe as a mechanism-attribution control: it freezes the generator, conditions it on randomly sampled demonstrations, scores candidates on validation data, and keeps the best---no learned update, reflection, or evolution. Across two optimizer families, DCPS replaces Proximal Policy Optimization (PPO) in StablePrompt, matching classification accuracy within 0.5 pp at roughly one-third the measured cost; in DSPy-based compound AI systems, under the reported budgets, it matches GEPA's macro accuracy on GPT-4.1-mini while using 55% fewer rollouts at 32% lower measured API cost on average. The audit reveals a task-dependent pattern: proposal and selection recover much of heavier optimizers' performance on structured and instruction-following tasks, while additional mechanisms retain advantages in some open-ended and reasoning-heavy settings. We recommend benchmarking prompt optimizers against generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy. Code and artifacts: this https URL.
Comments: 13 pages, 2 figures, 4 tables
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22605 [cs.AI]
  (or arXiv:2611.22605v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22605

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Bohe Chen [view email]
[v1] Tue, 17 Nov 2026 11:22:27 UTC (50 KB)

来源:arXiv cs.AI · arxiv.org