arXiv cs.AI· Mohammad Reza Rezaei, Dingxian Wang, Brett Levert, Celeste Fejzo, Andrew Rabinovich, Zhao Chen·· 2 小时前AI 评分42
PAYWORTHY：LLM 能否预测哪些自由职业任务适合 AI 智能体完成
PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
AI 导读
arXiv 论文提出 PAYWORTHY 基准，基于某自由职业平台 2020 至 2026 年 1880 万条帖子中抽样的 1,494 条真实职位，测试 18 个模型能否预测 AI 智能体完成任务的成功概率。
正文 · 原文
Abstract:Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed. We test whether frontier language models can make this call before any attempt. We propose PAYWORTHY which contains 1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026 on a freelancing platform. An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work. We ask 18 models to forecast how likely the agent is to succeed and measure how well each model's forecasts rank the accepted jobs above the rejected ones (AUC, where 0.5 is a random ranking). The answer depends on how the agent is described. Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86). Told instead that it is a generic agent with a browser and code execution, the same models rank the same jobs about as well as a random ordering (0.45--0.56). These results hold under judges from four other model families (Claude, GLM, Kimi and Qwen), and the informed forecasts also rank the verdicts of two human raters.
Code is available at this https URL
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05759 [cs.AI] |
| (or arXiv:2611.05759v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05759 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohammad R. Rezaei [view email]
[v1]
Tue, 3 Nov 2026 23:59:53 UTC (1,440 KB)
来源：arXiv cs.AI · arxiv.org