本地开源权重 LLM 智能体数据工程能力实证评估：移动出行工作流研究

View PDF HTML (experimental)

Abstract:Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasise textual responses or isolated code rather than complete engineering artifacts.
Objectives: We evaluate whether local LLM agents can produce correct and reliable data-engineering artifacts, isolate the contribution of a closed-loop workspace and of input-structure knowledge, and examine trade-offs in runtime, cost, and failure behaviour.
Methods: We introduce a benchmark of fifteen mobility data-engineering tasks with deterministic checkers for the scripts, tables, structured files, figures, and reports they produce. Sixteen configurations from six model families are evaluated in three conditions (chat-only, chat-schema, agent-with-tools) with five repetitions per model, condition, and task, yielding 3,600 scored attempts on a consumer-grade GPU.
Results: A 4-billion-parameter model at 4-bit reaches 77.3%, a 9-billion-parameter model 80.0% within about 6.5 GB, and a 12-billion-parameter model from another family 81.3%, against 98.7% for the highest-scoring configuration, which solves 14 of 15 tasks in all five repetitions. Above two billion parameters the workspace loop raises pass rates by 24.0 to 65.3 points over uninformed one-shot generation, and its contribution exceeds that of the input-structure summary in eleven of fifteen tasks.
Conclusion: Open-weight models that fit on one consumer GPU support a meaningful share of data-engineering work once given a workspace loop, although capability still separates usable from unusable configurations and high-stakes outputs need validation. Being mobility-specific and checker-based, the benchmark characterizes the evaluated configurations, not all data-engineering practice.

Comments:	34 pages, 4 figures, 10 tables. Revised version: new chat-schema condition, task-clustered statistics and pass^k reliability, five additional model families, title and abstract updated. Under review. Replication package (v1.1.2): this https URL
Subjects:	Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Databases (cs.DB); Machine Learning (cs.LG)
ACM classes:	D.2.8; I.2.7
Cite as:	arXiv:2610.11482 [cs.SE]
	(or arXiv:2610.11482v2 [cs.SE] for this version)
	https://doi.org/10.48550/arXiv.2610.11482 arXiv-issued DOI via DataCite

Submission history

From: Jorge García-Carrasco [view email]
[v1] Thu, 8 Oct 2026 08:26:46 UTC (63 KB)
[v2] Wed, 11 Nov 2026 15:24:25 UTC (72 KB)