Jev 在科学决策中的语义选择评测：评估语义判断及其下游影响

View PDF HTML (experimental)

Abstract:Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency and known mean cost among successful responses. Across three comparison models, seven wrong selections on one question about whether samples shared a culture changed downstream counts while preserving the correct final label. These results support using Jev to select relations when the evidence and candidates are prepared in advance, and show that evaluating it requires checking the relations and quantities that a workflow will reuse.

Comments:	9 pages, 1 figure, 6 tables. Code available at this https URL
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
ACM classes:	I.2.7
Cite as:	arXiv:2609.24965 [cs.CL]
	(or arXiv:2609.24965v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.24965 arXiv-issued DOI via DataCite

Submission history

From: Boyuan Deng [view email]
[v1] Mon, 21 Sep 2026 17:51:50 UTC (167 KB)
[v2] Wed, 23 Sep 2026 04:04:31 UTC (166 KB)
[v3] Thu, 15 Oct 2026 07:14:47 UTC (218 KB)
[v4] Sat, 24 Oct 2026 00:46:12 UTC (219 KB)