arXiv cs.AI· Hossam Elsafty, Tobias Deu{\ss}er, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa·· 3 小时前AI 评分30
Jev 阿拉伯语能力评测：比较决策模型与 LLM 在多方言任务上的表现
Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects
AI 导读
该 arXiv 评测用统一零样本协议在 9 个任务族、43 项任务上评估了 67 个系统，包括 47 个决策模型和 20 个大语言模型。Jev 1.14.0 平均任务主指标得 69.24，高于 Jev 1.13.0 的 68.77，低于最高分 Gemma-4-31B 的 71.81。
正文 · 原文
Abstract:How does Jev compare with other systems on Arabic tasks, and how does its performance change between Modern Standard Arabic (MSA) and dialects? We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. The Arabic-focused frozen suite includes English controls in three parallel resources, which we also remove in a separate descriptive comparison. Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. Removing the recorded English controls, repeated cross-task items or both leaves the full system order unchanged. Relative performance and gains between Jev versions vary across task families. In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. Paired contrasts with Jev 1.13.0, Gemma-4-31B and Jais-2-8B show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.
| Comments: | 28 pages, including references and appendices |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.10802 [cs.CL] |
| (or arXiv:2611.10802v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.10802 arXiv-issued DOI via DataCite |
Submission history
From: Hossam Elsafty [view email]
[v1]
Sat, 7 Nov 2026 15:30:03 UTC (104 KB)
[v2]
Thu, 12 Nov 2026 00:13:39 UTC (126 KB)
来源：arXiv cs.AI · arxiv.org