arXiv cs.AI· Jos\'e Roberto Homeli da Silva, Felipe Jorge do Nascimento, Lucas Coutinho Freitas, William Alberto Cruz Casta\~neda, Marcellus Amadeus·· 3 小时前AI 评分31
Brasilidade：衡量大语言模型巴西身份认同的新基准
Brasilidade: Measuring Brazilian Identity for Large Language Models
AI 导读
研究者推出 Brasilidade 基准，用于评估大语言模型在巴西葡萄牙语上的表现，首批开放语言、语法及变体领域的标准化任务。结果显示模型规模本身不决定成绩，多个中等规模模型超过了更大模型，而专门针对葡萄牙语训练的模型整体并未高于平均水平。不同任务难度差异显著，部分任务对多数模型极易，另一些则构成极大挑战。
正文 · 原文
Abstract:This work presents a new benchmark that aims to address the limitations of current methods for evaluating LLMs in Brazilian Portuguese. We introduced Brasilidade, an assessment methodology comprised of standardized tasks distributed across four different domains, and we are making available the tasks from its first domain, which address nuances of the language, grammar, and variations of Brazilian Portuguese. The results show that model size, by itself, does not determine performance; several medium-sized models outperformed larger models in the protocols evaluated. Specific models for Portuguese, in general, did not perform above average, despite their targeted training. Different task levels revealed significant nuances: while some tasks proved easy for most models, others posed extreme challenges, and some yielded mixed results, with wide variations in performance. This clearly illustrates the need for benchmarks that address the limitations of the models in Brazilian Portuguese, allowing future training cycles to seek to overcome them. Project repository this https URL
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.12906 [cs.CL] |
| (or arXiv:2611.12906v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.12906 arXiv-issued DOI via DataCite |
Submission history
From: William Alberto Cruz Castañeda PhD [view email]
[v1]
Thu, 15 Oct 2026 02:14:13 UTC (4,237 KB)
来源：arXiv cs.AI · arxiv.org