跳到正文
arXiv cs.AI· Astarag Mohapatra·· 4 小时前AI 评分36

UltraBERT-JEV:面向 System One 应用的编码器模型

UltraBERT-JEV: Encoder models for System One Applications

AI 导读

技术报告比较了微调 UltraBERT-380M 的四种方案,让编码器模型在单次前向传播中从多个选项中选择其一,无需生成文本。

正文 · 原文

View PDF HTML (experimental)

Abstract:We study encoder models that pick one of several options in a single forward pass, without generating text. This technical report compares recipes for fine-tuning UltraBERT-380M with one mask per option. The four recipes differ in how an option is scored and how the scores become probabilities; one follows the expert gate of DeepSeek-V3. Trained on the same data as Qwen3-0.6B with the LLM-as-Jev recipe (67.6% test accuracy), the encoder reaches 69.7% to 72.1%. Scoring an option by "yes" minus "no" instead of "yes" alone gives the larger gain, from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks (56.1% for the decoder). Sigmoid normalisation adds 0.8 test points; a square-root softplus adds none. One request at a time on an NVIDIA A100, the encoder takes 29 ms per decision against 68 ms, but uses more peak memory (2.8 against 2.2 GiB). An auxiliary head, trained on human-marked evidence and on labels selected with a reward adapted from SelfCite, predicts in the same pass the sentences annotated as support for the answer.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.17867 [cs.CL]
  (or arXiv:2611.17867v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.17867

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Astarag Mohapatra [view email]
[v1] Fri, 13 Nov 2026 07:58:46 UTC (31 KB)

来源:arXiv cs.AI · arxiv.org