权重空间编程修复视觉语言模型:SmolVLM-256M 与 LFM2-VL-450M 故障注入实验
Weight Space Programming for Vision Language Model Repair
AI 导读
一项 arXiv 研究让语言模型以无梯度方式直接编辑权重,修复 SmolVLM-256M 和 LFM2-VL-450M 两个视觉语言模型中被人为植入的权重故障。
正文 · 原文
Abstract:In weight-space programming, a language model edits a network's weights directly, without gradients. We ask what a successful repair of this kind relies on. We plant faults in the weights of two vision-language models, SmolVLM-256M and LFM2-VL-450M, and check each edit made by a coding agent against the original weights. Without being told where to look, the agent finds rescaled projector rows and noise in the vision encoder, infers the unknown scale factors, and returns most or all of the rescaled weights to their exact original values and accuracy to near its original level. These repairs depend on a trace of how the faults were computed. The original weights have 16-bit precision, and faults computed in 32-bit arithmetic leave low bits that only the damaged weights carry, a cue that a real failure need not leave. When we round those bits away, each model answers our test questions as before, but the agent, unaided, no longer repairs either model: on LFM2-VL accuracy recovers by about 4 points, against 17 from the unrounded weights. Marking the damaged weights again in their low bits brings the edits back to them. Rounded weights still hold statistical evidence of the rescaling. On SmolVLM the agent uses it once a sentence in the prompt, which we wrote knowing the fault, names such a statistic; on LFM2-VL, whose scale factors are larger, most sessions find the rescaled rows unaided, but none finds the noise. Swapped rows or columns, which keep the set of stored values, are not found unaided on either model within our budgets. We recommend that studies of weight repair report the number format of the weights and include a rounded control.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23615 [cs.CV] |
| (or arXiv:2611.23615v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23615 arXiv-issued DOI via DataCite |
Submission history
From: Avinash Goutham Aluguvelly [view email]
[v1]
Tue, 17 Nov 2026 23:07:36 UTC (298 KB)
[v2]
Thu, 19 Nov 2026 08:12:10 UTC (324 KB)
来源:arXiv cs.AI · arxiv.org