神经网络研究:Hidden not Deleted——网络如何抑制纠缠特征
Hidden not Deleted: How Networks Suppress Entangled Features
AI 导读
一项 arXiv 研究通过小型自编码器追踪神经网络停止输出某特征后是遗忘还是隐藏了该特征。在表示固定时,恢复单个偏置可通过暴露隐藏在输出激活阈值后的信号部分恢复目标检测,目标缺失输入测试可将其与单纯提升输出的干预区分开。研究发现恢复取决于特征几何结构和被训练的参数,输出沉默不足以证明遗忘。
正文 · 原文
Abstract:When a neural network stops reporting a feature, has it forgotten the feature or merely hidden it? Shared representations complicate the answer: removing one feature's direction can also damage another feature. We study this problem in a small autoencoder that compresses and reconstructs inputs, allowing individual feature contributions to be traced directly. With the representation fixed, restoring a single bias partially recovers target detection by exposing a signal hidden behind the output's activation threshold. Tests on target-absent inputs distinguish this recovery from interventions that simply increase output. The same bias intervention becomes less effective when the representation is allowed to change. We find that training only the other outputs can weaken the target's contribution, even without a loss on its own output. A conditional theoretical analysis explains how interference with retained predictions creates this pressure. Yet more flexible predictors still detect residual target information. These findings show how recovery depends on feature geometry and on which parameters are trained, and why a silent output is insufficient evidence of forgetting.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.27593 [cs.LG] |
| (or arXiv:2609.27593v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.27593 arXiv-issued DOI via DataCite |
Submission history
From: Akash Samanta [view email]
[v1]
Wed, 23 Sep 2026 09:10:12 UTC (349 KB)
[v2]
Fri, 13 Nov 2026 10:19:03 UTC (152 KB)
来源:arXiv cs.AI · arxiv.org