研究发现稀疏性驱动多任务神经网络涌现功能模块化
Sparsity-driven emergence of functional modularity in multitask neural networks
AI 导读
Bowen Shi 等人在 arXiv 发表的研究显示，稀疏正则化能在多任务神经网络中自发促进功能模块化，无需预设模块边界。研究在多层感知机和轻量级 transformer 中识别出紧凑、任务选择性的子网络 functional tokens：选择性保留这些 tokens 可维持对应任务功能，且其可在不重训的情况下经替代路径重组以重构学习到的复合计算。该特化优先在深层形成，浅层保留共享计算。
正文 · 原文
Abstract:Neural networks can learn multiple tasks, but how learning organizes shared computation into specialized and reusable components remains poorly understood. In this work, we show that sparsity inducing regularization promotes functional modularity in multitask networks without predefined module boundaries. Using functional profiling in multilayer perceptrons and a lightweight transformer, we identified compact, task-selective subnetworks, termed functional tokens. Across rule-based task families, selectively retaining these tokens preserves their associated functions while reducing performance on other tasks. Such specialization develops preferentially in deeper layers during training, while earlier layers retain shared computation. Regularization strength and form shape module compactness and specialization, while task weighting alters functional allocation. Moreover, extracted functional tokens can be recombined through alternative pathways to reconstruct a learned composite computation without retraining, preserving their connectivity and learned weights. Our results suggest that sparsity-constrained optimization shapes not only parameter use but also the organization and reuse of learned computation.
| Comments: | 34 pages, 6 figures, including Supplementary Information |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32153 [cs.LG] |
| (or arXiv:2610.32153v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32153 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bowen Shi [view email]
[v1]
Fri, 23 Oct 2026 04:38:24 UTC (5,601 KB)
来源：arXiv cs.AI · arxiv.org