SCAPE-T：基于 encoder-only Transformer 注意力量化语义复杂度
Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention
AI 导读
一项研究提出 SCAPE-T，一种基于 encoder-only Transformer 注意力的语义复杂度度量，将语义复杂度定义为唯一指定文本语义所需的编码长度。该指标结合反向加权自聚焦、CLS 聚焦与头间冗余三个子指标，在词长控制下成对准确率达 78.5%，比最强基线高 12.7 个百分点。消融实验显示各子指标均有实质贡献，且方法在两个留出域上泛化良好。
正文 · 原文
Abstract:Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort (readability, lexical difficulty), which correlates with surface features (word length, syntax), or quantify surface features directly (type-token ratio, word frequency). SCAPE-T targets semantic integration directly by combining three submetrics: (i) inverse-weighted self-focus (token-to-self), (ii) CLS focus (token-to-CLS), and (iii) inter-head redundancy (cosine similarity across heads), using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline); without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.
| Comments: | 12 pages; semantic complexity, complexity estimation, encoder attention, attention analysis, pairwise accuracy, readability, text simplification, transformers |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2610.33441 [cs.CL] |
| (or arXiv:2610.33441v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33441 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lev Mukhanov [view email]
[v1]
Sat, 24 Oct 2026 04:34:28 UTC (32 KB)
来源：arXiv cs.AI · arxiv.org