基于 Knowledge Space Theory 的大语言模型下一步学习推荐评测方法
Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory
AI 导读
一项研究提出用 Knowledge Space Theory（KST）作为形式化参照来评测 LLM 的下一步学习推荐，从任务目录和先修关系推导知识状态与推荐参照集。
正文 · 原文
Abstract:Selecting an LLM for next-step recommendation requires benchmarks that reflect the intended educational task and domain. We present a benchmarking method that uses Knowledge Space Theory (KST) as a formal evaluation reference. Given a task catalogue and an existing prerequisite relation, the method derives admissible knowledge states and the complete set of immediate next steps for each state. Recommendations generated by a model under a specified prompt are compared with these reference sets. A sample evaluation study uses 2,400 responses from six models across four mathematics domains, four prompt conditions, and two catalogue orders. Exact-set accuracy was 61.3% with descriptions only, 62.8% when models were asked to reconstruct prerequisites, 96.3% with supplied prerequisite relations, and 98.2% with the complete knowledge space. Performance varied across models and domains, and recommendation sets agreed across the two presentations in 85.25% of pairs. The benchmark supports comparisons of model-prompt combinations against a context-specific reference. Its scores describe structural recommendation accuracy under the evaluated conditions; they do not establish learning benefit or the necessity of using an LLM for selection.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13667 [cs.AI] |
| (or arXiv:2611.13667v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13667 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peter Steiner [view email]
[v1]
Tue, 10 Nov 2026 07:15:25 UTC (47 KB)
来源：arXiv cs.AI · arxiv.org