arXiv cs.AI· Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli·· 4 小时前AI 评分47
TherapyGym：评估与对齐治疗聊天机器人的临床保真度与安全性
TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
AI 导读
斯坦福等机构研究团队推出 TherapyGym 框架，从保真度和安全性两大临床维度评估并改进治疗聊天机器人：保真度用认知治疗量表（CTRS）自动化评估多轮 CBT 技术依从性，安全性覆盖未处理伤害或虐待等治疗特有风险。
正文 · 原文
Authors:Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli
Abstract:Large language models (LLMs) are increasingly used for mental-health support; yet prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy. We introduce THERAPYGYM, a framework that evaluates and improves therapy chatbots along two clinical pillars: fidelity and safety. Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks (e.g., failing to address harm or abuse). To mitigate bias and unreliability in LLM-based judges, we further release THERAPYJUDGEBENCH, a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians. THERAPYGYM also serves as a training harness: CTRS and safety-based rewards drive RL with configurable patient simulations spanning diverse symptom profiles. Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). Our work enables scalable development of therapy chatbots that are faithful to evidence-based practice and safer in high-stakes use.
| Comments: | Accepted to the NeurIPS 2026 Evaluations and Datasets Track |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) |
| Cite as: | arXiv:2603.18008 [cs.CL] |
| (or arXiv:2603.18008v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.18008 arXiv-issued DOI via DataCite |
Submission history
From: Fangrui Huang [view email]
[v1]
Mon, 23 Feb 2026 06:05:01 UTC (1,931 KB)
[v2]
Tue, 3 Nov 2026 22:41:02 UTC (1,981 KB)
来源：arXiv cs.AI · arxiv.org