预训练容量何时助力因果学习？从标量预测到反事实后训练
When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training
AI 导读
一项基于 Qwen 2.5 家族的研究考察预训练容量与因果任务适配后性能的关系，提出 CausalLLM 与 CausalGen 两个系统，涵盖反事实 SFT 与偏好对齐。
正文 · 原文
Abstract:How does pre-trained model capacity relate to performance after adaptation for causal tasks? We study two systems built on the Qwen 2.5 family: CausalLLM, which estimates scalar potential outcomes with treatment-specific heads, and CausalGen, which generates treatment-conditioned text or code using soft prefixes and factual supervised fine-tuning (SFT), counterfactual SFT, and preference alignment. Increasing backbone size from 0.5B to 1.5B yields mixed scalar-estimation results on IHDP, but improves CodeGen pass@1 from 0.267 to 0.575 and treatment-effect score from 0.231 to 0.515 after the complete generative pipeline. Within 0.5B CausalLLM, narrative inputs reduce mean sqrt(PEHE) from 1.842 to 1.780, a 3.4% improvement over tabular inputs; the 5.8% improvement over a no-LLM baseline combines changes in representation and input format. These descriptive comparisons motivate a distinction between available and convertible capacity. They do not isolate the contribution of post-training: the two systems differ in dataset, architecture, output modality, and supervision, and intermediate-stage evaluations are unavailable. With only two small backbones and three replications, the results support a hypothesis about capacity--adaptation alignment rather than a general scaling law or a demonstrated causal mechanism.
| Comments: | 10 pages. Accepted at the NeurIPS 2026 Workshop: Transitioning from Pre-Training to Post-Training |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33311 [cs.LG] |
| (or arXiv:2610.33311v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33311 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qi Xu [view email]
[v1]
Fri, 23 Oct 2026 21:27:48 UTC (1,395 KB)
来源：arXiv cs.AI · arxiv.org