CRUX: how experiments changed the Personas agent’s claims

The Personas agent planned a positive headline: changes to a model’s traits would transfer to behaviors it had not directly trained. An early test returned a null result, and the agent abandoned that claim. It then expanded a negative finding across more models. Later, a rival-method experiment overturned another headline. These were separate changes in what the research could support. CRUX’s reported trajectory, §4.2.

The assignment makes the first claim concrete. The agent was asked to change an LLM’s behavioral traits by intervening in its weights, investigate whether the resulting dimensions could be controlled separately and combined, and test at least one downstream behavior it had not directly trained. Changing behavior used in training would answer only part of that assignment. Transfer required evidence about what happened beyond the behavior directly trained. The Personas research question, §9.1.

This was one of two selected empirical research questions in CRUX’s shadow evaluation. The agents received questions from unpublished papers without receiving those papers or their findings. The original authors assessed the outputs against a conference-submission standard. They brought expertise in the exact questions, but also knew the papers were AI-generated and had developed their own approaches. Their judgments therefore need to be read as non-blind expert assessments. Evaluation design and limits, §§2–3.

The exploration account records an early narrowing of the search. The Personas agent planned to examine three methods over 36–48 hours. After quickly testing its first method and seeing basic positive results, it set the other hypotheses aside and ended exploration after about five hours. Those basic positives should not be confused with evidence for the planned transfer claim: the timeline separately records the null transfer test and the claim’s retirement. Exploration, §4.3; claim changes, §4.2.

The agent then scaled up the negative finding, reporting that it replicated on Qwen models ranging from 3B to 14B parameters and on Llama 3. CRUX preserved a potentially interesting narrow result: fine-tuning only on misaligned style did not produce broad misgeneralization. That finding could remain relevant even though the original positive headline had fallen. Its scope was the tested training and behaviors; it did not establish that transfer was impossible. Cross-model expansion and the useful minor finding, §4.2.

Editorial schematic, not experimental data. The dashed arrow marks the planned transfer, not an observed effect. A and B denote roles in the test. The layout encodes no numerical shares or effect sizes.

The later rival test asked a different question. CRUX reports that the agent evaluated a competing steering method with an identical training budget and that this overturned the paper’s headline. The timeline does not quote that later headline. It cannot safely be substituted for the initial transfer claim, which had already been retired. The later rival experiment, §4.2.

The human expert’s separate critique explains why comparison mattered: the value of the trait space was unclear when weight-space control did not outperform activation-space control under matched capacity. This identifies a question about the method’s added value. It does not recover the missing headline, and its capacity-matching condition must remain distinct from the timeline’s identical training budget. Repeating a negative observation on more models addresses whether that observation recurs in those tests. Establishing an advantage over an alternative requires evidence about the alternative. Expert comparison, Table 4.

There was also a question about what the behavioral measurements meant. The expert criticized apparently hand-picked traits and a weak, potentially circular measure. In the adjacent column of Table 4, the agent’s own final review made the measurement concern more concrete: how could a fixed lexicon measure verbosity rather than topic? These are two separately attributed criticisms. If a score followed topic, a change in it would not establish a change in verbosity. Repeating the score across models would not, by itself, resolve that problem. The table raises this concern; it does not independently prove that topic caused the results. Trait selection and measurement, Table 4.

Feedback did prompt stronger follow-up experiments. CRUX nevertheless describes the agents’ response to fundamental soundness objections as narrowing statements and adding caveats rather than rethinking the premises of the data or methods. The distinction matters here: a more cautious sentence can reduce what a paper asserts, but cannot establish that its measure captures the intended behavior. The eventual Personas assessment was 2/6, Reject, from the original-author experts. Follow-up tests, §4.2; feedback and assessment, §§4.1 and 4.4.

Substantial work had still been completed. CRUX records literature reviews, GPU-environment debugging, hundreds of experiments and robustness checks, and LaTeX paper production. Both main agents used Claude Opus 4.8 with extra-high reasoning on OpenClaw. Researchers supplied credentials and repository setup, manually patched an OpenClaw bug, granted a 24-hour extension and requested a clearer rewrite. Within that supported setup, the engineering record and the inadequately established contribution coexisted. Setup, interventions and engineering, §§3 and 4.5.

The second case, TabPFN, exposes a related inference boundary. Its assignment was to detect when a tabular foundation model’s deployment accuracy had materially deteriorated. Reviewer Viet Nguyen agreed that the particular internal signals discussed did not work, but rejected the leap from those failures to an absence of usable internal signals generally. Testing several candidates had not exhausted the space of possible signals. That objection also did not establish that a successful detector existed. TabPFN question and original-author critique, §§9.2 and 10.1.

For RSI, the cases provide evidence about choosing experiments and deciding what their results justify within these setups. They do not settle AI research’s general limits. CRUX did not test Anthropic’s strongest frontier R&D model as the research agent; that missing test is a limit on extrapolation, not evidence of stronger-model success. The authors also leave open whether these open-ended skills lie on the critical path of accelerating AI development. Neither the engineering achievements nor the failed contributions establish a general recursive self-improvement trajectory. Model limits and R&D implications, §7.

The Personas sequence leaves a specific difficulty visible. The transfer test addressed behavior beyond direct training; the repetitions extended a negative observation; the rival comparison challenged comparative value. The measurement criticism questioned what the observations meant in the first place. Completing those experiments and deciding which contribution they justified were different achievements.

Analysis dated 14 October 2026, examining arXiv 2607.27191v2. Its header states 7 August 2026; its manuscript displays 24 August 2026. Both are retained. This article presents author-reported trajectories and expert judgments with editorial explanations and figures, without independent replication or regrading results.