More tools can improve the score while old-task performance declines
EvoHarnessBench v3 separates two comparisons: final performance across task cohorts and retention within old cohorts. Its diagram makes the difference visible.
AnalysisExpand summaryCollapse summary
The paper reports higher aggregate pass rates with cumulative tools alongside negative old-cohort backward transfer for its deployment baselines. These results use different baselines. Reading the final row for ACC and pairing introduction and final cells for BWT explains the distinction; memory controls and a no-memory AutoGen counterexample keep the conclusion bounded.