What to measure, how to measure it, and how good the instrument is
An AAAI-27 workshop on how to evaluate autonomous systems that generate hypotheses, run experiments, and produce scientific claims.
Science is being automated. The methods used to evaluate it have not kept pace.
Autonomous research systems can generate hypotheses, design experiments, run code, and draft papers. The community lacks reliable benchmarks, verification protocols, and agreed measures of what these systems can and cannot do, so genuine advances are hard to separate from findings that are plausible-sounding and incorrect.
Benchmarks for research agents have moved quickly. AstaBench spans more than 2,400 problems across literature work, coding, data analysis, and end-to-end discovery, and ranks 57 agents on a cost-normalized leaderboard. PaperBench scores replication against rubrics written with the original authors, while MLR-Bench and MLAgentBench score stages of the machine learning research cycle. Each of them measures a system.
Each of those measurements also comes from an instrument, and the accuracy of the instrument is established against human agreement. AstaBench scores end-to-end discovery with an LLM judge reading an auto-generated rubric, checks a judge against expert ranking on one of its ten subsets at a Kendall τ of 0.467, and finds that a system left out of the rubric's construction scores lower on identical work. Its authors record that last result as an open question. This workshop is where that question is asked, of benchmarks, rubrics, and judges alike, so that a reported gap between two systems can be attributed to the systems.
This workshop brings together researchers in LLM and agentic AI, benchmark and evaluation methodology, statistics, AI for Science across chemistry, biology, physics, and materials, the metascience and peer-review community, and industry practitioners in AI-driven discovery. We organize the program by the level at which the measurement question sits, so the three areas below partition the problem.
We welcome technical, empirical, methodological, and position papers, including work that critiques current systems and evaluation practice.
Which constructs, and against what ground truth?
How is the instrument built?
How would we know the measurement holds?
Also in scope, expert elicitation treated as an instrument with its own error model, reproducibility infrastructure such as execution traces, seeds, and cost logs, hazard and dual-use assessment for autonomous research in chemistry and biology, and Goodharting once a metric becomes a target.
A shared task for building research systems—and the methods used to judge them.
Organized with XtalPi, the challenge will use a real-world molecular property prediction or drug–target interaction dataset. An expert-annotated corpus of AI-generated papers is planned for public release.
A seed corpus of 60 machine-generated manuscripts, produced in advance by the organizers using four public systems, is released on 2 October 2026. Each carries three independent expert scores on a published rubric with reported inter-rater reliability, so the achievable agreement ceiling for any evaluator is public. The evaluation track does not depend on the generation track, and an evaluator can be entered against the corpus alone.
The released corpus ships with execution traces, cost logs, multi-judge automated scores, and blind held-out validation outcomes, under CC BY 4.0 and Apache 2.0 on Hugging Face and Zenodo with a DOI, versioned and maintained for at least 24 months. It carries canary strings and a contamination-check protocol, and reserves a permanently unreleased holdout for future rounds. The scoring harness, including the adversarial robustness suite, is open source, so later work can reproduce the rankings and attack them.
Results are verified before notification, so the challenge session presents finalised results and no computation runs live. All data, tasks, and code reach every entrant on the same date. Organizers, their students, and employees of the data-contributing partner may enter, are ineligible for prizes, and never score their own track.
Run an AI Scientist system on the shared dataset and submit a report with its execution trace, seeds, code, and cost log. Reports are scored on expert review, automated review, and whether their empirical claims hold on a blind held-out split released only after the deadline. A reference open-weights baseline keeps entry from being gated on budget.
Submit an evaluator, whether a metric, rubric, judge pipeline, or verification tool, and run it over the corpus. Entries are ranked primarily on the AUROC for identifying which reports contain claims that fail blind validation, reported with per-report cost and with performance under a fixed adversarial rewrite suite. Agreement with expert review is a secondary diagnostic. Ranking on held-out outcomes breaks the circularity of scoring an evaluator against the same human signal this workshop treats as unreliable, which is what makes evaluator quality identifiable and not merely correlational.
Challenge entrants who cannot travel may still compete and appear on the leaderboard, and only in-person entrants receive an oral slot.
Challenge entrants who submit a system description are invited as co-authors on the joint overview paper.
Submissions should directly address the evaluation of autonomous scientific discovery.
Papers must use the official AAAI-27 two-column author kit and be submitted as an anonymized PDF through OpenReview. Submissions that compare systems must report at matched compute and token budget, or explain why matching is infeasible. Optional early submission opens ahead of the deadline, so reviewing proceeds on a rolling basis.
Original research on evaluation methodology, benchmarking, or verification. References do not count toward the page limit.
Perspectives, critiques, and emerging research directions.
A released benchmark, harness, corpus, or tool, submitted together with the artifact. Reviewed on whether the artifact runs and is documented.
Attempts to reproduce published AI Scientist results, including failures. No novelty bar, judged on rigor of reporting.
The submission deadline is 06:00 UTC, when the OpenReview portal closes. Other deadlines are 23:59 Anywhere on Earth, meaning UTC−12.
TBD
Université de Montréal and Mila
Evaluation that stays reliable under distribution shift.
University of California, Los Angeles
Uncertainty quantification for machine learning and mechanism design for LLMs.
University of Toronto
Causal reasoning with LLMs, from benchmarks to AI scientists.
University of North Carolina at Chapel Hill
Evaluation methodology and validity, drawing on statistics and biomedical data science.
Rutgers University
Decision-making under uncertainty and budget-constrained test-time compute for LLMs.
Gestala
Ultrasound brain-computer interfaces and the convergence of neuroscience and AI.
The committee combines expertise in agentic systems, evaluation metrology, causal and decision learning, and benchmark design.
McGill University
Auditable, budget-aware evaluation of agentic systems, through the CARE and DynaWeb projects
Co-organized MemAgent at ICLR 2025, Physical Understanding at NeurIPS 2026, and Agentic Robotics at CoRL 2026. Maintains the released corpus for at least 24 months
tianyu.shi3@mcgill.caUniversité de Montréal · Mila
Builds the scientific agent workflows this workshop measures, through AFlow, Data Interpreter and HoneyComb
Area Chair at NeurIPS, ICLR, ACL, EMNLP, NAACL and IJCAI, and Action Editor for Neural Networks. Based in Montréal for on-site delivery
bang.liu@umontreal.caSchool of Statistics and Data Science, Shanghai University of Finance and Economics
Statistical metrology of evaluation, the tier that asks how good the instrument is. Released the StatEval, StatProver and DataSciEval benchmarks with their leaderboard infrastructure
Area Chair for the NeurIPS 2026 Evaluations and Datasets Track. Organized the International Workshop on From Statistics to Artificial Intelligence, which has drawn more than 2,000 participants
zhoufan@mail.shufe.edu.cnServiceNow AI Research · McGill University · Mila
Builds benchmarks for agentic systems, including InsightBench for multi-step analytics agents, and works on relational reasoning and causal discovery
Organized the Bellairs Workshop on Causality in 2023 and 2024, and GroundedML at ICLR 2022
perouz.taslakian@mcgill.caSchulich School of Engineering, University of Calgary
Robustness and generalization, which decide whether an evaluation still holds under distribution shift, with explainability and efficient learning
Leads the Transdisciplinary AI Hub and holds the Schulich Research Chair in XAI. Member of Mila
samira.ebrahimikahou@ucalgary.caAdvising on workshop positioning, program design, and the evaluation, benchmarking, and verification frameworks the workshop develops.
University of North Carolina at Chapel Hill
Statistics, machine learning, and biomedical data science, with sustained work on AI and statistical methods in biomedical studies.
Editor of the Journal of the American Statistical Association, Applications and Case Studies. IEEE Fellow and ASA Fellow.
Delivery of the workshop, its outreach, and its public materials.
Toyota
Publicity chair
Chief Scientific Officer, XtalPi
Challenge sponsorship
Associate Professor, University of Calgary
Challenge infrastructure. Runs the scoring harness and leaderboard for the evaluation challenge.
Cloud and edge AI systems, federated learning and privacy-preserving machine learning.
Mila, Quebec AI Institute
Local arrangements. Handles on-site logistics and speaker and participant communications.
Trustworthy AI and the evaluation of agentic systems.
PhD Candidate, University of Calgary
Web chair. Builds and maintains the workshop website, its design, accessibility, and program pages.
PhD Candidate, Rutgers University
Speaker outreach. Handles speaker invitations and prepares the workshop poster and social-media announcements.
Self-evolving agents, on-policy distillation, and the reliability of LLM judges.
PhD Candidate, University of Calgary
Submissions and review. Supports workshop organization, coordinates the submission and peer-review process, and assists with reviewer recruitment and assignments.
Federated learning, agentic AI, and edge IoT.
PhD Candidate, Rutgers University
Coordination of workshop activities.
Efficient LLM post-training, adaptive test-time computation, and the theoretical foundations of reliable and resource-efficient AI.
Submissions are open on OpenReview. Challenge links will be added when available.