IA Week 2026 · Grenoble

AI (Co-)Scientists

Where we are, how to guess where it's going — and where I stand
Stéphane Lathuilière
Inria Grenoble · RobotLearn
IA Week 2026 · 16–19 November 2026 · organised by the MIAI Cluster, Université Grenoble Alpes

About me

Stéphane Lathuilière
Stéphane Lathuilière
  • Research Scientist, RobotLearn team, Inria Grenoble
  • Before: Associate Professor at Télécom Paris (2019–2024)
  • Research: generative models for image and video (pre- and post-training), multimodal LLMs, continual learning

How I started this journey

JB
Jon Barron
@jon_barron

My advice to PhD students in 2026:

1) If your advisor hasn't logged >100 hours in a modern agentic IDE, stop listening to your advisor.

2) Write your next paper inside your codebase as a .tex (with 80 char word-wrap). Force your advisor to read and make edits inside the IDE.

3:32 PM · Jun 4, 2026 (UTC)
Am I still a useful advisor?
My goal: log 100 hours in an agentic IDE!

Outline

  1. State of the art
  2. How to guess the future?
  3. Where I am — and where I want to go?
Part 1

State of the art of AI (co-)scientists

Definitions, key systems, and how to read the claims.

Levels of autonomy

L0ToolHuman does it all
L1AssistantHuman asks targeted questions
L2Stage co-pilotHuman delegates one stage
L3Multi-stage agentHuman approves plans, validates
L4Closed loopHuman sets the goal, audits the outcome
L5Open-ended researcherPicks its own problems
not demonstrated
human does the researchAI autonomy →no human
Of nine closed-loop L4 systems surveyed in 2026, none has an externally validated verifier in the loop.
arXiv 2608.05179 (Ding et al., 2026)

The research loop — and where AI is strong or weak

Literaturestrong recall · fake refs
→
Hypo­thesisnovel · often infeasible
→
Designgood protocols · lacks tacit knowledge
→
Execute in silicostrong engineering
→
Execute physicallyslow, costly, narrow
→
Analysefast · over-claims
→
Writefluent · sloppy
→
Verifythe bottleneck
Where AI is now strong: breadth, speed and engineering, and anywhere a cheap automatic checker exists (unit tests, Lean, benchmarks).
Where AI is still weak: research taste, backtracking, physical experiments, and honest uncertainty. Verification is the binding constraint.

The AI Scientist: the full loop, automated

Idea+ automatic novelty check
→
Experimentagent writes and runs code
→
Write-upLaTeX paper
→
AI reviewerscores the paper
~$15per generated paper (v1, 2024)
1 of 3AI-written papers passed peer review at an ICLR 2025 workshop
base modelpaper quality scales with the underlying LLM (Nature, 2026)
Workshops accept ~70%, humans picked what to submit, and the papers had citation errors. Passing review shows a reviewer was convinced, not that the result is true.

Are LLM ideas actually better? The two Stanford studies

1 · Judged on paper (2024)

  • 100+ NLP researchers, blind review
  • LLM ideas rated more novel, slightly less feasible
  • Low diversity at scale

2 · Judged after execution (2025)

  • 43 experts executed the ideas (100+ h each)
  • LLM ideas' scores dropped more
  • The gap closed, and on several metrics flipped
Ideation–execution gap: reading as novel ≠ working as research.

"Science factories": the billion-dollar bet

AI modelhypothesis + designRobot labruns experimentsReal dataverifies the ideaTrainingmodel learnsclosed loopproprietary data

Lila Sciences life sci · chemistry · materials

Automate the whole loop, from hypothesis to robot experiment to analysis.

Periodic Labs superconductors · semiconductors

The lab is a data engine: its experiments train frontier models.

Radical AI materials

Materials foundation models drive a self-driving lab.

Project Prometheus engineering · manufacturing

AI that learns from physical trial and error.
Core problem: LLMs are commoditised, so the moat is proprietary experimental data and verification by real experiment. Little public evidence of discoveries yet.

Checklist: how to read an "AI discovered X" claim

  1. Who chose the problem, and who validated the result? Were the validators independent of the developers?
  2. Is it new? Was a literature search done, and was it a rediscovery? (Erdős, A-Lab)
  3. What was the oracle? Lean, wet lab, simulator or LLM judge, and could it be gamed?
  4. How many attempts were made, and how were the shown ones selected? Look for best-of-n disclosure.
  5. Is it auditable? Code, seeds, traces, prompts.
  6. Does it generalise? Scale, a second dataset, a held-out test.
Systems that pass most of these checks: Erdős #728 (Lean), AlphaEvolve on 67 problems (successes and failures reported), Co-Scientist cf-PICI (an independent prior experimental result).
Part 2

How to guess the future?

Trend lines (METR, compute, task difficulty), signs of self-improvement, and the evidence that keeps us humble.

How METR measures a "time horizon"

Task suite228 software / ML / cyber tasks (TH 1.1)
→
Human baselinesexperts timed on each task
→
Agent runsrepeated runs, auto-scored
→
Logistic fitP(success) vs log(human time)
  • 50% horizon = the human task length at which the model succeeds half the time. The 80% horizon is much shorter, e.g. ~17 h vs ~3 h for the best 2026 model.

Doubling time of the 50% horizon

WindowDoubling≈ per year
since 2019~196 days×3.6
since 2023~131 days×7
since 2024~89 days×17

The METR time horizon: agents work for longer and longer

2 s30 s1 min8 min1 h8 h32 h20192020202120222023202420252026GPT-2: ~2 sGPT-2 · 2 sGPT-3: ~9 sGPT-3.5: ~36 sGPT-4: ~5 minGPT-4 · 5 minGPT-4o: ~9 minClaude 3.5 Sonnet (new): ~28 mino1: ~39 mino1 · 39 minClaude 3.7 Sonnet: ~59 mino3: ~1.5 hGPT-5: ~2.3 hGPT-5 · 2.3 hClaude Opus 4.5: ~4.8 hClaude Opus 4.5 · 4.8 hGemini 3.1 Pro: ~6.4 hClaude Mythos Preview: ~17.4 hClaude Mythos Preview · 17.4 h50% success horizon (human-expert task length), log scale
Approximate values from METR publications (v1.0/1.1 task suites: software, ML and cyber tasks). METR warns that horizons above ~16 h are unreliable as the suite saturates, and that analysis choices shift results by up to 20%. Doubling time ≈ 4–7 months.

Reading the METR curve with care (as of Nov 2026)

Where the frontier is

  • Best 50% horizon: ~17 h (Claude Mythos Preview); 80% horizon ≈ 3 h.
  • The task suite is saturating.

Why not to over-read it

  • Equally reasonable curve fits give very different horizons: up to −35% (50% horizon), up to ×2 (80% horizon).
  • Few long tasks, so the tail is thin.
  • Tasks are clean and auto-scored; research is not.
For research, the binding question is whether there is an automatic scorer.

What is self-improvement?

AI system vₙproposes a changetests it on taskskeeps it if bettervₙ → vₙ₊₁no humanin the loopWhat does the loop change?Scaffoldprompts, search, memorytoday · AIDE²Training stackkernels, schedules, datatoday · AlphaEvolveWeights / next modelAI does the AI researchthe “explosive” casestronger feedback
The forecasting question: does each turn of the loop make the next turn faster?

Progress in numbers (1/2): how hard are the solved tasks?

gold → 42/42Olympiad mathIMO · 2 × 4.5 h✓ grader / Leannear-saturatedML engineeringKaggle · days✓ leaderboard score15–41%Reproduce a paperML paper · weeks✓ target numbera few solvedOpen problemsErdős · decades✓ Lean0Open-ended researchmonths✗ no checker
The ladder is climbed fastest where a checker exists. Without one (open-ended research), progress is still near zero.

Progress in numbers (2/2): inputs, speed and efficiency

×5 / yr
frontier training compute since 2020 (doubling ≈ 5 months), Epoch AI
×7–17 / yr
METR 50% horizon growth (doubling 131 days since 2023, 89 days since 2024)
5 min → ~17 h
METR 50% horizon, GPT-4 (2023) → Claude Mythos Preview (2026): ≈ 200×
~$6–10M
for one claimed math result: 10k agents × 88 h (forced Navier–Stokes, Sep 2026), at public API prices
~$15
per generated paper, AI Scientist v1 (2024)
$12 · $66
per reproduction attempt · per confirmed reproduction, RECLAIM (2026)
+40% / yr
energy efficiency (FLOP/s per watt) of leading ML chips, Epoch AI
÷33 in 1 yr
energy per median Gemini prompt, model + software + hardware (Google, Aug 2025)
÷9–900 / yr
price to reach a fixed capability level, depending on the task, Epoch AI
Capability grows faster than compute, and capability per watt grows faster than chips: algorithms, post-training (RL) and scaffolds add on top of hardware.

Reproduction is still hard

0%25%50%75%100%PaperBench 2025 — best agentPaperBench 2025 — best agent: 21%21%PaperBench 2025 — ML PhDs (subset)PaperBench 2025 — ML PhDs (subset): 41%41%RECLAIM: given code + weights → just runRECLAIM: given code + weights → just run: 41%41%RECLAIM: given code only → retrainRECLAIM: given code only → retrain: 27%27%RECLAIM: given paper only → reimplementRECLAIM: given paper only → reimplement: 15%15%
best AI agenthuman baseline
RECLAIM (2026): 100 NeurIPS-2025 papers; target number, tolerance and GPU budget fixed in advance; graded from logs by a separate model. The 3 variants = what the authors released.
128 vs 73reproductions claimed by agents vs confirmed by audit
~70%of the compute budget left unused when an attempt fails
#1 errorimplementing without checking against the paper's numbers
The missing skill is not engineering but scientific self-checking.
arXiv 2504.01848 (PaperBench) · Salunkhe et al., arXiv 2609.28850 (RECLAIM, Sep 2026)

Why math went first

LiteratureMathlib, databases
→
Hypothesisbacklog of open problems
→
Designproof strategy
→
Execute in silicojust compute
→
Execute physicallynot needed
→
Analyse
→
Write
→
VerifyLean: exact, seconds
↺ cheap exact feedback → RL and search optimise against it directly

The general principle: AI progress scales with the speed and fidelity of the field's verifier

Mathexact proof checker
ML / CVnoisy metrics
Chemistry, materialsapproximate simulators
Biology, medicineslow wet labs, trials
fast, exact verifierslow, noisy verifier →

Three scenarios for 2030

Co-pilot plateau

AI is everywhere at L2–L3. Big speed-ups on coding and analysis, but verification capacity caps the end-to-end gain.

Verifiable-domain takeoff

Math, ML and the simulable parts of chemistry and materials see AI-led discovery. Biology and medicine stay human-validated.

Science factories

A few private labs with robots and models produce a large share of discoveries. That raises questions of access, publication and governance.
The deciding variable in each scenario is the cost of verification: formal tools, simulators, robotic labs and institutions.

Take-home messages

  1. AI (co-)scientists are real but uneven. They are strong at engineering, breadth and speed, and weak at research taste and verification.
  2. Progress follows verifiability: math leads, then ML, then simulable sciences, then wet-lab and clinical fields.
  3. Co-scientist beats autonomous scientist today. Outside math, every validated discovery had humans owning the validation. In math, Lean can own it (Erdős #728), but humans still pick the problems and check the formal statements.
  4. Judge claims by independent verification, not by peer-review acceptance or self-reported accuracy.
  5. Capital is betting on data and labs, not models. Watch for the verification infrastructure.
Part 3

Where I am — and where I want to go?

What changed in my daily research, a bet for computer vision, and two research directions.

Where I am on the autonomy scale

L0ToolHuman does it all
L1AssistantHuman asks targeted questions
L2Stage co-pilotHuman delegates one stage
L3Multi-stage agentHuman approves plans, validates
L4Closed loopHuman sets the goal, audits the outcome
L5Open-ended researcherPicks its own problems
me, today

What changed for me

  • I'm convinced by the co-scientist mode, at least.
  • Coding is ~10× faster.
  • I can go towards more technical problems.
  • I'm less scared of trying new topics.

Back-of-the-envelope (my own estimate)

Before
coding 65%
rest 35%
After (coding ÷10)
rest 35%
freed ≈ 58%

My scaffold (1/2): research as a task lifecycle

A Claude Code template I fork for each project: research state in plain markdown, plus 10 project-local skills.

Scratchadd-task: a one-line idea in task.md
→
Plannedplan-task: plans/T###.md
→
Executedexecute-task: code run, rawOutput/T###/
→
Analyzedanalyze-task: finding in exp.md

The research state

  • method.md: motivation, method, open questions
  • code.md: map of the codebase
  • task.md → exp.md: ideas → findings
  • related/: papers as markdown
  • logs.md: every action, timestamped
Generated task index page: tasks grouped by Scratch, Planned, Executed, Analyzed
Auto-generated result site: one page per task

My scaffold (2/2): design principles

Keep context small

  • Each skill reads only what it needs.
  • Papers enter as markdown; PDFs are blocked.
  • Literature search runs in a sub-agent.

Keep the human in control

  • Autonomy level per task: A0 (ask a lot) → A2 (full autonomy).
  • A "code epoch" flags plans made stale by later code changes.

Make everything traceable

  • Every skill logs its action.
  • Result pages are generated by scripts, never hand-edited.

Think, not only run

  • discover-paper / add-paper: find and link related work.
  • modeling-mode: connect open questions with results.

My bet for computer vision

CV research will shift from designing learning algorithms to steering agents that design them.

Why I think it is plausible

  • CV has good evaluators: benchmarks, metrics and simulators, so it sits high on the verifiability ladder.
  • Agents are strong at the engineering part of our papers.

What stays human

  • Choosing the problem and the evaluation, which is where most of the research taste lies.
  • Guarding against benchmark over-fitting by the search itself.
  • Understanding why a found algorithm works.

Two directions I want to pursue

1 · Improving RL post-training

  • RL post-training is what turned LLMs into the reasoning agents we saw today.
  • It is also a natural target for agentic search: rewards and data are searchable objects with measurable outcomes.

2 · Multimodal interaction systems

  • The question: how do we cast these problems so that agentic systems can be deployed on them?
  • That means asking what the verifier is, what can be simulated, and what needs a human in the loop.
The common thread with the rest of the talk: make the problem checkable, and agents can help.

My advice in 2026

SL
Stéphane Lathuilière
replying to @jon_barron

For advisors:

1) Target 100 hours of agentic coding.

For PhD students:

1) Still listen to your advisor.

2) Have at least one advisor with the 100 hours of agentic experience.

3) Don't prompt: work on your scaffold.

4) Focus on a problem with a weak verifier, or work on how to deploy AI scientists on your problem.

5) Study LLMs, even if they are not your core topic.

6) Pick problems that will matter with the models to come, not with today's models.

November 2026 · this talk

Thank you — questions?