How to Review an Autonomous Research Deliverable

Review the evidence behind the claim

Start with the main claim, then follow it through the evidence map, experiment plan and result artifacts. Treat polished prose and plots as pointers to inspect. A useful review says what the evidence supports, under which conditions, and what remains unknown.

The DeepGraph project page describes an evidence map, novelty check, experiment plan and first signals, with further work including experiments, a baseline, an evidence pack and a paper. Use the checks below to assess what each output supports and which evidence is still needed.

1. Evidence map and novelty

• Can important claims be traced to original sources, with gaps and counterevidence visible?
• Does the map distinguish source findings from interpretation, and show relevant dates and limits?
• For the novelty claim, what closest work was checked, when, and what exact difference is proposed?

A search log makes a novelty check repeatable; it cannot prove that no related work exists.

2. Experiment plan and baseline

Before reading outcomes, look for the hypothesis, comparison, data and split definitions, metrics, implementation details, compute budget, stopping rule and failure handling. Ask whether each measurement can distinguish the proposed explanation from plausible alternatives.

For the baseline, compare data access, preprocessing, tuning effort, hardware, compute accounting, attempts and stopping rules. Define “same compute”; document and explain any mismatch.

3. Multi-seed results and negative findings

Request each run as well as the aggregate. Check planned versus completed runs, exclusions, spread and failures, and confirm the summary matches the underlying values. A single favorable run does not show how consistently an outcome repeats.

Keep failed, null, inconclusive and unfinished work visible. Distinguish “not run,” “run failed,” “inconclusive” and “no observed effect,” and record deviations from the plan.

4. Trace claims to artifacts

Pick several numbers or conclusions and follow each to its analysis, run output, configuration and input references. Check that run records identify the code revision, environment, dataset version and split. A code link alone does not identify the evidence behind a result.

5. Write an actionable review

Separate supported claims from unresolved ones. For each judgment, name the artifact checked; for each gap, state the next experiment or inspection that could change the assessment. This keeps the decision tied to evidence rather than the polish of the report.