Our analysis methodology is centred round two core automated methods: an AI scientist for producing new scientific analysis and an automatic reviewer for rigorous analysis. These methods work in live performance to discover the potential of AI in accelerating scientific discovery.
The AI Scientist
The AI Scientist is an agentic system designed to autonomously conduct machine studying analysis. We current outcomes for 2 modes: a template-based system that extends human-provided code and a extra open-ended template-free system that operates with a lot much less prior steerage. The detailed prompts used for every system are offered in Supplementary Data sections A.1.1 and A.2.6. Extra outcomes and analyses of the papers generated by every system are offered in Supplementary Data sections B.1, C.1, C.2, D.1 and D.2.
Foundational applied sciences
Each variations are constructed upon autoregressive LLMs3,4,5, which be taught to generate textual content by modelling the conditional likelihood of a brand new token given previous tokens. By means of huge knowledge and mannequin scaling, LLMs exhibit human-like talents, together with reasoning and code technology. Agentic patterns49, comparable to few-shot prompting50 and self-reflection51, are leveraged by The AI Scientist to enhance efficiency and reliability. For code technology, the template-based system makes use of the state-of-the-art open-source coding assistant Aider52, which is designed to implement options, repair bugs or refactor code in current codebases. To go additional and successfully use extra test-time compute, the template-free system makes use of LLMs to energy a tree search with out counting on Aider.
Template-based AI Scientist
The system is supplied with a beginning code template that reproduces a easy coaching run from a well-liked algorithm on a normal benchmark (for instance, coaching a small transformer53 on the works of Shakespeare). Its workflow unfolds in three phases:
-
1.
Thought technology: The method begins with a easy experiment outlined by a human-provided code template. The system then enters an iterative loop of concept technology and refinement utilizing LLMs as a mutation operator. In every iteration, it proposes a batch of latest analysis concepts which might be variations or extensions of current concepts in its rising archive. Every concept is a structured object containing a descriptive title, a abstract of the core speculation, an in depth experimental plan, and self-assessed scores for interestingness (1–10 scale), novelty (1–10 scale) and feasibility (1–10 scale). This iterative progress of an concept archive was impressed by open-endedness algorithms that keep a various assortment of artefacts20,54. To implement novelty, every proposed concept is routinely checked towards the scientific literature by means of the Semantic Scholar API31; concepts with excessive semantic similarity to current works are discarded. The system is prompted to behave as an ‘bold AI PhD pupil who’s trying to publish a paper that can contribute considerably to the sphere’. For the novelty evaluation, the system conducts as much as ten rounds of literature search queries, and in every spherical, the system can refine its search based mostly on earlier outcomes.
-
2.
Experiment execution: As soon as a promising concept is chosen from the archive, the system devises a multi-step experimental plan with as much as 5 experiments. It then executes this plan sequentially utilizing Aider to switch the codebase. A key function of this section is its robustness to runtime errors. The system routinely detects execution failures, captures the error logs and invokes an occasion of the Aider agent52 to carry out automated debugging. The Aider agent is prompted with the failing code and the error message, and it then generates a patch, with as much as 4 reattempt cycles per experiment. The corrected code is then used to rerun the experiment with a timeout of seven,200 s per experiment. All experimental outcomes, together with metrics, generated plots and observations, are logged in an experimental journal. This journal serves as a type of reminiscence and informs the next steps within the experimental plan.
-
3.
Manuscript technology: Upon finishing the experimental section, the system synthesizes the findings right into a full scientific paper. To take action, it makes use of Aider to populate a normal convention LaTeX template. Aider writes sections, together with the introduction, strategies, outcomes and conclusion. The outcomes part is written by analysing the experimental journal, summarizing key findings and embedding the generated figures. To situate the work inside the broader scientific context, the system constructs a associated work part by querying the Semantic Scholar API for related literature (as much as 20 search rounds) and producing summaries for every cited paper. The manuscript undergoes a number of passes of automated modifying and refinement to enhance readability and coherence. Lastly, the system compiles the LaTeX supply and routinely corrects any compilation errors (as much as 5 correction rounds) to provide a remaining PDF.
Template-free AI Scientist
To beat the restrictions of a set beginning codebase, we developed a template-free model able to extra open-ended discovery. We use OpenAI’s o3 for concept technology and code critique throughout experiments as a consequence of its robust reasoning capabilities, Anthropic’s Claude Sonnet 4 for code technology, OpenAI’s GPT-4o for cost-efficient vision-language duties and OpenAI’s o4-mini for cost-efficient reasoning throughout the evaluate stage. This model introduces a number of key enhancements.
Generalized concept technology
The ideation course of utilized by the system is extra summary and never tethered to an preliminary code implementation. It begins by producing high-level analysis proposals that resemble the summary of a scientific paper. These proposals articulate a analysis downside, suggest a brand new technique and hypothesize the anticipated outcomes. To make sure the proposals are each grounded and new, this course of is tightly built-in with a literature evaluate module that queries exterior tutorial databases to establish data gaps and keep away from rediscovering current work. The system makes use of structured prompts to information concept technology and reflection rounds to refine proposals based mostly on the literature search outcomes (see Supplementary Data part A.2.6 for prompts).
Experiment progress supervisor
Actual-world scientific experimentation usually proceeds by means of distinct levels, from preliminary feasibility assessments to detailed ablation analyses. To emulate this structured strategy, we launched an experiment progress supervisor to coordinate 4 clearly outlined levels of scientific experimentation: (1) begin with a preliminary investigation to check fundamental viability, (2) tune the hyperparameters for optimization, (3) execute the principle analysis agenda and (4) conclude with ablation research to grasp the contribution of various parts. Every stage has express stopping standards. Stage 1 concludes when a fundamental working prototype has efficiently executed. Stage 2 ends when the experiments stabilize, as indicated by convergence in coaching curves and profitable execution throughout at the very least two datasets. Phases 3 and 4 conclude when the allotted computational funds is exhausted. Every stage conducts its personal tree search. The specifics of this tree search course of are detailed within the following bullet level. Every node has a most experiment runtime of 1 h. On the finish of every stage, an LLM-based evaluator assesses all leaf nodes and selects probably the most promising one to function the basis for the subsequent stage of exploration, thus successfully pruning much less promising analysis avenues.
Parallelized agentic tree seek for experimentation
To handle the complexity of open-ended analysis, the sequential workflow of the template-based model of The AI Scientist is changed with a parallelized agentic tree. Determine 3a is an summary of the strategy and Fig. 3b exhibits a tree generated by an precise run. By default, it makes use of Claude Sonnet 4 for code technology. We offer outcomes for various LLM mannequin decisions in Fig. 1b.
Every experimental node inside the agentic tree search undergoes the next execution cycle. First, Claude Sonnet 4 generates each a concrete experimentation plan and the related Python code to implement the experiment. The generated code is instantly executed in a Python interpreter. If the execution encounters an error, the error message is recorded and the node is marked as buggy, ending the present execution cycle for that node. If the execution succeeds, the experiment proceeds to the plotting section.
The system is prompted to avoid wasting all related experimental outputs (coaching and validation metrics, losses and so forth) into structured numpy information. Within the plotting section, The AI Scientist reads these saved outcomes and the code and generates visualizations that summarize and illustrate the findings. These visualizations are subsequently handed to a vision-language mannequin (VLM) for critique. Any points flagged by the VLM (comparable to unclear labels, lacking legends or deceptive visualizations) consequence within the node being marked as buggy, and this suggestions is recorded for future debugging. Nodes that efficiently execute and move the VLM evaluate with out concern are designated as non-buggy.
Every node is outlined as a set comprising an experiment script (for instance, a Python file), a textual description of the high-level plan applied within the script, an execution error hint (if relevant), experiment runtime, efficiency metrics recorded throughout the experiment, code critique from OpenAI o3 after operating the script, a visualization script, file paths to the generated figures, suggestions from a VLM on these figures and the ultimate standing of the node (both buggy or non-buggy).
At every iteration, the system selects a number of nodes from the prevailing tree to develop in parallel. With a predefined likelihood, a buggy node is chosen (thus prioritizing error decision and debugging); in any other case, a non-buggy node is chosen for additional refinement and enchancment. When selecting between non-buggy nodes, the system makes use of a best-first search technique guided by GPT-4o, which evaluates candidates based mostly on elements like efficiency metrics, coaching dynamics and the standard of the plots generated. The chosen nodes are expanded by creating a brand new little one node. The system makes an attempt debugging if the guardian node was buggy or refines and improves upon the earlier experiment if the guardian was non-buggy. Claude Sonnet 4 is used to generate the plan and experiment code for every new little one node, after which all new nodes are executed concurrently in parallel, which drastically accelerates the exploration course of. Along with buggy and non-buggy nodes, the system makes use of specialised node variants tailor-made to particular experimental wants:
-
Hyperparameter nodes systematically discover different hyperparameter configurations throughout stage 2. The system maintains information of beforehand examined hyperparameters to stop redundant experiments. Errors encountered throughout hyperparameter tuning set off the creation of corresponding debug nodes.
-
Ablation nodes consider essential ablation research throughout stage 4. This assesses the significance of varied parts or assumptions underlying the experiment. Like hyperparameter nodes, beforehand examined ablation circumstances are tracked to keep away from repetition, and debugging nodes are created in response to any errors encountered.
-
Replication nodes execute replicates of their guardian experiments utilizing completely different random seeds. Sometimes, a number of replication nodes are created to allow the calculation of statistical measures (imply and s.d.) of experimental outcomes, which boosts the robustness of the outcomes.
-
Aggregation nodes are particular nodes created to consolidate and visualize the mixed outcomes of replication nodes. Not like different node sorts, aggregation nodes don’t conduct new experiments however merely generate a Python script to mixture and summarize earlier outcomes. The script produces figures that explicitly present imply and s.d.
The structured design of the experimental levels and tailor-made node sorts facilitates systematic exploration throughout all levels. Not like some LLM brokers that rigidly observe predefined, fine-grained workflow graphs, The AI Scientist adopts a looser construction that guides your entire empirical analysis cycle, thus enabling versatile system behaviour whereas sustaining coherence throughout iterative levels. See Supplementary Data sections A.2.6 and A.2.9 for the prompts and detailed hyperparameters, respectively.
VLM integration
This technique incorporates VLMs utilizing GPT-4o to analyse and supply suggestions on visible knowledge. Throughout experimentation, the plots generated are fed to a VLM, which is prompted to behave as a scientist and critique them. For instance, it would flag nonsensical axes or points within the high quality of generated examples or recommend clearer methods to current the information. This suggestions is used to generate new experimental nodes within the tree search geared toward addressing the recognized points. Throughout manuscript preparation, the VLM assesses the alignment between figures and their corresponding captions to make sure that a caption precisely describes the plot and highlights the important thing takeaways, thus bettering the general high quality and readability of the paper. The VLM evaluations embody detailed analyses of determine content material, caption accuracy and integration with the principle textual content (see Supplementary Data part A.2.6 for prompts).
Generalized dataset entry
To broaden its analysis capabilities, the system is prompted to dynamically combine datasets from public repositories by formulating queries to the HuggingFace Hub55. A set of ten instance datasets out there on HuggingFace is listed within the immediate, and the system can routinely generate the data-loading code wanted to make use of a specific dataset in its experiments. This strategy partially relaxes the constraint of working with a set, predefined set of datasets by permitting human scientists to simply replace the candidate record. For datasets not out there on HuggingFace, human scientists can obtain them from public knowledge repositories (for instance, open-access archives), retailer them domestically, and add utilization directions to the immediate. These domestically saved datasets can then be used alongside HuggingFace datasets by The AI Scientist (see Supplementary Data part A.2.6 for prompts).
Enhanced manuscript writing
The template-free system strikes away from the incremental Aider-based strategy to direct LaTeX technology utilizing a reasoning mannequin comparable to OpenAI’s o156 adopted by reflection51. The system first aggregates experimental outcomes from a number of levels into compound figures utilizing a devoted plot-aggregation step. The manuscript-writing course of contains particular prompts for various workshop codecs (for instance, the ICBINB workshop specializing in detrimental outcomes), with detailed pointers for every part, together with the title, summary, introduction, strategies, experiments and conclusions. The system undergoes a number of reflection cycles, every time incorporating suggestions from LaTeX linters and VLM evaluations to enhance determine high quality and textual content–determine alignment (see our code and Supplementary Data part A.2.6 for prompts and full particulars).
The whole technology course of for the template-free system usually takes from a number of hours to over 15 h, relying on downside complexity.
The Automated Reviewer
To evaluate the standard of the AI-generated analysis, we constructed an automatic reviewer utilizing o4-mini57. This part was designed to emulate the peer-review strategy of a top-tier machine studying convention by adhering to the official NeurIPS reviewer pointers. The agent processes the PDF of a manuscript to provide a structured evaluate, together with numerical scores for soundness, presentation and contribution, together with a listing of strengths and weaknesses and a preliminary settle for or reject choice (Supplementary Data part A.3). All prompts used for The Automated Reviewer are offered in Supplementary Data part A.3.1.
Assessment course of
The Automated Reviewer follows a multistage course of. First, the system is prompted with the position: ‘You might be an AI researcher who’s reviewing a paper that was submitted to a prestigious ML venue.’ The evaluate immediate supplies the paper content material together with detailed NeurIPS reviewer pointers and asks for a structured JSON response, together with a abstract, strengths, weaknesses, questions, limitations, moral considerations and numerical scores (soundness, presentation, contribution, total rating 1–10 and confidence stage). To enhance robustness, the ultimate evaluation is a meta-review that ensembles 5 impartial evaluations. The 5 evaluations are generated for every paper and aggregated right into a single meta-review, with an LLM taking the position of an space chair to seek out consensus among the many particular person evaluations.
Validation
We benchmarked The Automated Reviewer towards human choices utilizing ICLR knowledge from the publicly out there OpenReview dataset33. The Automated Reviewer achieved a comparable balanced accuracy with people (69% versus 66%; see Supplementary Data part A.3.2 for particulars) and a better F1 rating in contrast with inter-human group settlement (0.62 versus 0.49) within the NeurIPS 2021 consistency experiment34, for which roughly 10% of submissions have been randomly chosen and despatched to 2 impartial evaluate committees, thus offering a real-world benchmark of inter-reviewer consistency (Desk 1). These outcomes point out that LLM-based brokers can present helpful suggestions that aligns with the opinion of the typical human knowledgeable. We spotlight that there was a special set of paper submissions within the ICLR and NeurIPS paper swimming pools and, thus, a shift within the distribution, in order that this comparability will not be precise. Nevertheless, ICLR is the one principal machine studying convention that releases all settle for and reject choices, which allowed us to carry out the evaluation, and the NeurIPS 2021 experiment is the one trendy model of the human consistency experiment, and is, thus, the one doable comparability.
Ethics approval
This research obtained ethics approval from the College of British Columbia Behavioral Analysis Ethics Board (Protocol No. H24-02652). The analysis was carried out in full cooperation with the ICLR convention management and the related workshop organizers. In accordance with the authorised protocol, human contributors (peer reviewers) have been knowledgeable {that a} small variety of submissions to the workshop have been AI-generated, though not which particular papers. Contributors had the choice to choose out of reviewing any probably AI-generated manuscripts. All AI-generated submissions have been withdrawn following the evaluate course of, whatever the final result.
 research1,2. Although the community has made substantial progress in automating individual components of the scientific process, a system that autonomously navigates the entire research life cycle—from conception to publication—has remained out of reach. Here we present a pipeline for automating the entire scientific process end to end. We present The AI Scientist, which creates research ideas, writes code, runs experiments, plots and analyses data, writes the entire scientific manuscript, and performs its own peer review. Its ideas, execution and presentation are of sufficient quality that the manuscript generated by this AI system passed the first round of peer review for a workshop of a top-tier machine learning conference. The workshop had an acceptance rate of 70%. Our system leverages modern foundation models3–5 within a complex agentic system. We evaluate The AI Scientist in two settings: a focused mode using human-provided code templates as an initial scaffold for conducting research on a specific topic and a template-free, open-ended mode that leverages agentic search for wider scientific exploration6,7. Both settings produce diverse ideas and automatically test, report on and evaluate them. This achievement demonstrates the growing capacity of AI for making scientific contributions and signifies a potential paradigm shift in how research is conducted. As with any impactful new technology, there could be important risks, including taxing overwhelmed review systems and adding noise to the scientific literature. However, if developed responsibly, such autonomous systems could greatly accelerate scientific discovery. An artificial intelligence system can produce research papers with minimal human involvement, even passing the first round of peer review for the workshop of a main machine learning conference.){kind=link}