There's a GitHub repo that quietly crossed 68,000 stars in under six weeks. No product. No SaaS. No landing page. Just a Python file, a Markdown file, and a ruthlessly simple idea: what if you gave an AI agent a real machine learning training setup and told it to just... run experiments while you sleep?
That's autoresearch by Andrej Karpathy, released in March 2026. And while the ML research community has been busy dissecting the architecture choices and benchmark numbers, the more interesting thing - at least for anyone building AI products for business - is the pattern it demonstrates.
This isn't just a research toy. It's a working blueprint for a class of autonomous AI agent that most people haven't started building yet.
What autoresearch Actually Is
The premise is stated plainly in the README:
Give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the code, trains for 5 minutes, checks if the result improved, keeps or discards, and repeats.
You go to bed. The agent runs ~100 experiments. You wake up to a log of what worked and what didn't, and - if the agent did its job - a measurably better model than the one you had the night before.
The repo contains three files that matter:
prepare.py- Fixed. The agent cannot touch this. It handles data preparation, the tokenizer, the evaluation function (evaluate_bpb), and the time budget. It is the ground truth.train.py- The only file the agent edits. This is the full GPT model definition, the optimizer (Muon + AdamW), the training loop, and all the hyperparameters. Architecture, batch size, learning rate, attention pattern - everything is fair game.program.md- Not Python code. Markdown. This is the human's interface to the agent. It defines the workflow, the rules of engagement, the logging format, the decision criteria. This is what you as the human iterate on.
The simplicity here is deliberate and important. There's one file you control (the instructions), one file the agent controls (the training code), and one file neither of you can touch (the evaluation harness). That's the whole system.
How the Loop Works - Step by Step
The actual experiment loop the agent runs is defined in program.md and it goes like this:
1. Setup Phase
The agent reads the repo, checks that training data exists, creates a new git branch (e.g. autoresearch/apr8), and establishes a baseline by running train.py as-is. This first run is never skipped - you always need to know where you started.
2. The Experiment Loop (runs indefinitely until you stop it)
Each iteration:
- The agent looks at the current git state and decides what to try next - a new architecture idea, a different optimizer configuration, a change to batch size or sequence length.
- It directly edits
train.pywith the change. - It git commits the change.
- It runs
uv run train.py > run.log 2>&1- output redirected to a file so it doesn't flood the agent's context window. - It reads the key metric:
grep "^val_bpb:\|^peak_vram_mb:" run.log - If the metric improved (lower
val_bpb= better), it advances the branch - that commit becomes the new baseline. - If the metric was equal or worse, it
git resetback to the previous commit and tries something else. - It logs the result to
results.tsvwith the commit hash, the metric, VRAM usage, status (keep/discard/crash), and a short description.
The fixed time budget is the key design decision. Every training run takes exactly 5 minutes of wall clock time, regardless of what the agent changes. This makes runs directly comparable - a smaller model that gets more gradient steps in 5 minutes can be compared apples-to-apples against a larger model that gets fewer. No experiment can game the metric by simply training longer.
The metric (val_bpb - validation bits per byte) is also vocabulary-size-independent, which matters because the agent might try changing the tokenizer or vocabulary size as part of its experiments.
3. The Crash Protocol
If a run crashes (OOM, import error, logic bug), the agent makes a judgment call:
- Easy to fix (typo, missing import): fix and re-run.
- Fundamentally broken idea: log as
crash, discard, move on.
The instruction in program.md is explicit: never stop to ask the human if you should continue. The human might be asleep. You are autonomous. If you run out of ideas, think harder.
The Architecture Under the Hood
For context on what the agent is actually modifying, the baseline train.py implements a GPT-style transformer with some modern choices:
- Grouped Query Attention (GQA) - separate
n_headandn_kv_headfor query and key/value projections - RoPE (Rotary Positional Embeddings) - standard now in most modern LLMs
- Value Embeddings - an alternating pattern where some layers include a learned value embedding gate, influenced by recent research showing improved representation learning
- Banded/Sliding Window Attention - the
WINDOW_PATTERNconfig alternates between local (sliding window) and global attention layers, allowing the model to handle long sequences efficiently on a single GPU - Muon optimizer - a recently proposed optimizer that works in the orthogonal gradient space, combined with AdamW for embedding layers
- RMSNorm - via
F.rms_norm, the standard now across most transformer implementations - Flash Attention 3 - with automatic fallback between Hopper-specific and general implementations
This is not a toy GPT-2 clone. This is a serious, modern training setup running on a single H100.
Why This Matters: The Pattern, Not Just the Repo
The specific task autoresearch does (optimize LLM training) is narrow. But the pattern it demonstrates is broadly applicable.
Here's the pattern abstracted:
1. Fix the evaluation function - it cannot be touched
2. Fix the constraints (time, resources, dependencies)
3. Give the agent one file it can modify
4. Give the human one file to write (the instructions)
5. Loop: modify → evaluate → keep/discard → repeat
6. Log everything to a durable record
7. Never stop for permission
This is an autonomous optimization loop over a well-defined metric with a human-controlled instruction layer. That structure maps onto a surprising number of business problems.
Business Applications: Where This Pattern Actually Applies
The Obvious Ones (Worth Stating Clearly)
ML Platform Teams at Mid-Size Companies
Most ML teams spend disproportionate time on hyperparameter tuning and architecture ablations. These are exactly the boring-but-necessary experiments nobody wants to run manually on a Friday afternoon. An agent running the autoresearch loop on your internal model training setup - overnight, on your own infra, logging to your own results store - is a weekend's implementation away. For a broader picture of how agentic AI fits into enterprise strategy, the principles here apply directly.
Data Science Teams Doing Forecasting
Whether it's demand forecasting, churn prediction, or revenue projection, the core loop is identical: try a configuration, evaluate on held-out data, keep or discard. The time budget metaphor translates directly - cap each candidate model at N minutes of training time and compare on your validation metric.
The Niche Use Cases (The Ones Worth Thinking About)
These are the applications of this pattern that don't show up in the "AI for business" roundup articles.
1. Autonomous Strategy Backtesting for Quant Firms
Quant firms already do backtesting loops. What they don't usually have is an agent that autonomously writes the strategy variants, runs the backtest, evaluates on Sharpe ratio or drawdown, and advances or reverts. The autoresearch pattern maps directly: the backtesting engine is prepare.py (fixed, ground truth), the strategy file is train.py (what the agent modifies), and the human writes the instructions for what kinds of modifications are in-scope.
A mid-size prop trading firm could run overnight strategy research on historical tick data without a single human writing a line of strategy code.
2. Actuarial Model Optimization for Insurance AI
Actuaries build pricing models that are retrained periodically on new claims data. The models themselves - GLMs, gradient boosted trees, neural network tabular models - have dozens of configuration choices. An agent running an autoresearch-style loop against a validation set of historical claims, optimizing for calibration error or log loss, could replace what is currently a manual, expert-driven quarterly process.
The critical piece: the evaluation harness (the held-out validation set, the metric definition) stays locked. Only the model configuration moves.
3. Industrial Sensor Anomaly Detection with Latency Constraints
Manufacturing plants run predictive maintenance on sensor data. The model that detects bearing failure or temperature anomalies needs to be periodically refit as equipment ages. An agent that runs overnight on each production line's sensor history, trying different model architectures and window sizes, and keeps whatever reduces false negatives (missed failures) while staying under a false positive budget - that's a direct application of this loop.
The time budget constraint becomes a latency constraint: each candidate model must be able to run inference in under X milliseconds on the edge device.
4. Legal Contract Classification Fine-Tuning
Law firms and legal tech companies are fine-tuning document classification models on labeled contract clauses. (For context on how intelligent document processing already operates in finance and logistics, the same pipeline assumptions apply here.) The label taxonomy changes as regulation evolves. An agent that autonomously experiments with fine-tuning recipes, data augmentation strategies, and prompt templates - evaluated against a human-labeled validation set that the agent cannot touch - and runs overnight on the firm's GPU, would meaningfully compress the iteration cycle on model improvement.
This one is interesting because the "training file" the agent modifies isn't just hyperparameters - it could include the few-shot examples in the prompt, the classification taxonomy definition, or the fine-tuning data filtering logic.
5. Ad Bid Strategy Optimization via Simulation
Performance marketing agencies run thousands of ad campaigns. Bid strategy parameters - target CPA, bid multipliers by audience segment, dayparting weights - are currently tuned manually or via platform auto-bidding. An agent that modifies bid configuration files, runs a 24-hour live test window (or a simulation against historical auction data), evaluates on ROAS, and keeps or reverts - that's a business-grade autoresearch loop that runs continuously, not just overnight.
The unusual part: the "metric" here is revenue, not loss. The evaluation harness is a live spend tracker or simulation engine. The agent modifies bid configs instead of Python training code. The structure is identical.
6. AI Energy Grid Load Forecasting at Utilities
Grid operators need short-term load forecasting models (24-72 hour horizon) that are retrained daily on new smart meter data. The models are sensitive to weather features, calendar effects, and regional anomalies. An agent that runs overnight, trying different feature engineering approaches, model architectures, and ensemble weightings against yesterday's actuals - evaluated on MAPE or RMSE - and produces a log of what improved the forecast, is a direct deployment of this pattern.
The interesting constraint here: the evaluation function includes not just accuracy but reliability. A model that's slightly less accurate but much more stable (lower variance across days) might be the preferred keep outcome. That's a multi-objective version of the loop - still expressible in program.md.
7. Autonomous Drug Candidate Screening for Biotech AI
Small biotech firms running computational screening against protein targets have a similar loop: try a molecular descriptor featurization, run a model training pass on the known active/inactive compound library, evaluate enrichment factor on a held-out set, keep or discard. The agent doesn't need to understand chemistry - it needs to understand how to modify the featurization and model config file and how to read the enrichment metric from the output log.
This is a domain where the overnight loop is natural because each experiment already takes non-trivial compute, the number of things to try is enormous, and the expert's time is expensive.
8. Firmware Compiler Flag Optimization for Embedded Systems
Compiler teams at semiconductor companies run extensive benchmark suites to measure the effect of optimization flag combinations on code size and execution speed. An agent that modifies compiler flag configuration files, runs the benchmark suite, evaluates on a composite score (code size vs. speed tradeoff), and advances or reverts - that's a clean autoresearch application. The benchmark suite is prepare.py. The flag config file is train.py. The tradeoff function is the metric.
This is niche because the "model" here is compiled firmware, not a neural network. The pattern doesn't care.
9. Autonomous A/B Test Hypothesis Pre-Screening
Product teams run A/B tests but are often bottlenecked on what to test. An agent that autonomously generates test hypotheses (UI copy variants, pricing page layouts, onboarding flow modifications), evaluates each against historical user behavior data in a simulation, and surfaces only the hypotheses that pass a minimum expected lift threshold - before any real test runs - compresses the funnel from "idea" to "worth testing." The simulation quality matters, but even a coarse simulator is useful for filtering.
10. Enterprise RAG Retrieval Pipeline Tuning
Enterprise knowledge bases need retrieval pipelines tuned for each customer's document corpus. Chunking strategy, embedding model selection, re-ranker configuration, BM25 weight - all of these affect retrieval quality and all of them are currently tuned manually or not at all. An agent that runs an autoresearch-style loop against a labeled query-document relevance set, trying different RAG pipeline configurations, evaluating on NDCG@10, and keeping improvements - that's a product feature, not an internal research tool. Building a retrieval-augmented system from scratch? See our guide on building AI agents end to end.
What program.md Actually Teaches Us About Agent Design
The most underappreciated part of autoresearch is not the training loop. It's the instruction file. This connects directly to a broader debate about custom AI versus off-the-shelf solutions - the program.md pattern only works if you control the instruction layer.
program.md is where Karpathy has encoded:
- The constraint space - what the agent can and cannot modify
- The evaluation criteria - what "better" means and why
- The decision policy - when to keep, when to discard, when to give up
- The exception handling - what to do when things crash
- The autonomy guarantee - never ask for permission, loop forever
This is a model for how to write agent instructions that actually produce reliable autonomous behavior. Most agent prompts are task descriptions. program.md is a protocol. It defines not just what to do but how to handle every class of exception, how to log results, and what constitutes a decision boundary.
For business deployments, this is the part that matters most. The difference between an agent that runs reliably overnight and one that stops after two iterations to ask "should I continue?" is entirely in how the instructions are written.
Limitations and Honest Caveats
autoresearch requires a single NVIDIA GPU - currently an H100 is the tested platform, though community forks exist for AMD, MacOS (MLX), and Windows RTX setups. This means it's not a zero-infrastructure tool. You need compute.
The loop is also genuinely unsupervised. The agent will try things that don't work. It will crash occasionally. The results.tsv log is your audit trail, but there's no human in the loop during execution. For research contexts, this is a feature. For production model deployment pipelines, you'd want additional guardrails.
The pattern also requires a clean, well-defined evaluation function. The reason autoresearch works is that val_bpb is a single number that definitively orders experiments. If your business problem has a fuzzy or expensive-to-compute evaluation function, the loop becomes harder to run at speed.
The Bigger Picture
Karpathy ends the README intro with a playful note about the "10,205th generation of the codebase" - a riff on where autonomous research loops eventually lead if you run them long enough.
The less playful version: we're at the very beginning of a shift where the iteration cycle for model development - and, by extension, for any system that can be formalized as "modify → evaluate → keep/discard" - becomes the domain of agents rather than humans.
autoresearch is a working implementation of that shift on a problem small enough to understand completely. The architecture fits in one file. The instructions fit in one Markdown document. The metric is one number.
That's the point. Keep it small enough to trust, then extend the pattern to your actual problem.
The businesses that figure out how to deploy this loop - with their own evaluation functions, their own constraint files, and their own program.md - are going to run circles around the ones still doing hyperparameter tuning by hand.
ValueStream AI builds autonomous AI systems for business operations. If you're exploring how autonomous research loops or agentic AI could apply to your organization's specific domain, reach out.