Andrej Karpathy, the AI visionary once at OpenAI and Tesla, is pushing the boundaries of artificial intelligence with a new architecture that ditches the usual retrieval-augmented generation (RAG) approach. His system uses a dynamic markdown library that evolves autonomously, maintained entirely by AI agents.
From Human Coding to Agentic Engineering
Karpathy’s journey into AI has long been about shifting the way we interact with code. Once, developers painstakingly wrote every line of software. Now, Karpathy calls this shift "agentic engineering," where humans don’t write most code anymore but instead direct AI agents to do the heavy lifting.
He believes this transformation is fundamental. Instead of typing code, developers spin up swarms of AI agents that collaborate to build, test, and improve software. The approach not only speeds up development but also changes the skill set needed from human engineers. The key skill becomes managing and orchestrating AI rather than traditional programming.
The LLM Knowledge Base: A Self-Updating Markdown Library
Karpathy’s latest architecture revolves around an evolving knowledge base that sidesteps the typical RAG framework. Instead of relying on static retrieval systems that pull from fixed databases, his design lets AI maintain a markdown library that updates itself over time.
This isn’t just a static repository of facts. The markdown files grow, reorganize, and refine themselves as the AI agents run. The system effectively becomes a living document, continuously enhanced by the AI’s own outputs and discoveries. It’s a radical departure from earlier methods where retrieval and generation were separate and rigid processes.
Autoresearch and the Self-Improvement Loop
Karpathy’s autoresearch project offers a glimpse into how this architecture works in practice. He set up autonomous AI coding agents to run hundreds of experiments over just a couple of days.
These agents didn’t just follow instructions; they designed new experiments, tweaked training code, collected data, and optimized the neural network architectures—all without human intervention.
Over 48 hours, the agents completed 700 experiments and found 20 optimizations that sped up model training. When applied to a larger model, these tweaks reduced training time by about 11%. Shopify’s CEO Tobias Lütke also tested autoresearch with his company’s data, reporting a 19% performance boost after letting the AI run overnight.
The broader implication is clear: AI systems can now accelerate their own development cycles through continuous self-improvement loops. Karpathy describes this as the "loopy era" of AI, where agents improve code, run tests, learn from failures, and iterate endlessly. This cycle could soon be standard practice at leading AI labs.
Rethinking Software Creation with Software 3.0
Karpathy frames this evolution as "Software 3.0," the next stage in how software is made. The first era was rule-based programming, where humans explicitly wrote every instruction. The second introduced neural networks, where models learned from data rather than being hand-coded.
Now, Software 3.0 leverages large language models (LLMs) as programmable engines that understand natural language prompts. Anyone can describe what they want in plain English, and the AI generates the corresponding code. Karpathy likens these LLMs to simulators of human thought—possessing vast memory and the ability to generate complex outputs but also prone to hallucinations and errors.
This shift lowers the barrier to software creation dramatically. It democratizes programming, enabling people without formal coding skills to build applications and tools. But it also demands new ways of managing AI behavior, debugging, and ensuring reliability.
The Future of AI Research and Autonomous Agents
Karpathy’s work hints at a future where AI research itself becomes largely automated. Instead of human researchers manually adjusting models and running experiments, swarms of AI agents could handle these tasks autonomously, collaborating asynchronously like a digital research community.
This vision raises both excitement and concern. On one hand, it promises rapid advances and breakthroughs.
On the other, it edges toward the controversial idea of recursive self-improvement, where AI systems might rapidly outpace human control. Karpathy acknowledges the complexity but views this evolution as an engineering challenge rather than a distant sci-fi fantasy.
His open-source projects, including MicroGPT—a minimal GPT model coded in just a few hundred lines of Python—aim to make these concepts accessible. They help developers and researchers understand, extend, and build upon AI systems that are both powerful and transparent.
Karpathy’s evolving markdown knowledge base and autoresearch agents mark a major step toward AI systems that manage and improve themselves. As frontier labs adopt these approaches, the way AI is developed, deployed, and maintained is set to transform — fast.
