Still Optimizing Your RAG for Agents? LLM Wikis Might Be Here to Stay (for Now)

If you work with AI agents, you have probably heard of RAG, short for Retrieval-Augmented Generation. It is a technique for giving an AI agent or multi-agent system access to knowledge. However, it has serious drawbacks, and there might be a better new (or old, depending on your perspective) idea from Andrej Karpathy.

Quick Recap: How RAG Works and Where It Breaks#

In a RAG system, the provided data sources, mostly documents, are split into chunks and transformed into embeddings: numerical representations of those chunks. The embeddings are loaded into a vector database, and that’s it. The details are more complex when you build this in Azure Foundry, AWS, or any other stack, but in a nutshell, that is the process.

When the agent receives a question (a query), the system performs a similarity search, reads the results, and the LLM formulates an answer from them.

This design has drawbacks. First, the agent has to run the search again and again for every question. That is not only an efficiency problem (and, with consumption-based billing, increasingly a cost problem), but also a quality problem:

• No verification against the source. There is often no check of the answer against the originally loaded documents, so the agent can answer completely wrong even though the correct information was provided.
• Stale duplicates. If someone uploads a newer version of a document, both versions can remain in the pipeline and the vector database. Which one is “correct” when the similarity search returns both?
• Unstable reasoning. The model reasons over whatever the similarity search returned and draws a conclusion, and that conclusion can change over time.

Depending on the use case, these points can be more severe, less severe, or not relevant at all, and mitigation strategies exist. But the core problem remains.

It gets more interesting in enterprise environments, where knowledge is everywhere: some in SharePoint, some in Confluence, some in code comments, some in databases, and, let’s be honest, some only in the head of the senior engineer who has been around for 15 years. From what I have seen, most initiatives to consolidate knowledge failed. Even in billion-dollar Fortune 500 companies, critical confidential knowledge is managed in Excel. I have seen highly critical engineering knowledge in a large organization kept in one single, huge Excel sheet. So no, the problem is not solved.

LLM Wiki and How It Can Help#

A few months ago, Andrej Karpathy proposed the idea of an LLM Wiki, which tries to resolve some of the problems above. The core idea: knowledge is synthesized and stored persistently, instead of being regenerated from scratch every time someone asks a question.

When the system receives a new source, it reads it once. It draws conclusions, extracts knowledge, reasons over potentially important contradictions, and then creates or updates an entry in a wiki as a human-readable Markdown file. This is similar to how skill files are managed today, with one key difference: the wiki is the system’s actual knowledge source, not a description of how the system should operate.

The next time a question on that topic comes up, the system goes back to the wiki and uses the stored information as its knowledge source, instead of retrieving and synthesizing the same knowledge at query time over and over again.

The idea is already moving beyond Karpathy’s original proposal. Google, for example, has published an “Open Knowledge Format” (OKF). The concept is very new: at the time of writing, Google’s version was at 0.2, published only weeks after the two tech leads first introduced it. Expect new developments soon, because the shortcomings of RAG are pressing.

Google explicitly states that OKF is not a new system or implementation, but a standard for structuring these wikis: living knowledge bases that are updated regularly but must remain manageable by humans. They propose a structure like this (see the link above for details):

sales/
├── index.md
├── datasets/
│   ├── index.md
│   └── orders_db.md
├── tables/
│   ├── index.md
│   ├── orders.md
│   └── customers.md
└── metrics/
    ├── index.md
    └── weekly_active_users.md

I think OKF will follow the same path as skill files: a standard emerges, and to avoid confusion the big vendors follow, implement, and use it, keeping the AI landscape at least a little less confusing.

Key Advantages of LLM Wikis, and Is It Really New?#

Let’s be honest: storing knowledge in structured files, linking information together, and keeping it under version control is not new. Not even remotely. Wikis have existed for decades, developers have kept documentation next to code forever, and metadata-as-code is nothing revolutionary.

The interesting part (at least for me) is not the storage format. Surprise, surprise. It is who creates and maintains the knowledge. Instead of humans constantly trying to keep a corporate wiki alive (which, let’s be honest, everyone hates), an AI system consumes new sources, identifies changes, updates related files, and maintains the knowledge continuously. We humans can read it, use it, and correct it, which is much harder, or at least takes more effort, in a RAG system.

Does this make LLM Wikis better than RAG in every respect? Of course not. But they have five key advantages:

1. Synthesize once, reuse often: Knowledge is synthesized once, reused, and continuously updated.
2. Human-readable: The knowledge lives in Markdown files.
3. Versioning and change tracking: Put the wiki in a Git repository and version-control it.
4. Contradictions handled at ingestion: For example, the agent opens a pull request that a human has to approve.
5. Portability: The wiki is just a Git repo, so you can clone it elsewhere and use it.

Does an LLM Wiki Solve All Your Knowledge Problems?#

TL;DR: no. It is not the holy grail, and there are still open questions:

• Who decides which source wins? A human reviewer (slow again, like a traditional wiki) or a machine (giving control away again)?
• How do you spot incorrect data once it has entered the wiki? Manual control?
• How do you manage millions of knowledge objects and keep the data compliant (GDPR and more)?
• How quickly does knowledge become outdated, and who makes sure outdated knowledge is removed from the wiki?

Traditional RAG systems are not dead (yet), but knowledge management for AI agents is evolving rapidly.