GuideIntermediate

RAG without the hype · when you ACTUALLY need it (and when it's throwing money away)

If you made it this far in the series, you've already given your AI memory and learned how not to lose your work. The next rung shows up when your AI has to handle A LOT of information: dozens of PDFs, a huge manual, transcripts, your whole business knowledge base. That's where RAG shines: instead of feeding everything to the AI every time (crazy expensive in tokens), it hands over only the relevant little pieces — and that's where you save a ton of money. BUT there's an uncomfortable truth almost nobody tells you: if your information is small, setting up a RAG just overcomplicates things and costs you more. In this guide you'll understand what RAG is in plain terms, why it saves tokens, the exact rule (from Anthropic itself) for knowing when you SHOULD, and the THREE ways to implement it depending on your level: no code, with an open-source repo, and the RAG behind a super-app — the very one we use, from the inside. No hype, in order, step by step.

Jun 19, 202612 min

You're on rung 3 of the series

This guide assumes you've already seen the one on [memory for your AI](/recursos/memoria-para-tu-ia-herramientas) (rung 1) and the [GitHub](/recursos/guarda-todo-en-github-antes-de-que-la-ia-lo-rompa) one (your safety net). Here we step up a level: what to do when there's TOO MUCH information for the chat. You don't need to code.

When does the need show up? (the exact moment)

The need arises at a very specific moment: when your AI has to work with a lot of information at once and starts to fail. Typical signs: you upload 30 PDFs and the AI "forgets" half of them; you paste in a long manual and it answers by mixing things up or making stuff up; or you have to paste the same giant document into every conversation because it doesn't hold onto it. That's when someone tells you "you need a RAG."

The double pain this guide tackles

There are two pains, not one. The first: your AI makes up or ignores your long documents, or it blows up your token bill by feeding them in whole over and over → expensive and unreliable. The second, the flip side: out of fear of falling short, you set up a RAG you didn't need → free infrastructure and complexity. A well-placed RAG solves the first (and saves a lot); a badly-placed one is the second. This guide gives you the rule to get it right.

RAG in plain terms (without a single technical word)

RAG stands for "retrieval-augmented generation." It sounds awful, but the idea is simple: instead of feeding ALL your documents to the AI every time (crazy expensive, and impossible if there are many), you store your documents separately and, when you ask something, a search engine retrieves only the relevant little pieces and hands them to the AI so it answers from those. Search first, answer after.

Picture it like this · the librarian

Without RAG, you're asking your AI to read the entire library every time you ask it something — slow, expensive, and it gets dizzy. With RAG, you have a librarian who knows the shelves: you ask, they go, they bring back the 3 books that matter, and the AI answers from those. It doesn't read everything: it brings back just what's needed.

The big bonus: because the AI answers from specific pieces of YOUR documents, it can cite the source for you ("this comes from the manual, page 12"). That's what kills the hallucinations: if it's not in your documents, it doesn't make it up.

The big benefit: RAG saves TONS of tokens (when the corpus is large)

Here's the main reason RAG exists, and it's no small thing: saving money. Every word you feed the AI costs tokens, and tokens cost cash. If you have 5,000 pages of documents and you paste them all in on EVERY question, you pay a fortune (and often they don't even fit). RAG solves that: instead of sending the 5,000 pages, it sends only the 3 or 4 little pieces that actually answer your question.

The math that explains it

Without RAG: 5,000 pages × every question = thousands of tokens each time. With RAG: only the relevant pieces = a few hundred tokens. Same result, a fraction of the cost. That's why RAG is the standard way to give an AI access to large information without going broke. That saving is real and ongoing… as long as your corpus is genuinely large. If it's small, the math flips (we cover that in the golden rule, below).

Why are they called "chunks"? (the key piece of RAG)

Chunk means "piece" or "bit." It's the central piece of RAG, so it's worth understanding. Your documents are NOT stored whole: they're cut into small pieces — each chunk is usually a paragraph or a couple of them — and each piece is stored with a kind of "fingerprint of its meaning."

Why chop it up instead of storing the whole document?

Because when you ask something, you don't want the RAG to bring back a 300-page book — you want the exact paragraph that answers. Chopping into chunks lets it bring back exactly that: the small, precise unit that's useful, not the whole document. Less noise, sharper answers, and far fewer tokens. That's what makes RAG precise AND cheap at the same time.

The golden rule · do you really need RAG?

Here's the fact almost no "guru" tells you, and it comes from Anthropic itself (the creators of Claude). The rule is surprisingly generous:

What Anthropic says, verbatim

"If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG." Translation: if all your material fits in ~500 pages, you DON'T need RAG. You paste it all into the AI and you're done. Simpler, more accurate, and with today's "prompt caching," cheaper too.

Why does it matter so much? Because 500 pages is a LOT for most projects. Your brand manual, your templates, your business policies, a couple of books… it usually fits. And if it fits, setting up a RAG just overcomplicates your life for no reason. RAG is for when you genuinely go past that size: huge libraries, thousands of documents, data that changes all the time.

You (really) need RAG if…

Your material does NOT fit in ~500 pages (thousands of documents, a large database, years of transcripts).

The information changes often and you don't want to re-paste everything each time.

You're building an app for many users who query a large corpus (e.g. an assistant that answers questions about all your documentation).

You need source citations systematically and at scale.

You DON'T need RAG (save yourself the hassle) if…

Your material fits in ~500 pages → paste it into the context. With prompt caching, repeating it is cheap — cheaper than setting up a RAG.

You just want the AI to remember your rules and your project → that's memory (rung 1), not RAG.

You have a handful of documents you check now and then → upload them to the chat when you need them.

You're just getting started → you're almost certainly one or two rungs below RAG.

The judgment habit (the most important thing)

Before stepping up a rung, make it a reflex to ask yourself: "does this fit in one conversation, or is it genuinely too much?" Don't level up because it's trendy. Start simple (memory/context), and move up to RAG only when the rung below actually falls short. That one question saves you time and money over and over.

The full staircase (so you see where RAG fits)

RAG isn't the first step or the last — it's a rung on a staircase. Climb it in order; don't skip rungs just because it's trendy:

From simple to powerful

1 · Memory / context — your rules and your project in a file or in the chat. For almost everything. (Rung 1 of the series.)

2 · Paste everything + caching — if your material fits in ~500 pages, feed it all in. No RAG.

3 · No-code RAG — when it's too much: a search engine that brings back the relevant pieces. (This guide.)

4 · "Real" RAG — for apps at scale, with a vector database. Open-source, now with a bit of technical work.

5 · Knowledge graph — when you also need the AI to understand HOW everything connects. (Graphify, at the end.)

The 3 ways to implement a RAG (pick based on your level)

There isn't just one way to set up a RAG: there are three, from least to most effort and power. What matters is choosing the one that actually fits you, not the flashiest one. Here they are, and then we'll go through them one by one:

Your three options

Way 1 · No code — you drag documents into a ready-made tool (NotebookLM, Custom GPT). Zero technical work. For most people.

Way 2 · With an open-source repo — you install something like RAGFlow or AnythingLLM, full control, free. Your coding agent helps you.

Way 3 · Custom-built (super-app) — you build your own engine for an app at scale. Engineer-level. I'll show you what ours looks like.

Way 1 · Set up a RAG WITHOUT writing code (the easy path)

If you passed the filter and RAG really is for you, good news: you don't need to program or set up databases. There are tools where you drag in your documents and that's it — you've got your smart search engine with citations. Two that anyone can use today:

NotebookLM (Google · free)

You upload your documents (PDFs, websites, text) to a "notebook" and ask questions in natural language. It answers citing the exact source of every claim — goodbye hallucinations. It's pure RAG, without you having to know what RAG is. (Size and source-count limits change by plan; check the current ones when you use it.)

Custom GPT (ChatGPT · paid plan)

You create a GPT, upload your files, and ChatGPT builds the search engine under the hood automatically. You just talk to it. Ideal if you already pay for ChatGPT and want an assistant that knows YOUR documents.

Anti-hallucination rule

When you set up your no-code RAG, ALWAYS give it this instruction: "answer only with what's in my documents and cite the source; if it's not there, say so clearly instead of making it up." That turns your RAG into a reliable source, not a fortune-teller full of confidence.

Way 2 · With an open-source repo (more control)

When your project grows or you want control on your own server, there are powerful, free open-source tools. The two friendliest for non-experts (your AI helps you install them):

infiniflow/ragflow

REPO

RAGFlow — the leading open-source RAG engine, built for non-developers: you assemble your pipeline visually and it understands complex documents (Word, PDFs, Excel, scans, images). It answers with source citations. Blends RAG with agent capabilities.

PythonApache-2.0View on GitHub

Mintplex-Labs/anything-llm

REPO

AnythingLLM — an "all-in-one" desktop app to have your own RAG chat over your documents, local and private. "Stop renting your intelligence, make it yours." Ideal if you want your data to never leave your machine.

JavaScriptMITView on GitHub

Paste it to your agent · install a serious RAG with a repotexto

I want to set up an open-source RAG over my documents using one of these repos:
- RAGFlow (https://github.com/infiniflow/ragflow) — visual RAG, ideal if I want something complete.
- AnythingLLM (https://github.com/Mintplex-Labs/anything-llm) — desktop app, local and private.

Guide me step by step in simple language, assuming I don't know how to code:
1. Help me choose which one suits me based on my case: [describe your documents, whether you want everything to be local/private, and your operating system].
2. Tell me how to install it (with its requirements, e.g. Docker) and do it yourself wherever you can.
3. Help me upload my documents and create the knowledge base.
4. Configure it to answer ONLY with what's in my documents and to ALWAYS cite the source.
5. Give me 3 test questions to confirm it's reading my documents properly.

If there's a step I have to do by hand, tell me with exact instructions.

Leave it to your coding agent

Don't install anything by hand if you use an agent like Claude Code: the prompt above makes it carry the heavy lifting. You just choose the repo, provide the documents, and verify. Remember to save your progress in GitHub before installing new stuff (see the GitHub guide in the series).

Way 3 · The RAG behind a super-app (how ours works)

The third way is the most serious: when you build a super-app — a platform with tons of information and many users — no turnkey tool cuts it. That's where you build your own custom RAG engine. It's engineer-level (you do it yourself, or your team does, directing coding agents), but we'll show it to you from the inside, no hype, so you can see how far it can go: this is how the RAG engine we use in our applications works.

Heads up · this is the RAG, a PIECE — not the full brain

Let's clear something important up so there's no confusion: what you see here is the search engine (the RAG), which is ONE piece. The full "brain" of an app — what's called the brain — is more than that: it combines this RAG with each user's memory and, sometimes, a graph of connections. The RAG is the "what it knows about your documents"; the brain is the whole brain. We cover that in the next resource in the series.

The pieces of our RAG (in production)

Vector store: PostgreSQL with the pgvector extension (on Supabase) + an HNSW index to search fast across hundreds of thousands of vectors.

Embeddings: the model that turns each chunk into its "meaning fingerprint" — Gemini with 768 dimensions, on the free tier (cost $0).

Scale: more than 15,000 chunks indexed (documents, code, notes) — impossible to fit in a chat; here RAG isn't a luxury, it's the only way.

Hybrid search: 70% by meaning (vector) + 30% by exact word (text), with re-ranking that surfaces the most relevant. More accurate than either one alone.

A self-healing index: the system re-indexes what's broken, flags what's stale, and discards what's useless — the RAG keeps itself healthy on its own. (This is search-engine maintenance, not the "user's memory" — that's another piece, the brain's.)

Chunks of ~200 tokens each: the sweet spot between "bring back the exact paragraph" and "don't lose the context."

What this teaches you

Notice the pattern, because it's the lesson of the whole series: start cheap (free embeddings, a database you already have) and get sophisticated only when scale demands it (15,000+ pieces). You don't need this to get going — but now you know where a RAG grows to when you build something genuinely large. Move up a rung only when the one below actually falls short.

In NeuralOS, this RAG engine is part of the product

That custom RAG engine is exactly what we're bringing to NeuralOS: every app and every agent with its own search over its large-scale documents, without you setting up pgvector or embeddings. The vision is already drawn in the interface; the cloud engine is part of the path we're building. The idea: that you get Way 3 without having to be an engineer.

Before the prompt: an honest expectation

Way 3 is not a copy-and-paste that comes out perfect on the first try — that would be selling you hype. It's an engineering project of several days, with architecture decisions. That's why the prompt below doesn't say "build it for me now": it's an architect's prompt that makes the AI PLAN before coding. That order — decide the architecture first, code after — is exactly what avoids the expensive mistakes at the start.

Copy it and paste it to your coding agent (Claude Code, Cursor) when you're about to build a custom RAG. It comes with the decisions that work for us built in, so it starts on solid ground instead of improvising:

Paste it to your agent · architect of a custom RAG (Way 3)texto

You're going to help me build a custom RAG for a large-scale application. Do NOT write code yet. First, PLAN with me, because the most expensive mistake is coding without having decided the architecture.

STEP 1 — Interview me (one question at a time, in simple language):
- What information will the RAG index and how much? (document types, approximate volume)
- How many users and how many queries per day do you expect?
- Does the data change often? Do I need to delete/update chunks?
- Is it multi-tenant (several customers with separate data)? Is there sensitive data?
- What stack do I already use (database, language, where it's hosted)?

STEP 2 — Propose the architecture, using these proven decisions as a starting point (and tell me whether, in my case, it's worth changing them and why):
- Vector store: PostgreSQL + pgvector with an HNSW index (if I already use Postgres/Supabase, reuse it).
- Embeddings: a cheap or free model (e.g. Gemini), storing the dimension you choose.
- Chunking: pieces of ~200 tokens with a little overlap, and store metadata (source, date, section).
- Hybrid search: combine vector + exact text, and a re-ranking to surface the most relevant.
- Isolation: if it's multi-tenant, ALWAYS filter by the customer id on every query.

STEP 3 — Give me a PHASED PLAN (what to build first, what to leave for later) with a minimal phase that works end-to-end before optimizing.

STEP 4 — List the 5 most common mistakes in a production RAG and how to avoid them from day one (e.g. not filtering by tenant, badly-cut chunks, not handling updates, trusting without citing the source, not measuring answer quality).

When we approve the plan, only then do we start building phase 1, step by step, and we save the progress in GitHub at every phase that works.

Combine it with the rest of the series

This is the moment where EVERYTHING in the series comes together: save every phase in GitHub (your safety net), audit with the C-A-R protocol before calling a phase good, and if the project becomes huge, put a graph on top of it (Graphify). A well-built production RAG rests on all the previous rungs.

Prompt 1 · Diagnosis — do I need RAG or not?

Before setting anything up, let your AI tell you which rung you're really on. Copy it and paste it to ChatGPT, Claude, or your agent:

Paste it to your AI · rung diagnosistexto

I want to know if I really need a RAG or if I'm overcomplicating things. Ask me these questions one by one, wait for my answer, and at the end tell me which rung I'm on and why:

1. How many documents do I have, roughly, and what size? (a rough total-pages estimate)
2. Does that information change often or is it stable?
3. Do I need it to cite the source of each answer?
4. Is it just for me, or for an app that many people will use?
5. What AI tool do I use today?

Rules for your recommendation:
- If all my material fits in ~500 pages (about 200,000 tokens), tell me I do NOT need RAG: that I should paste everything into the context, it's simpler and more accurate.
- If it's clearly more than that, or changes a lot, or is for many users, recommend RAG and tell me whether the no-code path (NotebookLM / Custom GPT) is enough or whether I need something open-source.
- Don't push me toward over-engineering. Recommend the SIMPLEST rung that solves my case, and justify it.

Prompt 2 · Build your no-code mini-RAG, hallucination-proof

If the diagnosis said "yes, no-code RAG," this prompt guides you to set it up and, above all, to verify that it's really reading your documents and not making things up:

Paste it to your AI · build and verify your RAGtexto

I'm going to set up a no-code RAG to query my documents. Guide me step by step, in simple language, assuming I don't know how to code.

1. Recommend between NotebookLM (free) and a ChatGPT Custom GPT based on my case: I have [describe your documents: how many, what type, what I'll query them for].
2. Give me the exact steps (with clicks) to create the space and upload my documents.
3. Write me the instructions I should give it so it answers ONLY with what's in my documents, cites the source, and clearly says "it's not in the documents" instead of making it up.
4. Give me 3 test questions to confirm it's really reading my documents (and not answering from general memory).
5. Tell me how to notice if it's making things up, and what to adjust if it happens.

Keep it simple. If there's something I can only do myself on the website, tell me with the exact clicks.

Common mistakes (so you don't fall for them)

Setting up RAG when pasting the document was enough

Mistake number one. If your material fits in the context, RAG only adds complexity and cost. First the 500-page rule, then you decide.

Confusing RAG with memory

RAG does NOT remember your conversations or your preferences — that's memory (rung 1). RAG searches your documents. They're different things that complement each other, not rivals.

Trusting without verifying the citations

A RAG can bring back the wrong piece and sound very confident. That's why prompt 2 includes test questions: trust, but verify that it cites the source and that the source says what it claims.

The next rung · from one piece to the full brain

RAG is a powerful piece, but it's just one: it searches your documents. When you build a real app, that app also needs to remember each user and, sometimes, understand how everything connects. All three together — RAG + memory + graph — form what's called a brain, your app's own brain. That's the next resource, and it brings together everything you've seen.

The brain · give your app its own brain (the next rung)

RAG is one of its pieces. The brain brings them all together: what your app knows and what it remembers about each user.

Your AI's memory · rung 1 (if you skipped it)

Before RAG, what you need is almost always simple memory. Start here.

#rag#documentos#contexto#sin-codigo

Ready to build?