RAG Over Your Own Docs With pgvector: Chunking, Retrieval and Answers You Can Trust

The first documentation bot I built answered every question confidently. That was the problem. Ask it about a setting that had been renamed two releases ago and it would explain the old one in clean, well-formatted prose, with no hint that the page it was paraphrasing no longer existed. Nobody notices a wrong answer that sounds right, and that's the kind that does the damage.

This post covers how I now build RAG with pgvector over a team's own documentation: product docs, internal runbooks, a support knowledge base. Most of the work isn't the model. It's how you chunk the docs, how you retrieve them, how you force the answer to cite its sources, how you keep the index in step with the docs, and how you measure whether the whole thing can be trusted. I'll go through each in the order I build them.

When RAG Beats Fine-Tuning (and When It Doesn't)

Retrieval-augmented generation means you don't teach the model your docs. You look up the relevant passages at question time and hand them to the model as context. Fine-tuning changes the model's weights. RAG changes what the model reads.

For documentation, RAG wins almost every time:

• Docs change. A fine-tuned model is a snapshot. A RAG index can be updated the minute a page is merged.
• You need citations. RAG knows which passage it used. A fine-tuned model can't tell you where a fact came from.
• Access control. You can filter retrieval by who's asking. You can't filter what's baked into weights.
• Cost of a mistake. When a RAG answer is wrong, you can open the retrieved chunks and see why. Debugging a fine-tune is guesswork.

Fine-tuning still has a place: teaching a format, a tone or a narrow classification task. It doesn't belong in "answer questions about our docs." If you want the theory behind the trade-off, the IBM Generative AI Engineering certificate I took covers both paths, and the RAG and LangChain course in it is a solid foundation.

Here's the whole pipeline I'll build up through the rest of the post:

Chunking Documentation: Headings, Size, Overlap, Metadata

Chunking decides what the model can see, which makes it the most underrated step in the pipeline. If a chunk cuts a procedure in half, retrieval can be perfect and the answer will still be wrong, because the model only got steps 1 to 3 of a 6-step process.

My default for documentation is structure-aware chunking: split on headings first, and only fall back to size-based splitting when a section is too long. Docs are written by people who already grouped related ideas under headings. Use that work instead of throwing it away with a fixed character window.

The rules I start with:

• Split on H2/H3 boundaries. Each section becomes a candidate chunk.
• Carry the heading path. Prefix every chunk with its breadcrumb (Billing > Invoices > Refunds). The heading often contains the exact words the user types, and the chunk body doesn't repeat them.
• Cap the size, then split with overlap. When a section is too long, split it by paragraph with a small overlap so a sentence that bridges two chunks appears in both.
• Never split code blocks or tables. A half table is worse than no table.
• Store metadata. Source URL, title, heading path, product version, language, last-modified date and an access scope. You'll filter on these later.

Different kinds of documents want different strategies:

Doc type	Chunking strategy	Watch out for
Reference docs (API, config)	One chunk per endpoint or setting, with its heading path	Parameter tables split across chunks
How-to guides	One chunk per section; keep numbered steps whole	Procedures cut mid-way
Conceptual pages	Heading-based, paragraph split with overlap	Long intros that dilute the embedding
FAQs	One chunk per question-answer pair	Questions stored without their answers
Changelogs / release notes	One chunk per version entry, tagged with version	Old entries outranking current docs
Runbooks	Whole runbook if short, otherwise per phase	Losing the "when to use this" preamble

The changelog row matters more than it looks. My confident bot from the opening wasn't hallucinating. It was faithfully citing a release note about the old setting, which outranked the current reference page. That's a metadata problem, and you fix it with filters and freshness, not a better model.

Embeddings + pgvector: Schema Shape, Indexes and Filters

I default to Postgres with pgvector for documentation RAG, because the docs index is just another table. It sits next to your users, your permissions and your audit log. You get transactions, backups, row-level security and plain SQL joins, and you don't have to run a separate vector database. On Supabase the extension is one toggle away.

Here's the shape of a chunks table I'd start with. It's a generic sketch, not a schema to copy:

-- illustrative shape only
create table doc_chunks (
  id            bigint primary key generated always as identity,
  doc_id        text not null,          -- stable id of the source page
  heading_path  text not null,          -- "Billing > Invoices > Refunds"
  content       text not null,          -- chunk text, heading path prefixed
  content_hash  text not null,          -- detects unchanged chunks on re-ingest
  source_url    text not null,
  doc_version   text,                   -- product version the page applies to
  access_scope  text not null,          -- who may retrieve this chunk
  updated_at    timestamptz not null,
  embedding     vector(1536),           -- dimension must match your model
  fts           tsvector generated always as (to_tsvector('english', content)) stored
);

A few decisions in there are deliberate. content_hash makes re-ingestion cheap. access_scope lets retrieval filter by permission before anything reaches the model. And the fts column sets up hybrid search in the next section.

HNSW or IVFFlat?

pgvector gives you two approximate nearest-neighbour index types:

• HNSW: better speed-recall trade-off, you can build it on an empty table, and it handles inserts gracefully. It uses more memory and takes longer to build. I use it by default.
• IVFFlat: builds faster and uses less memory, but it clusters the data you already have, so build it after loading, and rebuild it when the corpus changes a lot.

For a documentation corpus, which is usually small next to what these indexes can handle, HNSW with cosine distance is the boring, correct choice. Tune ef_search up if recall looks weak.

Filters and approximate indexes

This one bites people. If you run a vector search with WHERE access_scope = 'public' AND doc_version = 'v3', the approximate index finds the nearest candidates first and the filter runs afterwards. With a selective filter you can ask for 10 results and get 3. Recent pgvector versions add iterative index scans to mitigate this, but I still check result counts in my evals. For very selective filters, partial indexes per scope or a larger candidate pool both help.

Hybrid Retrieval for RAG with pgvector: Vector + Full-Text, Reranking, Top-k

Pure vector search is great at meaning and bad at exact strings. Ask about an error code, a CLI flag or a product name, and the embedding sees roughly "some technical identifier." Full-text search nails those and misses paraphrases. Documentation questions are full of both, so I run both and fuse the results.

The retrieval step I use:

1. Vector search: top 20–40 candidates by cosine distance.
2. Full-text search: top 20–40 candidates by ts_rank against the fts column.
3. Fuse with Reciprocal Rank Fusion (RRF): score each chunk by 1 / (k + rank) in each list and sum. RRF only needs ranks, not comparable scores, which is exactly what you have when combining cosine distance with ts_rank.
4. Rerank the fused top candidates with a cross-encoder or an LLM reranker if latency allows.
5. Keep the final top-k small: usually 4–8 chunks. More context isn't more accuracy. Past a point it's more noise for the model to get distracted by.

Supabase's hybrid search guide uses the same RRF approach in a single SQL function, and it's a good reference. I used a variant of the idea in my public local-rag-chatbot lab: it generates several rewrites of the user's question (multi-query), retrieves for each, and fuses the lists with RRF. That lab uses Chroma rather than pgvector, but the retrieval logic carries over unchanged.

Query rewriting deserves a mention on its own. Users ask "why is my invoice wrong?" and the docs say "billing discrepancies." Generating two or three rewrites before retrieval is one of the cheapest recall improvements I know.

Grounded Answers: Citations and Refusal When Context Is Missing

Retrieval gets the right passages in front of the model. The generation prompt decides whether the model sticks to them. Here's the contract I put in the system prompt, in sketch form:

You answer questions using ONLY the numbered sources below.
- Cite every claim with its source number, like [2].
- If the sources do not contain the answer, say you don't know
  and suggest where the user might look. Do not guess.
- If sources conflict, prefer the most recent and say so.

Sources:
[1] {heading_path} ({source_url}, updated {updated_at})
{content}
...

Three things make this work in practice:

• Number the sources, then check the numbers. After generation, verify in code that every citation points to a chunk that was actually retrieved. A citation to [7] when you sent five sources is a hallucination you can catch for free.
• Make refusal a first-class answer. "I couldn't find this in the docs" plus a link to search is a good answer. Grade it as a success in your evals when the answer really isn't there, or the model learns that guessing pays.
• Add a retrieval gate. If the best fused score is below a threshold, or no chunk survives the filters, skip generation and return the fallback. The cheapest hallucination is the one you never generate.

Treat retrieved content as untrusted input, too. A docs page, a pasted ticket or a community answer can contain text that reads like instructions. I cover that attack surface in Red-Teaming Your LLM Agent. If your assistant can also do things, not just answer, keep the retrieval layer away from the action layer. That separation is the core of Separate Talking From Acting.

Keeping the Index Fresh When Docs Change

A stale index is worse than no index. It's confidently wrong, as my first bot showed. Freshness is a pipeline problem, and I treat it like any other data pipeline:

• Trigger on change, not on a calendar. Re-ingest when the docs repo merges or the CMS publishes. A nightly full rebuild is a fallback, not the strategy.
• Diff by hash. Re-chunk the changed page, compare content_hash per chunk, and only re-embed what actually changed. Embedding is the expensive step.
• Delete aggressively. When a page is removed or renamed, its chunks must go in the same transaction. Orphaned chunks are where the "renamed setting" bug lives.
• Version-tag everything. If you document multiple product versions, filter by the user's version, or at least rank current docs higher.
• Re-embed on model change. Vectors from different embedding models aren't comparable. Changing the model means a full rebuild into a new column or table, then a switch-over.
• Show the date. Put updated_at in the citation. Users trust an answer more when they can see it came from last week's page.

Evaluating a RAG Bot: Retrieval Hit Rate, Faithfulness, Answer Quality

You can't tune what you don't measure, and "it answered my five test questions" isn't a measurement. I split RAG evaluation into three layers, because each one fails for different reasons:

Layer	Question it answers	How I measure it
Retrieval	Did the right chunk make it into the top-k?	Hit rate and MRR against a labelled set of question → expected chunk
Faithfulness	Is every claim in the answer supported by the retrieved chunks?	LLM judge checks each claim against the sources; citation validity checked in code
Answer quality	Did it actually help? Did it refuse when it should?	Rubric-based LLM judge plus a set of unanswerable questions that must get "I don't know"

The unanswerable set is the most valuable thing in the suite. Write questions your docs genuinely don't cover (near-misses, deprecated features, things users assume exist) and require a refusal. Without them, you're optimising a bot that never says no.

Keep retrieval evals separate from generation evals. When answer quality drops, the first question is always "did retrieval change?" If the hit rate held and faithfulness dropped, look at the prompt or the model. If the hit rate dropped, look at chunking, filters or the index. And run each eval several times before you trust a delta: say faithfulness moves by a couple of points between two runs of the same config. That's noise, not a regression. I go deeper on scoring, judges and run-to-run variance in How to Build an Eval Gym for Conversational AI Agents. If your RAG assistant also calls tools, tool-call evals cover that half.

Then wire it into the workflow. A docs change that drops the retrieval hit rate should fail CI, the same way a broken test does. Tracing every production query with something like Langfuse gives you real questions to grow the eval set from, and that matters more than any synthetic set you write up front.

A Launch Checklist for Your RAG pgvector Assistant

This is the checklist I go through before a docs assistant gets real users.

Corpus and chunking

• Every source is listed, owned, and has a re-ingest trigger
• Chunks split on headings, carry their heading path, and never cut code blocks or tables
• Metadata includes source URL, version, access scope and updated_at

Storage and retrieval

• pgvector enabled, HNSW index on the embedding column, embedding dimension matches the model
• Full-text column and index in place; hybrid retrieval fused with RRF
• Access-scope filtering applied in the query, before results reach the model
• Filtered queries checked for "asked for 10, got 3"

Generation

• System prompt requires citations and allows refusal
• Citation numbers validated in code after generation
• Retrieval gate returns a fallback when nothing relevant is found
• Retrieved content treated as untrusted input

Freshness

• Hash-based incremental re-embedding
• Deleted and renamed pages remove their chunks
• Embedding-model change has a documented rebuild path

Evaluation and operations

• Labelled retrieval set, faithfulness judge and an unanswerable-question set
• Evals run in CI on docs and prompt changes, several runs per config
• Production queries traced and sampled into the eval set
• A clear path to a human when the bot can't help: see Human Handoff for AI Chat Agents

How to start this week

1. Pick one doc section people actually ask about, not the whole site.
2. Write 30 real questions from support threads or chat history, plus 10 the docs can't answer.
3. Chunk by heading, load into a pgvector table, and measure the retrieval hit rate before writing a single prompt.
4. Add full-text search and RRF, and measure again.
5. Only then add generation, citations and the refusal path.

Measuring retrieval first changes how you spend your time. Most of my improvements have come from chunking and filters, not from prompt edits.

Conclusion

RAG over your own docs isn't a model problem. It's a data pipeline with a language model at the end. Chunk along the structure your writers already gave you, retrieve with both meaning and keywords, make the model cite or refuse, keep the index in step with the docs, and measure each layer separately. Do that and Postgres with pgvector is all the infrastructure you need.

If you're starting from scratch, clone local-rag-chatbot to play with multi-query retrieval and RRF locally, then port the pattern to pgvector. And if you build the assistant with a coding agent, write the spec first. What Is Spec-Driven Development? explains why that pays off on exactly this kind of multi-stage system.