Project

local-rag-chatbot

Fully local RAG chatbot on LangGraph, Chroma and Ollama — Multi-Query Fusion with Reciprocal Rank Fusion, SQLite chat history, zero API keys.

  • #ai
  • #rag
  • #langgraph
  • #python
  • #open-source

local-rag-chatbot is an open-source RAG (Retrieval-Augmented Generation) chatbot I built that runs entirely on a laptop: drop documents in a folder, run one command, and chat with them. No API keys required. Source: github.com/AleSZanello/local-rag-chatbot.

The problem

I built it because many "local RAG" projects still call a cloud API for embeddings, ship very large models or skip persistence. This one keeps every piece local: CPU embeddings with sentence-transformers (about 80 MB), a file-backed Chroma vector store, and a roughly 2 GB local LLM through Ollama (llama3.2:3b in the quick start).

Architecture

The chat brain is a LangGraph graph rather than a single-shot RetrievalQA chain, with seven nodes:

  1. ingest_question — pulls the latest user message into state
  2. expand_queries — the LLM writes three paraphrases (Multi-Query)
  3. retrieve — a Chroma similarity search per paraphrase
  4. fuse_rrf — Reciprocal Rank Fusion into the top-k unique chunks
  5. grade_context — a binary "is this enough to answer?" check
  6. generate — an answer with [source:chunk] citations, or
  7. fallback — "I don't know based on your docs"

The grade step is a conditional edge, so the bot declines instead of hallucinating when retrieval comes back thin. RRF is a pure, deterministic function (score = Σ 1/(k + rank), k = 60), easy to unit-test. Conversations persist through a SqliteSaver checkpointer, and the REPL supports /new, /sources, /reset and /exit.

Ingestion and providers

Ingestion loads Markdown, text and PDF, chunks recursively, and is idempotent: files are hashed and already-indexed ones are skipped. An LLMProvider protocol makes the model pluggable through one environment variable — Ollama by default, or OpenAI, Anthropic Claude or Google Gemini. Tests use a fake chat model and run offline.

The retrieval and graph patterns come from my Agentic AI Engineering Specialization specialization, which I re-implemented against a real vector store.

Status

Live, 2026, MIT licensed.

Related