Can you run RAG without internet access?

Yes. Retrieval-augmented generation needs an embedding model, somewhere to store vectors, a language model and a way to feed retrieved passages into the prompt. All four run locally. The internet is only needed to download the models and software once, on a connected machine, before they are moved across.

How much memory does an embedding model need?

Less than a chat model. EmbeddingGemma has 300 million parameters, and the Q8_0 GGUF from ggml-org is about 334 megabytes on disk. The index adds more: at 768 dimensions in 32-bit floats, each chunk costs 3,072 bytes, so 100,000 chunks come to about 307 megabytes.

Why do my llama.cpp embeddings fail with an input is too large error?

Embedding models read the whole input in one pass, so the input has to fit in one physical batch. The error means a chunk is longer than the server's ubatch size. Raise the ubatch size and the context size together, or cut your chunks shorter.

Do I need a vector database for offline RAG?

Not at first. For tens of thousands of chunks, SQLite plus a NumPy dot product is enough and is easy to copy to a site as one file. A dedicated vector database is worth adding when brute force search gets too slow or the index no longer fits in memory.

How do you audit what an offline RAG system retrieved?

Write a log row for every query: the time, the query text, the ids of the chunks returned and their scores. Add the checksum of the embedding model file. Later, anyone can reconstruct which passages an answer was built from and which model version found them.

How to build offline RAG that works air-gapped

A site with no link still has questions. What is the lockout procedure for this pump? Which valve isolates that line? Where is the sampling schedule? The answers sit in manuals and procedures, and an AI system that can read them is only useful if it works when the network is down.

That is a retrieval problem, and it fits on one machine. Offline RAG is retrieval-augmented generation in which the embedding model, the vector store, the language model and the log all run inside the site, with no network call at query time. Here is how to build it, with the parts that usually go wrong.

What does offline RAG need?

Four parts, all local.

Part	Job	What to use in this guide
Embedding model	Turns text into vectors so similar meaning sits close together	EmbeddingGemma 300M as a GGUF file, served by `llama-server`
Vector store	Holds one vector per chunk of your documents	A single SQLite file
Language model	Writes the answer from the retrieved passages	Any model you already run, such as one from our offline vLLM and llama.cpp guide
Retrieval log	Records what was retrieved for every answer	A second table in the same SQLite file

Most guides stop after the first three. The fourth is the one that matters when someone asks, a month later, why the system said what it said.

Which embedding model should you run offline?

Pick a small, dedicated embedding model that your runtime already supports. We use EmbeddingGemma 300M here because Google publishes a model card with exact prompt formats, and ggml-org publishes a ready-made GGUF file for it.

Google describes EmbeddingGemma as a 300 million parameter open embedding model with a 2K token input limit and 768-dimension output (Google AI for Developers). The ggml-org Q8_0 file is 333,590,944 bytes, about 334 megabytes, so the model is small next to the language model it feeds.

On 6 October 2026 Google released EmbeddingGemma 2, a 740 million parameter multimodal model that runs in about 191 megabytes of active memory for text only (Google Developers Blog). Google ships it as LiteRT bundles. Before you switch, confirm that the runtime you use at the site supports it, and rebuild the whole index if you change models.

How do you serve the embedding model with llama.cpp?

Download the GGUF on a connected machine, copy it across with a checksum manifest, then start the server from the local file:

llama-server -m /models/embeddinggemma-300M-Q8_0.gguf \
  --embeddings --host 127.0.0.1 --port 8081 \
  -c 2048 -ub 2048

--embeddings restricts the server to embedding use and is meant for dedicated embedding models, according to the llama.cpp server documentation. Bind it to 127.0.0.1 so nothing else on the network can reach it. Use -m with a path, not -hf, because -hf downloads from Hugging Face.

The /v1/embeddings endpoint is OpenAI-compatible, and its vectors are normalised with the Euclidean norm. That detail makes search simple: for normalised vectors, the dot product is the cosine similarity.

Why do prompts matter for embeddings?

EmbeddingGemma expects a short prompt before every input, and the prompt differs for queries and documents. Google’s model card gives the retrieval formats:

Query: task: search result | query: {text}
Document: title: {title or none} | text: {text}

Google recommends them for retrieval, and skipping them raises no error, so a drop in quality would be silent. Add the prefixes in your own code, always the same way at index time and query time.

How do you chunk, embed and store documents?

Split each document into passages of about 250 words with a 40 word overlap, embed each one, and store the vector as raw bytes. This sketch uses only the Python standard library and NumPy:

import json, sqlite3, time, urllib.request
import numpy as np

URL = "http://127.0.0.1:8081/v1/embeddings"

def embed(text):
    req = urllib.request.Request(URL, json.dumps({"input": text}).encode(),
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return np.array(json.load(r)["data"][0]["embedding"], dtype=np.float32)

def chunks(text, words=250, overlap=40):
    w = text.split()
    for i in range(0, max(len(w), 1), words - overlap):
        yield " ".join(w[i:i + words])

def build(paths):
    db = sqlite3.connect("index.db")
    db.execute("create table if not exists chunk(id integer primary key, src text, body text, vec blob)")
    for p in paths:
        for c in chunks(open(p, encoding="utf-8").read()):
            v = embed(f"title: {p} | text: {c}")
            db.execute("insert into chunk(src, body, vec) values (?,?,?)", (p, c, v.tobytes()))
    db.commit()

Chunks of 250 words stay well under the 2K token input limit. That matters because of the next section.

How do you search the index and log what came back?

Load the vectors into one matrix, multiply by the query vector, take the top few, and write a log row before returning anything:

def search(q, k=3):
    db = sqlite3.connect("index.db")
    rows = db.execute("select id, src, body, vec from chunk").fetchall()
    m = np.stack([np.frombuffer(r[3], dtype=np.float32) for r in rows])
    scores = m @ embed(f"task: search result | query: {q}")
    top = np.argsort(-scores)[:k]
    db.execute("create table if not exists retrieval_log(ts text, query text, chunk_ids text, scores text)")
    db.execute("insert into retrieval_log values (?,?,?,?)",
               (time.strftime("%Y-%m-%dT%H:%M:%S%z"), q,
                json.dumps([rows[i][0] for i in top]),
                json.dumps([round(float(scores[i]), 4) for i in top])))
    db.commit()
    return [(rows[i][1], float(scores[i]), rows[i][2]) for i in top]

Pass the returned passages to your language model as context, and keep the log. Also store the SHA-256 of the embedding model file next to it. With the query, the chunk ids, the scores and the model checksum, you can rebuild exactly what the system saw.

This is a minimal sketch, not a hardened product. It reads the whole index on every query, which is fine for a first site and wasteful at scale.

How big does the index get?

Use arithmetic, not guesses. One chunk at 768 dimensions in 32-bit floats is 768 times 4 bytes, which is 3,072 bytes.

Chunks	Vector size at 768 dimensions	Vector size at 256 dimensions
10,000	30.7 MB	10.2 MB
100,000	307.2 MB	102.4 MB
1,000,000	3.07 GB	1.02 GB

These figures count vectors only, not the chunk text. The 256-dimension column is possible because EmbeddingGemma supports Matryoshka truncation: cut the vector to 512, 256 or 128 dimensions, then re-normalise. Google’s model card reports an English MTEB v2 mean task score of 69.67 at 768 dimensions and 68.37 at 256, so you give up about 1.3 points for a threefold smaller index. Test that trade on your own documents before you commit.

The brute force search itself is 100,000 vectors times 768 numbers, or 76.8 million multiply-adds per query. That is a short job for NumPy on a modern CPU. When it stops being short, that is your signal to add an approximate index.

Which errors break offline RAG?

“input is too large to process. increase the physical batch size”. Embedding models read the whole input at once, so a chunk longer than the server’s -ub value fails. The llama.cpp maintainers describe this on the issue tracker: increase --ubatch-size and keep the context size large enough to match. Our command above sets both to 2,048. Shorter chunks are the other fix.

Results that look plausible but are poor. Check the prompt prefixes first. Nothing errors if they are missing, so compare against a few queries you know the answer to.

Different results after a model change. Vectors from two models cannot be compared. Rebuild the entire index, and record which model built it.

A server that works only on the bench. Something fetched a file you did not stage. Run the whole pipeline with networking removed, as in our offline staging guide, and fix what breaks.

What does a retrieval log buy you?

A log turns “the AI said so” into “the AI was shown these three passages, from these files, found by this model version.” That is the difference between a demo and something an operator, an auditor or an insurer can rely on.

It also handles the harder problem that comes next. A procedure changes, the index goes stale, and no link exists to refresh it. The log tells you which answers used the old text. Our design guide for AI on intermittent links covers how to sync updates when a window opens, and the GPU memory planning guide covers fitting the language model beside the embedder.

Where to go next

If you need retrieval over manuals and procedures at sites that may be cut off, Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off, with a durable record of what ran. See how the platform works, or talk to us about the sites you have in mind.