Caching Embeddings in a RAG Pipeline: Why Recomputing Them on Every Request Doesn't Scale

A typical RAG pipeline turns every chunk of text into a vector before it can be compared against the user's query. That step — calling an embeddings model, whether an external API or something running locally — carries a cost that becomes obvious the moment the system starts seeing real traffic: latency on every call, per-token billing if the provider is external, and GPU or CPU cycles if the model runs in-house. The mistake we made early on was treating that cost as a normal part of serving a query, when in reality most of the time an embedding gets recomputed, the content being vectorized hasn't changed since the last time around.

A typical ingestion pipeline splits a document into fixed-size chunks, generates one embedding per chunk, and stores that vector in a vector store for later retrieval. Up to that point, the embedding is computed once per chunk and the cost is reasonable. The problem shows up in two separate places: when the same document gets reprocessed — because the ingestion pipeline runs again, because a metadata field was corrected and it triggers the whole flow, or because a reindexing job runs without the underlying content having actually changed — and when several data sources share identical or near-identical chunks, which is common in product catalogs, knowledge bases built from repeated templates, or documentation that reuses the same blocks of text in multiple places. In both cases, without a cache, you pay the same compute cost for a result you already had.

What we store and under what key

The first important decision isn't where to store the embedding, but what key identifies it. The initial temptation is to use the document identifier plus the chunk index (document_id + chunk_index) as the cache key, since that's already available in the ingestion pipeline and requires computing nothing extra. That's the decision that ended up breaking, and we come back to it below.

The key that actually works is a hash of the chunk's normalized content, combined with the embeddings model's identifier and version. Normalizing before hashing matters: two chunks that differ only in trailing whitespace, consecutive line breaks, or the capitalization of a word shouldn't produce two different cache entries for what is, in practice, the same semantic content.

import hashlib

def normalize(text: str) -> str:
    return " ".join(text.strip().split())

def cache_key(text: str, model: str, model_version: str) -> str:
    content = normalize(text)
    content_hash = hashlib.sha256(content.encode("utf-8")).hexdigest()
    return f"{model}:{model_version}:{content_hash}"

Including the model and its version in the key isn't optional. If an embedding was computed with one model and the team later migrates to another — or the same provider ships a new version of the model with a different dimensionality or retrained weights — the old vectors aren't comparable to the new ones in the same space. Without that part of the key, a model change leaves embeddings from two different generations mixed in the same cache, indistinguishable until someone notices that searches start returning results that don't make sense.

Where the cache lives

We store cached embeddings in a Postgres table, not an in-memory store like Redis, because the cache needs to survive restarts of the ingestion service, and because the volume of chunks in a document catalog keeps growing steadily — it isn't a bounded working set that comfortably fits in memory. The table is simple: the cache key as the primary key, the serialized vector, and a timestamp of when it was computed.

create table embedding_cache (
    cache_key text primary key,
    embedding vector(1536) not null,
    created_at timestamptz not null default now()
);

The ingestion function checks that table before calling the model:

def get_or_compute_embedding(text: str, model: str, model_version: str, connection):
    key = cache_key(text, model, model_version)
    row = connection.execute(
        "select embedding from embedding_cache where cache_key = %s", (key,)
    ).fetchone()
    if row is not None:
        return row[0]

    vector = call_embeddings_model(text, model)
    connection.execute(
        "insert into embedding_cache (cache_key, embedding) values (%s, %s) "
        "on conflict (cache_key) do nothing",
        (key, vector),
    )
    return vector

The on conflict do nothing exists because the ingestion pipeline runs with parallelism: it's entirely possible for two workers to process the same chunk at nearly the same moment, before either has finished writing its result. Without that clause, the second worker to arrive would fail on a primary key violation caused by a race condition that has nothing to do with an actual bug.

What broke: invalidating wrong after the source content changed

The first version of this system, before we moved the key to a content hash, used document_id + chunk_index as the key. It worked fine in testing, because in testing documents don't change. In production, a document gets updated: someone fixes a paragraph, adds a section, or reorders the content. When the ingestion pipeline reprocessed that document, the chunk at position chunk_index = 3 was no longer the same text as before, but the cache key stayed identical because it depended on position, not content. The result was that the cache confidently returned the old text's embedding for a chunk that now held different text.

This didn't raise any error. No exception alerted anyone. The pipeline finished its run, reported success, and the vector store ended up with a vector that semantically represented the previous content, indexed under the new content. The symptom that finally made it visible was that a search against a recently updated document kept surfacing that document as relevant for questions matching its old version, and as barely relevant for questions about what it now actually contained. Nobody reported a system error: someone reported that "search feels out of date," which, once investigated, turned out to be literally true in the most direct sense possible.

The fix was the key change described above: by basing the cache key on the normalized content hash, an edited chunk automatically produces a different key, because its content is different. No explicit invalidation logic is needed — no DELETE triggered by an update event, no TTL expiring old entries — because a cache entry never stays tied to something that changed: if the content changes, the key changes, and the old entry simply stops being looked up. It still exists in the table, but orphaned, and that gets handled separately.

Purging orphaned entries

Moving the key to content solves invalidation, but introduces a different problem: the embedding_cache table accumulates rows for content no longer referenced by any current chunk, because every document edit leaves behind the entry for the previous text. A separate job, running at low frequency, compares the keys present in embedding_cache against the keys actually used by the active vector store, and deletes in batches the ones with no current reference. Running it infrequently is intentional: the value of those orphaned rows as a cache for content that might reappear — a paragraph that gets reverted, a section that gets restored — outweighs the cost of keeping them around for a few extra days.

What we check before calling an embeddings cache done

Before considering an embeddings cache implementation finished, we verify that the cache key derives from the chunk's normalized content, never from its position or from an external identifier that could drift out of sync with the actual content; that the embeddings model and its version are part of that key, so a model change never mixes vectors from two different generations; that the cache write tolerates race conditions between parallel workers without failing; and that there's a separate process for purging orphaned entries, independent of the invalidation logic, because moving invalidation to content removes the risk of serving a stale vector, but not the need to clean up what that stale content leaves behind. When those conditions hold, caching embeddings stops being a shortcut that can silently return outdated results, and becomes what it should have been from the start: never paying again for a computation whose result you already have.