RAG with pgvector in Postgres: why chunking decides relevance before the model does

A RAG (retrieval-augmented generation) system built on Postgres with pgvector looks, on paper, like a solved problem: generate embeddings for the documents, store them in a vector column, index that column, and on every query search for the nearest fragments by cosine similarity. What breaks in practice is almost never the embedding model or the Postgres index. It breaks earlier, in how the documents were cut into fragments before they were embedded. A chunking strategy that slices a paragraph in half, separates a heading from its content, or mixes two unrelated sections into the same fragment ruins retrieval even when the embedding model is good and the index is configured correctly, because the stored vector no longer represents a coherent concept to search against.

How similarity search is structured in pgvector

pgvector's mechanics are simple, which makes it easy to underestimate how much weight sits on what happens before a row ever reaches it. You add a vector column to the table holding the fragments, index that column, and compare with the cosine distance operator:

create extension if not exists vector;

create table document_chunks (
  id bigserial primary key,
  document_id bigint not null references documents(id),
  chunk_index int not null,
  content text not null,
  embedding vector(1536)
);

create index on document_chunks
  using hnsw (embedding vector_cosine_ops);

And the retrieval query at runtime looks for the k fragments closest to the embedding of the user's question:

select content, embedding <=> $1 as distance
from document_chunks
order by embedding <=> $1
limit 5;

This schema has no way of knowing whether the content in a given row is a complete, self-contained paragraph or half a sentence cut off by an arbitrary character limit. Postgres correctly finds the vector closest to the question; it has no way of knowing whether that vector represents a whole idea or a fragment of one. That responsibility sits entirely on how the rows in document_chunks were built, and that's where whether the search returns anything useful gets decided.

Why chunking outweighs the embedding model

An embedding of a text fragment is, in practice, a vector summary of its semantic content. If the fragment holds a single coherent idea, the vector points clearly toward that idea in embedding space, and a related question lands close to it. If the fragment contains the tail end of one idea and the start of a completely different one -something that happens constantly when text is split by character count without looking at the content- the embedding model has no choice but to produce a vector that is, in essence, an average of both ideas. That vector ends up close to neither of the questions you'd expect it to answer, because it doesn't represent either one precisely.

The most common version of this problem isn't dramatic: it's a section heading that lands in one fragment while the content of that section lands in the next. A question about "how to configure the retry timeout" can land semantically very close to the heading "Retry configuration" and not to the fragment that actually contains the steps, if the chunk boundary split the two apart. The system retrieves the title without the content, the generating model receives a fragment that names the topic without explaining it, and the answer ends up vague or made up, not because the generating model hallucinates for no reason, but because the context it was handed was already incomplete.

Naive chunking by character count

The first version we tried split text into fixed-size chunks without looking at the document's structure at all:

function chunkByCharacters(text, size = 1000, overlap = 100) {
  const chunks = [];
  let start = 0;

  while (start < text.length) {
    const end = Math.min(start + size, text.length);
    chunks.push(text.slice(start, end));
    start += size - overlap;
  }

  return chunks;
}

It's about as simple an implementation as you can write, which is exactly why it's tempting to leave it as-is when the immediate goal is just "make the RAG pipeline work." The problem shows up in any document with real structure: technical manuals, API documentation, contracts with numbered clauses. A code sample can get split down the middle between two fragments, a table can lose its header row to a different fragment than its data, and a numbered list can start in one fragment and end in the next without either one making complete sense on its own. The hundred-character overlap helps a few cuts avoid landing at the worst possible point, but it doesn't fix the underlying issue: the function has no concept of a paragraph, a heading, or a code block. It only knows how to count characters.

Chunking that respects document structure

The fix wasn't tuning the chunk size or the overlap, it was changing the cutting criterion itself: split first along the document's structure -headings, paragraphs, code blocks- and only subdivide by size when an individual structural section exceeds the embedding's token budget.

function chunkByStructure(markdown, maxTokens = 500) {
  const sections = markdown.split(/(?=^#{1,3} )/m);
  const chunks = [];

  for (const section of sections) {
    if (estimateTokens(section) <= maxTokens) {
      chunks.push(section.trim());
      continue;
    }

    const paragraphs = section.split(/\n\n+/);
    let current = '';

    for (const paragraph of paragraphs) {
      if (estimateTokens(current + paragraph) > maxTokens && current) {
        chunks.push(current.trim());
        current = '';
      }
      current += paragraph + '\n\n';
    }
    if (current.trim()) chunks.push(current.trim());
  }

  return chunks.filter(c => c.length > 0);
}

The difference from the naive version is that this function never cuts inside a paragraph and never separates a heading from the start of its own section: the first level of splitting respects the document's natural boundaries, and only when an entire section is too large for one embedding does it subdivide further, and even then it does so by whole paragraph, not by character count. A code block wrapped in triple backticks stays inside a single paragraph for the purposes of this split, so it doesn't get cut in half either, unless the block alone exceeds the token budget by itself, a case worth handling separately by keeping the block as its own fragment instead of forcing it under the limit.

What didn't work: raising the overlap to compensate for bad cuts

Before changing the cutting criterion, we tried fixing the problem a different way: keep the character-based chunking, but raise the overlap from a hundred characters to several hundred, on the idea that a bigger overlap would lower the odds of a complete idea getting split across two fragments. It didn't work, and it introduced a new problem on top. With more overlap, consecutive fragments ended up sharing most of their content, so their embeddings landed very close to each other in vector space. A search that used to return five fragments with distinct information started returning several near-identical copies of the same passage instead, because all of them fell within the question's similarity radius at once. The query's limit 5 filled up with variations of the same fragment rather than covering the five most relevant parts of the document for that question, and the context handed to the generating model ended up less diverse, not more complete. The actual fix wasn't in the amount of overlap, it was in no longer cutting the text without looking at its structure.

When it's worth investing in this

Character-based chunking with no structure is fine for short, flat documents, like a FAQ list where each entry is already self-contained and smaller than the token limit. It stops being fine as soon as documents have real hierarchy -manuals, technical documentation, specifications- because there the unit of meaning isn't a fixed number of characters, it's the section, paragraph, or code block as the author defined it. The question we use to decide how much to invest in chunking is simple: if you showed someone an isolated fragment, with none of the surrounding document, would it make complete sense on its own? If the answer is consistently no, the problem isn't the embedding model or the pgvector index, it's the cut that was made before either of them ever got involved.