Embedding dimensions and the storage they cost

If you’ve built a RAG pipeline, you’ve picked an embedding model, glanced at its dimension count, and moved on without thinking about it again. That number, 384, 768, 1536, 3072, is a direct multiplier on how much disk and RAM your vector database eats, and it shows up on your infrastructure bill whether you notice it or not.

What a dimension actually is

An embedding model turns a chunk of text into a list of floating point numbers, a vector. Each number in that list is one dimension. A 768-dimension embedding is a list of 768 floats that together describe where that chunk sits in a high-dimensional semantic space, where “close” vectors mean “similar meaning” according to whatever the model learned during training.

More dimensions mean more room to encode distinctions. A model with 1536 dimensions can, in theory, separate finer shades of meaning than one with 384. That’s why bigger embedding models tend to score better on retrieval benchmarks. But “in theory” is doing a lot of work in that sentence, and the gains are not linear. Going from 384 to 768 often buys you a real, noticeable jump in retrieval quality. Going from 1536 to 3072 usually buys you a lot less, because most of the useful semantic signal is already captured well before you hit that range. You’re paying full price for the last stretch of diminishing returns.

The storage math, worked out

This part is just arithmetic, not a benchmark claim, so let’s do it.

A standard float32 number takes 4 bytes. A 1536-dimension embedding, stored as float32, is 1536 x 4 bytes = 6,144 bytes, just over 6KB, per vector. Double the dimensions to 3072 and you double the raw storage to just over 12KB per vector.

Now scale that to a real corpus. Say your RAG pipeline has chunked your documents into 10 million pieces, which is a completely normal size for a company knowledge base plus a few years of support tickets and product docs. At 1536 dimensions, that’s roughly 60GB of raw vector data. At 3072 dimensions, you’re at roughly 120GB. That’s before you’ve stored a single word of the original text, before metadata like source URLs or timestamps, and before whatever index structure your vector database builds on top to make search fast.

That last part matters. Most production vector databases don’t do a flat linear scan over every vector. They build a graph or tree structure, commonly HNSW, so a query doesn’t have to touch every row. Building that structure means storing extra data on top of the raw vectors: links between neighboring nodes, layer assignments, sometimes duplicate copies of vectors at different graph levels. The exact overhead depends on the index parameters you choose and the database you’re running, so I won’t hand you a made-up multiplier. What’s safe to say is that the number you get from dimensions times bytes is a floor, not the actual bill.

Cutting dimensions without gutting quality

You don’t have to accept the default dimension count a model ships with. Two mechanisms let you shrink the vector without starting over.

The first is a model trained with Matryoshka representation learning. OpenAI’s text-embedding-3-small and text-embedding-3-large both support this: you can request a shorter vector, say 256 dimensions out of a 1536-dimension model, by truncating the output and the model was specifically trained so that the early dimensions carry most of the useful signal. This is not the same as truncating an ordinary embedding, which just throws away information at random. A Matryoshka-trained model front-loads the important structure into the first N dimensions on purpose, so truncation degrades gracefully instead of falling apart.

The second is picking a smaller model outright. Open source options like BAAI’s bge-small (384 dimensions) or sentence-transformers’ all-MiniLM-L6-v2 (also 384 dimensions) exist specifically because most retrieval tasks don’t need the full expressive range of a 3072-dimension model. If your corpus is a product FAQ or a narrow internal wiki, the semantic space you’re actually searching over is small, and a smaller embedding model can cover it just fine.

Either way, the move is the same: don’t default to the largest embedding your provider offers because it feels safer. Test a smaller one against your own queries and your own documents. Retrieval quality is corpus-dependent. A benchmark score from a model’s release blog post was measured on someone else’s dataset, not yours.

Quantization is the other lever

Dimension count controls the length of the vector. Quantization controls how many bytes each number in that vector takes. You don’t have to store every dimension as a 4-byte float32.

Dropping to float16 halves storage instantly, 6KB becomes 3KB per vector at 1536 dimensions, with a quality hit that’s usually small because embeddings don’t need full float32 precision to preserve their relative distances. Going further to int8 quantization, where each dimension is compressed to a single byte using a scale factor, cuts storage to a quarter of the float32 size. Several vector databases support this natively now, computing distances directly on the quantized representation instead of decompressing first, so you get the storage win without paying it back in query latency.

Quantization and dimension reduction stack. A 3072-dimension float32 vector at 12KB can become a 1024-dimension int8 vector at 1KB, a 12x reduction, if your retrieval quality holds up at that setting for your data. Whether it holds up is an empirical question for your corpus, not something I can promise you in the abstract.

The part that never shows up in the pricing page

Vector storage cost isn’t just disk. Most managed vector databases keep the index in RAM for query speed, and RAM is the expensive resource, not disk. A collection that comfortably fits on a cheap SSD can force you onto a much larger, much pricier instance the moment it needs to live in memory for HNSW traversal to stay fast. This is the real reason dimension count matters more than people expect: it’s a memory tier decision. Doubling your embedding dimensions can be the difference between an instance that costs you noticeably more per month and one that doesn’t, depending on where the size crosses your provider’s tier boundaries. I’m not going to hand you a dollar figure, because pricing varies by vendor and changes over time, but the mechanism is the same everywhere: bigger vectors mean more RAM, and RAM is billed at a premium over disk.

What to actually do

Before you lock in an embedding model for a RAG pipeline, work out the storage math for your actual corpus size at the dimension count you’re considering, the same way we did above. If the number looks fine at 10,000 chunks and terrifying at 10 million, that’s worth knowing before you build, not after your vector database bill triples.

Then test smaller. If your provider supports Matryoshka truncation, try 512 or 768 dimensions against your own retrieval queries before defaulting to the full 1536 or 3072. If it doesn’t, try a smaller model in the same family or from a different provider. Measure recall on questions you actually expect users to ask, not on a public benchmark that has nothing to do with your documents. In a lot of RAG pipelines, the extra dimensions you’re paying to store never get used for anything, because the queries were never going to need that much resolution in the first place.

Storage is cheap per gigabyte in isolation. It stops being cheap once you multiply it by millions of chunks and re-embeddings every time you update a model, and once RAM tiers get involved. Treat dimension count as a cost decision, not just a quality knob, and check it against your own data before you commit.

Want more breakdowns like this on what AI tools actually cost to run in production? Head back to the AI Tool Gazette homepage for the rest of our explainers.