Choosing an Embedding Model: Dimensions, Cost and Chunk Size

The practical default for a new search or retrieval-augmented generation (RAG) app is OpenAI's text-embedding-3-small at its default 1536 dimensions, with documents split into chunks of about 512 tokens and about 25 percent overlap. Move to text-embedding-3-large, fewer dimensions or different chunk settings only when a test on your own questions shows a measurable reason to.

Who this is for and what you will have

This guide is for developers building semantic search or RAG over their own documents who need to make three linked decisions: which embedding model to use, how many dimensions to store, and how to cut documents into chunks. By the end you will have a short decision procedure, a Python snippet that creates embeddings with a chosen dimension count, chunking settings for Azure AI Search or a library such as LangChain, and a list of storage settings that change your index size.

What an embedding model decides for you

An embedding model turns text into a fixed-length list of numbers. Similar meanings produce vectors that are close together, and search compares the query vector with stored chunk vectors. Three choices follow from the model:

• Quality. Larger models generally retrieve better, but only your own data can tell you whether the difference matters.
• Vector size. The number of dimensions drives index storage and memory, and affects how fast comparisons run.
• Input limit. The model can only embed so many tokens at once, which forces you to chunk long documents.

Changing the model later means re-embedding every document, because vectors from different models are not comparable. Decide early, and keep the model name stored next to your index so queries use the same one.

The OpenAI embedding models compared

OpenAI's embeddings guide lists three models. The figures below come from that page.

The pages-per-dollar column assumes about 800 tokens per page, and the guide labels ada-002 as a legacy option. MTEB is a public benchmark of embedding quality across many tasks, so a gap of a couple of points does not tell you how the models behave on your documents.

For cost, the embeddings guide does not publish a per-token price; it points to OpenAI's pricing page, which lists embeddings per million tokens. Check that page on the day you plan the project. The pages-per-dollar column gives you the ratio, though: text-embedding-3-small embeds roughly six or seven times as many pages per dollar as text-embedding-3-large.

Azure OpenAI

If you deploy the same models in Azure OpenAI, you call them through a deployment name instead of a model name, and prices and quotas come from Azure's own pricing page and your deployment settings. The Azure AI Search documentation cites the 8,191-token input limit for text-embedding-3-small, so keep each chunk well under 8,000 tokens in practice. For pricing and rate-limit behaviour, see Azure OpenAI pricing: PTU vs pay-as-you-go and Azure OpenAI 429 rate limit fix.

Choosing the number of dimensions

The two newer models accept a dimensions parameter that returns a shorter vector. According to the OpenAI guide, shortening removes values from the end of the vector, and the model is trained so that the shortened result still carries useful meaning. The guide gives two concrete examples:

• text-embedding-3-large shortened to 256 dimensions still outperforms ada-002 at its full 1536 dimensions on MTEB.
• If your vector store only supports 1024 dimensions, set dimensions to 1024.

Fewer dimensions mean smaller vectors, so the index is cheaper to store and quicker to search. The trade is some retrieval quality, and how much depends on your content.

from openai import OpenAI
 
client = OpenAI()
 
resp = client.embeddings.create(
    model="text-embedding-3-large",
    input=["How do I reset a user's password?"],
    dimensions=1024,
)
 
vector = resp.data[0].embedding
print(len(vector))  # 1024

Normalization

OpenAI embeddings are normalized to length 1. That means cosine similarity can be computed as a dot product, and cosine similarity and Euclidean distance produce identical rankings. If you shorten a vector yourself instead of passing dimensions, re-normalize it to unit length (L2) afterwards, otherwise similarity scores are distorted. The guide recommends passing dimensions at creation time, which avoids the problem.

A simple decision rule

1. Start with text-embedding-3-small at 1536 dimensions.
2. If retrieval misses on your test questions, try text-embedding-3-large at its default or at a reduced size such as 1024.
3. If storage or latency is the constraint, reduce dimensions on whichever model you chose and re-test.
4. Keep the model name and dimension count as configuration, not constants spread through code.

Chunking: why and how

Chunking is strictly required only when a document is larger than the model input limit. The Azure AI Search chunking guide adds a second reason: it also helps when content is poorly represented as a single vector. A wiki page covering several unrelated sub-topics might fit in 8,192 tokens, but one vector for the whole page blurs the topics together.

Common techniques

The Microsoft documentation lists these approaches:

Size and overlap defaults

Microsoft recommends starting with a chunk size of 512 tokens, about 2,000 characters, and an initial overlap of 25 percent, which is 128 tokens. The optimal overlap varies: highly structured data may need less, while conversational or narrative text may benefit from more.

The Azure AI Search Text Split skill example recommends a different starting point when measuring in characters: maximumPageLength of 2000 and pageOverlapLength of 500. The two recommendations describe about the same thing, since 2,000 characters is roughly 512 tokens, and 500 characters is 25 percent of 2,000.

Overlap has a cost. Each overlapping token is embedded twice, and embedding is billed on input tokens, so 25 percent overlap raises the number of tokens you embed by roughly a third compared with no overlap, and also creates more vectors to store. Microsoft's own chunk-count table for one sample e-book shows the effect: at a maximum page length of 1000 characters, adding 200 characters of overlap raised the chunk count from 172 to 216.

Factors to weigh

The Microsoft guidance lists three factors:

• The shape and density of your documents. If passages need to stay intact, use larger chunks and sentence-aware splitting.
• The queries users ask. Larger chunks and overlap preserve context for questions about specific details.
• The chat model you use for answers. Pick a chunk size that works for every model in the pipeline, for example both the embedding model and the summarization model.

Measure your real token counts before settling on numbers. The Microsoft example counts tokens with tiktoken, finds an average of 189 tokens per page in its sample PDF, and concludes that a smaller chunk size makes sense there. It also warns that an overlap value that is too large relative to chunk size can result in no overlap appearing at all, because the Text Split skill requires pageOverlapLength to be less than half of maximumPageLength.

Setting it up in Azure AI Search

If you use Azure AI Search with integrated vectorization, the Text Split skill performs the chunking. The relevant parameters are:

• textSplitMode: pages (the default, chunks of multiple sentences) or sentences.
• maximumPageLength: the maximum characters (or tokens in the preview token mode) per chunk. The splitter avoids breaking sentences, so actual sizes vary.
• pageOverlapLength: characters carried from the end of one chunk to the start of the next. It must be less than half of maximumPageLength.
• maximumPagesToTake: how many chunks to take from a document. The default of 0 takes all of them.

A skill definition using the recommended starting values looks like this:

{
  "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
  "name": "split-text",
  "context": "/document",
  "textSplitMode": "pages",
  "maximumPageLength": 2000,
  "pageOverlapLength": 500,
  "inputs": [
    { "name": "text", "source": "/document/content" }
  ],
  "outputs": [
    { "name": "textItems", "targetName": "pages" }
  ]
}

For a full pipeline from Blob Storage, see Azure AI Search integrated vectorization from Blob Storage. If you are still choosing a vector store, pgvector vs Azure AI Search vs Qdrant compares the options.

Match the index to the embedding

The dimensions value on the vector field in the index must equal the length of the vectors you send. If you create embeddings with dimensions=1024, the index field needs "dimensions": 1024, or indexing fails.

Storage settings that change cost

Embedding size also shapes what you pay to keep vectors. Azure AI Search stores more than one copy of a vector field, and the documentation describes how to drop optional copies:

Two warnings from the documentation are worth repeating. With stored set to false, a partial update using merge or mergeOrUpload that omits the vector field loses the existing vector without an error, so every update must include the whole vector field. And these properties are fixed when the index or field is created, so decide before loading data. The stored setting affects storage on disk, not memory, and does not change query execution.

Verifying your choices

Do not rely on benchmark scores alone. Build a small evaluation set and run it each time you change a setting.

1. Collect 20 to 50 real questions and note which document or section should answer each.
2. Index your documents with the candidate configuration.
3. For each question, check whether the correct chunk appears in the top 3 and top 5 results.
4. Record the result next to the model, dimensions, chunk size and overlap used.
5. Change one setting at a time. If you change the model and the chunk size together, you will not know which one caused the difference.

Also check token counts on a sample of chunks, using the same tokenizer family as your model, so that no chunk is near the 8,192-token limit.

Troubleshooting

Indexing fails with a dimension mismatch

The vector field's dimensions does not match the embedding length. Check the dimensions argument in your embedding call and in the index definition, and remember that an existing vector field's dimension count cannot simply be edited in place; create a new field or index.

Results get worse after shortening vectors

If you truncated vectors yourself, confirm you re-normalized them to unit length. If you used the dimensions parameter, try a higher value, or switch to the large model at a reduced size, and re-run your evaluation set.

Answers lack context from the previous paragraph

Chunks may be too small or have too little overlap. Raise the overlap toward 25 percent or prefix the document title or heading to each chunk, which is the custom combination approach in the Microsoft guidance.

Search returns near-duplicate chunks

Overlap that is too high produces repeated text across neighbouring chunks. Lower the overlap, or deduplicate results by source document before sending them to the chat model.

Embedding requests are rejected for length

An input over the model limit is a chunking problem. Count tokens before you call the API and split anything that is too long.

Checklist

• Start with text-embedding-3-small at 1536 dimensions and keep the model name as configuration.
• Pass dimensions when creating embeddings rather than truncating vectors afterwards.
• Chunk at about 512 tokens (2,000 characters) with about 25 percent overlap as a starting point.
• Match the index field dimensions to the embedding length.
• Decide on stored, quantization and rescoreStorageMethod before loading data, because they are mostly irreversible.
• Build a small evaluation set and change one setting at a time.
• Check the current pricing page before estimating cost, since the embeddings guide does not list per-token prices.

References

• OpenAI embeddings guide: https://developers.openai.com/api/docs/guides/embeddings
• OpenAI API pricing: https://developers.openai.com/api/docs/pricing
• Chunk documents (Azure AI Search): https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents
• Eliminate optional vector instances (Azure AI Search): https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-storage-options
• Text Split skill: https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-textsplit