# Choosing an Embedding Model: Dimensions, Cost and Chunk Size

> How to pick an OpenAI embedding model, set the dimensions parameter and choose chunk size and overlap for a search or RAG app, with Azure AI Search settings.

Source: https://www.epifive.com/blog/embedding-model-dimensions-cost-chunking · Published 2026-10-26 · EpiFive

## Key takeaways

- text-embedding-3-small has 1536 default dimensions and text-embedding-3-large has 3072, and both accept up to 8192 input tokens.
- The dimensions parameter shortens an embedding, and OpenAI states that text-embedding-3-large at 256 dimensions still outperforms ada-002 at 1536 on MTEB.
- Start chunking at 512 tokens with 25 percent overlap, then measure retrieval on real questions before changing it.
- Embedding cost is billed on input tokens, so overlap raises the token count you pay to embed.
- In Azure AI Search, vector dimensions, stored, quantization and rescoreStorageMethod all affect storage and are mostly irreversible after index creation.

The practical default for a new search or retrieval-augmented generation (RAG) app is OpenAI's text-embedding-3-small at its default 1536 dimensions, with documents split into chunks of about 512 tokens and about 25 percent overlap. Move to text-embedding-3-large, fewer dimensions or different chunk settings only when a test on your own questions shows a measurable reason to.

## Who this is for and what you will have

This guide is for developers building semantic search or RAG over their own documents who need to make three linked decisions: which embedding model to use, how many dimensions to store, and how to cut documents into chunks. By the end you will have a short decision procedure, a Python snippet that creates embeddings with a chosen dimension count, chunking settings for Azure AI Search or a library such as LangChain, and a list of storage settings that change your index size.

## What an embedding model decides for you

An embedding model turns text into a fixed-length list of numbers. Similar meanings produce vectors that are close together, and search compares the query vector with stored chunk vectors. Three choices follow from the model:

- **Quality.** Larger models generally retrieve better, but only your own data can tell you whether the difference matters.
- **Vector size.** The number of dimensions drives index storage and memory, and affects how fast comparisons run.
- **Input limit.** The model can only embed so many tokens at once, which forces you to chunk long documents.

Changing the model later means re-embedding every document, because vectors from different models are not comparable. Decide early, and keep the model name stored next to your index so queries use the same one.

## The OpenAI embedding models compared

OpenAI's embeddings guide lists three models. The figures below come from that page.

| Model | Default dimensions | Max input tokens | MTEB score | Approx. pages per US dollar |
|---|---|---|---|---|
| text-embedding-3-small | 1536 | 8192 | 62.3% | 62,500 |
| text-embedding-3-large | 3072 | 8192 | 64.6% | 9,615 |
| text-embedding-ada-002 | 1536 | 8192 | 61.0% | 12,500 |

The pages-per-dollar column assumes about 800 tokens per page, and the guide labels ada-002 as a legacy option. MTEB is a public benchmark of embedding quality across many tasks, so a gap of a couple of points does not tell you how the models behave on your documents.

For cost, the embeddings guide does not publish a per-token price; it points to OpenAI's pricing page, which lists embeddings per million tokens. Check that page on the day you plan the project. The pages-per-dollar column gives you the ratio, though: text-embedding-3-small embeds roughly six or seven times as many pages per dollar as text-embedding-3-large.

### Azure OpenAI

If you deploy the same models in Azure OpenAI, you call them through a deployment name instead of a model name, and prices and quotas come from Azure's own pricing page and your deployment settings. The Azure AI Search documentation cites the 8,191-token input limit for text-embedding-3-small, so keep each chunk well under 8,000 tokens in practice. For pricing and rate-limit behaviour, see [Azure OpenAI pricing: PTU vs pay-as-you-go](https://www.epifive.com/blog/azure-openai-pricing-ptu-vs-pay-as-you-go) and [Azure OpenAI 429 rate limit fix](https://www.epifive.com/blog/azure-openai-429-rate-limit-fix).

## Choosing the number of dimensions

The two newer models accept a `dimensions` parameter that returns a shorter vector. According to the OpenAI guide, shortening removes values from the end of the vector, and the model is trained so that the shortened result still carries useful meaning. The guide gives two concrete examples:

- text-embedding-3-large shortened to 256 dimensions still outperforms ada-002 at its full 1536 dimensions on MTEB.
- If your vector store only supports 1024 dimensions, set `dimensions` to 1024.

Fewer dimensions mean smaller vectors, so the index is cheaper to store and quicker to search. The trade is some retrieval quality, and how much depends on your content.

```python
from openai import OpenAI

client = OpenAI()

resp = client.embeddings.create(
    model="text-embedding-3-large",
    input=["How do I reset a user's password?"],
    dimensions=1024,
)

vector = resp.data[0].embedding
print(len(vector))  # 1024
```

### Normalization

OpenAI embeddings are normalized to length 1. That means cosine similarity can be computed as a dot product, and cosine similarity and Euclidean distance produce identical rankings. If you shorten a vector yourself instead of passing `dimensions`, re-normalize it to unit length (L2) afterwards, otherwise similarity scores are distorted. The guide recommends passing `dimensions` at creation time, which avoids the problem.

### A simple decision rule

1. Start with text-embedding-3-small at 1536 dimensions.
2. If retrieval misses on your test questions, try text-embedding-3-large at its default or at a reduced size such as 1024.
3. If storage or latency is the constraint, reduce dimensions on whichever model you chose and re-test.
4. Keep the model name and dimension count as configuration, not constants spread through code.

## Chunking: why and how

Chunking is strictly required only when a document is larger than the model input limit. The Azure AI Search chunking guide adds a second reason: it also helps when content is poorly represented as a single vector. A wiki page covering several unrelated sub-topics might fit in 8,192 tokens, but one vector for the whole page blurs the topics together.

### Common techniques

The Microsoft documentation lists these approaches:

| Approach | What it does | When to use |
|---|---|---|
| Fixed-size chunks | Splits at a set size with some overlap | Simple default for prose |
| Variable-size chunks | Splits on sentences, line ends or Markdown and HTML headings | Documents with clear structure |
| Semantic chunks | Groups sentences into meaningful units that can span pages | Content where topics do not follow page breaks |
| Custom combinations | Mixes the above, for example by prefixing the document title to chunks from the middle of a long document | When chunks lose context without the title |

### Size and overlap defaults

Microsoft recommends starting with a chunk size of 512 tokens, about 2,000 characters, and an initial overlap of 25 percent, which is 128 tokens. The optimal overlap varies: highly structured data may need less, while conversational or narrative text may benefit from more.

The Azure AI Search Text Split skill example recommends a different starting point when measuring in characters: `maximumPageLength` of 2000 and `pageOverlapLength` of 500. The two recommendations describe about the same thing, since 2,000 characters is roughly 512 tokens, and 500 characters is 25 percent of 2,000.

Overlap has a cost. Each overlapping token is embedded twice, and embedding is billed on input tokens, so 25 percent overlap raises the number of tokens you embed by roughly a third compared with no overlap, and also creates more vectors to store. Microsoft's own chunk-count table for one sample e-book shows the effect: at a maximum page length of 1000 characters, adding 200 characters of overlap raised the chunk count from 172 to 216.

### Factors to weigh

The Microsoft guidance lists three factors:

- The shape and density of your documents. If passages need to stay intact, use larger chunks and sentence-aware splitting.
- The queries users ask. Larger chunks and overlap preserve context for questions about specific details.
- The chat model you use for answers. Pick a chunk size that works for every model in the pipeline, for example both the embedding model and the summarization model.

Measure your real token counts before settling on numbers. The Microsoft example counts tokens with `tiktoken`, finds an average of 189 tokens per page in its sample PDF, and concludes that a smaller chunk size makes sense there. It also warns that an overlap value that is too large relative to chunk size can result in no overlap appearing at all, because the Text Split skill requires `pageOverlapLength` to be less than half of `maximumPageLength`.

## Setting it up in Azure AI Search

If you use Azure AI Search with integrated vectorization, the Text Split skill performs the chunking. The relevant parameters are:

- `textSplitMode`: `pages` (the default, chunks of multiple sentences) or `sentences`.
- `maximumPageLength`: the maximum characters (or tokens in the preview token mode) per chunk. The splitter avoids breaking sentences, so actual sizes vary.
- `pageOverlapLength`: characters carried from the end of one chunk to the start of the next. It must be less than half of `maximumPageLength`.
- `maximumPagesToTake`: how many chunks to take from a document. The default of 0 takes all of them.

A skill definition using the recommended starting values looks like this:

```json
{
  "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
  "name": "split-text",
  "context": "/document",
  "textSplitMode": "pages",
  "maximumPageLength": 2000,
  "pageOverlapLength": 500,
  "inputs": [
    { "name": "text", "source": "/document/content" }
  ],
  "outputs": [
    { "name": "textItems", "targetName": "pages" }
  ]
}
```

For a full pipeline from Blob Storage, see [Azure AI Search integrated vectorization from Blob Storage](https://www.epifive.com/blog/azure-ai-search-integrated-vectorization-blob). If you are still choosing a vector store, [pgvector vs Azure AI Search vs Qdrant](https://www.epifive.com/blog/pgvector-vs-azure-ai-search-vs-qdrant) compares the options.

### Match the index to the embedding

The `dimensions` value on the vector field in the index must equal the length of the vectors you send. If you create embeddings with `dimensions=1024`, the index field needs `"dimensions": 1024`, or indexing fails.

## Storage settings that change cost

Embedding size also shapes what you pay to keep vectors. Azure AI Search stores more than one copy of a vector field, and the documentation describes how to drop optional copies:

| Setting | Effect | Caveat |
|---|---|---|
| `"stored": false` on a vector field | Removes the retrievable copy of the source vector; documented as saving up to 50% storage per field | Set only at index creation and irreversible; `retrievable` must then be false |
| `rescoreStorageMethod: discardOriginals` in `vectorSearch.compressions` | Drops the full-precision originals used for rescoring | Irreversible and can reduce search quality; the default is `preserveOriginals` |
| Scalar or binary quantization | Compresses vectors in the index | Rescoring of scalar-quantized vectors requires keeping the originals |

Two warnings from the documentation are worth repeating. With `stored` set to false, a partial update using `merge` or `mergeOrUpload` that omits the vector field loses the existing vector without an error, so every update must include the whole vector field. And these properties are fixed when the index or field is created, so decide before loading data. The `stored` setting affects storage on disk, not memory, and does not change query execution.

## Verifying your choices

Do not rely on benchmark scores alone. Build a small evaluation set and run it each time you change a setting.

1. Collect 20 to 50 real questions and note which document or section should answer each.
2. Index your documents with the candidate configuration.
3. For each question, check whether the correct chunk appears in the top 3 and top 5 results.
4. Record the result next to the model, dimensions, chunk size and overlap used.
5. Change one setting at a time. If you change the model and the chunk size together, you will not know which one caused the difference.

Also check token counts on a sample of chunks, using the same tokenizer family as your model, so that no chunk is near the 8,192-token limit.

## Troubleshooting

### Indexing fails with a dimension mismatch

The vector field's `dimensions` does not match the embedding length. Check the `dimensions` argument in your embedding call and in the index definition, and remember that an existing vector field's dimension count cannot simply be edited in place; create a new field or index.

### Results get worse after shortening vectors

If you truncated vectors yourself, confirm you re-normalized them to unit length. If you used the `dimensions` parameter, try a higher value, or switch to the large model at a reduced size, and re-run your evaluation set.

### Answers lack context from the previous paragraph

Chunks may be too small or have too little overlap. Raise the overlap toward 25 percent or prefix the document title or heading to each chunk, which is the custom combination approach in the Microsoft guidance.

### Search returns near-duplicate chunks

Overlap that is too high produces repeated text across neighbouring chunks. Lower the overlap, or deduplicate results by source document before sending them to the chat model.

### Embedding requests are rejected for length

An input over the model limit is a chunking problem. Count tokens before you call the API and split anything that is too long.

## Checklist

- Start with text-embedding-3-small at 1536 dimensions and keep the model name as configuration.
- Pass `dimensions` when creating embeddings rather than truncating vectors afterwards.
- Chunk at about 512 tokens (2,000 characters) with about 25 percent overlap as a starting point.
- Match the index field dimensions to the embedding length.
- Decide on `stored`, quantization and `rescoreStorageMethod` before loading data, because they are mostly irreversible.
- Build a small evaluation set and change one setting at a time.
- Check the current pricing page before estimating cost, since the embeddings guide does not list per-token prices.

## References

- OpenAI embeddings guide: https://developers.openai.com/api/docs/guides/embeddings
- OpenAI API pricing: https://developers.openai.com/api/docs/pricing
- Chunk documents (Azure AI Search): https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents
- Eliminate optional vector instances (Azure AI Search): https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-storage-options
- Text Split skill: https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-textsplit

## Frequently asked questions

### Which OpenAI embedding model should I start with?

Start with text-embedding-3-small. It has 1536 default dimensions, the lowest cost of the current models and an 8192-token input limit. Move to text-embedding-3-large only if an evaluation on your own queries shows that retrieval quality is not good enough.

### Can I shorten an embedding without re-embedding my documents?

OpenAI documents that shortening removes values from the end of the vector, and that a manually shortened vector must be re-normalized to unit length. The simplest and recommended approach is to pass the dimensions parameter when you create the embedding, so stored vectors and query vectors always match.

### What chunk size should I use for RAG?

Microsoft recommends starting with 512 tokens, roughly 2,000 characters, with 25 percent overlap, which is 128 tokens. The Azure AI Search Text Split skill example uses 2000 characters with 500 characters of overlap. Treat either as a starting point and compare retrieval results on real questions.

### Do I need to chunk documents that fit in the model input limit?

Not always. Chunking is required only when a document exceeds the model input limit, but it also helps when one vector represents several unrelated topics poorly. A wiki page with many sub-topics usually retrieves better when split at a finer grain.