Semantic Search With Azure OpenAI Embeddings in Python: A Minimal Example

To build semantic search with Azure OpenAI in Python, split your text into chunks, send the chunks to an embeddings deployment to get a vector for each, embed the user's query the same way, and rank chunks by cosine similarity to the query. For a small collection you can do all of this with the openai package and NumPy, with no database. This article shows a complete, minimal script and the limits to respect.

Who this is for and what you will have

This is for developers who want to understand embedding-based search before reaching for a vector database or a managed search service. By the end you will have a Python script that embeds a handful of text passages, answers a natural-language query with the closest passages, and handles batching and the documented request limits. It is deliberately small: the aim is to see every moving part. For choosing models, dimensions and chunk sizes in more depth, see /blog/embedding-model-dimensions-cost-chunking.

How embedding search works

Microsoft describes an embedding as a vector of floating-point numbers that represents the semantic meaning of text. Similar text produces vectors that are close together, which is what makes embeddings useful for vector search, recommendations, classification and clustering.

The search flow has two phases:

Index time:  documents -> chunks -> embeddings API -> vectors stored in memory/disk
Query time:  query text -> embeddings API -> query vector -> compare with stored vectors -> top results

Closeness is usually measured with cosine similarity: the cosine of the angle between two vectors. A higher score means more similar meaning. Because you compare the query vector to every stored vector, the approach is exact and simple, and fine for thousands of chunks. Larger systems use approximate nearest-neighbour indexes instead.

Prerequisites

• An Azure subscription and an Azure OpenAI resource with an embedding model deployment. Microsoft's examples use text-embedding-3-small.
• Your resource endpoint, in the form https://YOUR-RESOURCE-NAME.openai.azure.com.
• For Microsoft Entra ID authentication, an identity with the Cognitive Services OpenAI User role on the resource, and the Azure CLI signed in with az login for local development.
• Python and pip.

Microsoft recommends Entra ID authentication for the v1 embeddings API because it avoids storing long-lived credentials. API key authentication is also supported. Install the packages:

pip install openai azure-identity numpy

Remember that the model value in each request is your deployment name. If you called the deployment something other than text-embedding-3-small, use that name.

Step 1: Create the client

This follows Microsoft's documented Python pattern: the OpenAI client pointed at the /openai/v1/ endpoint, with a bearer token provider for the https://ai.azure.com/.default scope.

from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
 
ENDPOINT = "https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/"
DEPLOYMENT = "text-embedding-3-small"  # your deployment name
 
token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(base_url=ENDPOINT, api_key=token_provider)

DefaultAzureCredential uses your signed-in identity locally and can use a managed identity when the code runs in Azure.

Step 2: Prepare and chunk the text

Embedding models accept a maximum of 8,192 tokens per input, and a vector for a whole long document blurs many topics together. Splitting text into smaller passages gives sharper matches. The sample below uses a naive approach: split on blank lines, then combine short paragraphs up to a character budget. It is a teaching sketch, not a tuned splitter.

def chunk_text(text: str, max_chars: int = 800) -> list[str]:
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for p in paragraphs:
        if current and len(current) + len(p) + 2 > max_chars:
            chunks.append(current)
            current = p
        else:
            current = f"{current}\n\n{p}" if current else p
    if current:
        chunks.append(current)
    return chunks
 
 
DOCUMENT = """
Contoso Support Hours
 
Our support desk is open Monday to Friday, 9:00 to 17:00. Requests sent
outside those hours are answered the next working day.
 
Password Resets
 
Users can reset their own password from the sign-in page by choosing
"Forgot password". If self-service reset is not enabled for your account,
contact the IT help desk.
 
Expense Claims
 
Submit expense claims in the finance portal within 30 days of purchase and
attach a receipt for every item.
 
Remote Access
 
Staff working off site connect through the company VPN. Install the client
from the software portal and sign in with your work account.
"""
 
chunks = chunk_text(DOCUMENT, max_chars=300)

The text is invented sample content for illustration. In real use, tune the chunk size to your documents and consider overlap between neighbouring chunks.

Step 3: Embed the chunks in batches

The embeddings endpoint accepts either one string or an array. Microsoft documents these limits: the maximum array size is 2,048 inputs, each input is at most 8,192 tokens, and each request has a 300,000-token aggregate limit across all inputs, above which it fails with HTTP 400. A simple batch size keeps you well inside the array limit.

import numpy as np
 
 
def embed(texts: list[str], batch_size: int = 64) -> np.ndarray:
    vectors = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i : i + batch_size]
        response = client.embeddings.create(model=DEPLOYMENT, input=batch)
        vectors.extend(item.embedding for item in response.data)
    return np.array(vectors, dtype=np.float32)
 
 
doc_vectors = embed(chunks)
print(doc_vectors.shape)  # (number_of_chunks, dimensions)

If you want smaller vectors, the third-generation models support a dimensions parameter, which Microsoft says gives you more control over compute, memory and storage cost. It is not supported in all versions of the OpenAI 1.x Python library, so upgrade with pip install openai --upgrade if you use it. Whatever you choose, use the same value for documents and queries. The model output sizes documented by Microsoft are 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large.

Step 4: Search with cosine similarity

Normalise the vectors once, and cosine similarity becomes a dot product.

def normalise(m: np.ndarray) -> np.ndarray:
    norms = np.linalg.norm(m, axis=-1, keepdims=True)
    return m / np.clip(norms, 1e-12, None)
 
 
doc_matrix = normalise(doc_vectors)
 
 
def search(query: str, top_k: int = 3):
    q = normalise(embed([query]))[0]
    scores = doc_matrix @ q
    best = np.argsort(scores)[::-1][:top_k]
    return [(float(scores[i]), chunks[i]) for i in best]
 
 
for score, text in search("How do I reset my password?"):
    print(f"{score:.3f}  {text[:70]!r}")

You should see the password-reset chunk ranked first, even though the query does not repeat its exact wording. Try a query such as working from home connection and compare it with the remote-access chunk. Scores are relative: use them to rank, and treat any cut-off threshold as something you tune on your own data.

Save the index

Recomputing embeddings on every run costs time and tokens. Persist the vectors and the chunks together:

import json
 
np.save("vectors.npy", doc_vectors)
with open("chunks.json", "w", encoding="utf-8") as f:
    json.dump(chunks, f)

Load both with np.load and json.load on startup. Remember that the stored vectors are only valid for the model and dimensions that created them. If you change either, rebuild the index.

Verification

• doc_vectors.shape[0] equals the number of chunks, and shape[1] equals the expected dimensions for your model and settings.
• A query that paraphrases a chunk returns that chunk in the top results.
• A query unrelated to the content produces visibly lower scores than a relevant one.

Troubleshooting

Microsoft's embeddings documentation lists these status codes:

Other issues to watch for:

• Scores look random. You may have embedded documents and queries with different models or dimensions.
• Results ignore a clearly relevant passage. The chunk may be too large or too small. Adjust the chunk size and rebuild.
• Rate limit responses. Keep total tokens per minute below the quota assigned to your deployment; Microsoft points to the Azure OpenAI quotas and limits page for current figures. Add retry with backoff for production use.

Limits of this approach

Embedding models might be unreliable or pose social risks in certain cases, and Microsoft points readers to its Responsible AI content. Brute-force comparison scales linearly with the number of chunks, so a large or frequently updated corpus calls for a vector index or a managed service such as Azure AI Search. The example also ignores access control: if documents have different permissions, filter results by permission before returning them.

Where to go next

Once the script works, the pieces map directly onto larger designs. The chunk_text function becomes your ingestion pipeline, the embed function is called both at ingestion and at query time, and the NumPy matrix is replaced by a vector index. Keep the interface of search the same, so that the rest of your application does not care where the vectors live. A common next step is to pass the top chunks to a chat model as context so that it can answer in plain language, which is the retrieval-augmented generation pattern.

Checklist

• Deployment name matches the model value in your code.
• Endpoint ends in /openai/v1/.
• Entra ID role assigned, or an API key stored outside the source code.
• Documents and queries use the same model and dimensions.
• Batches stay under 2,048 inputs and 300,000 total tokens.
• Vectors and chunks are saved together and rebuilt when the model changes.

References

• Generate embeddings with Azure OpenAI
• Foundry Models sold by Azure (embedding models)
• Azure OpenAI quotas and limits