Queue-Based Load Leveling Pattern with Azure Service Bus

Queue-based load leveling puts a durable queue between the code that receives work and the service that performs it. Requests arrive at whatever rate users produce them, the queue absorbs the burst, and the consumer drains it at a steady rate the back end can sustain. On Azure the usual building blocks are an Azure Service Bus queue and an Azure Functions consumer with bounded concurrency.

Who this is for

This is for developers and architects whose API or web app writes to a database or calls a partner service that times out or throttles during traffic spikes. By the end you will have a design with a producer, a Service Bus queue, a bounded consumer, a dead-letter strategy and the metrics to check that leveling works.

What the pattern solves

Microsoft's description of the pattern starts from a common problem. Many cloud solutions run tasks that call services, and intermittent heavy loads can cause performance or reliability problems for a service. When several tasks run concurrently against the same service, it is hard to predict request volume. A demand peak can overload the service and make it respond slowly, and flooding it with concurrent requests can make it fail from contention.

The solution is to place a queue between the task and the service. The task posts a message containing the data the service needs, the queue stores it until the service retrieves it, and the two run asynchronously. According to Microsoft, this provides three benefits:

• Availability. Service delays don't immediately affect the application, which can keep posting messages even when the service is unavailable or not processing.
• Scalability. The number of queues and the number of consumers can vary with demand.
• Cost control. You need enough service instances for the average load rather than the peak load.

When it does not fit

Microsoft lists two cases where the pattern might not be suitable: when the caller requires a low-latency, synchronous response, and when volume is predictably low and stable so that queueing adds complexity for little benefit. Message queues are one-way. If a task expects a reply, you need a separate mechanism for the service to send one.

Reference design

  Web app / API instances            Azure Service Bus            Azure Functions               Back end
  (many, bursty)                     queue "orders"               consumer (bounded)            (database or partner API)
 
  instance 1 --\
  instance 2 ---+--> send message ---> [ m m m m m m m ] ---> maxConcurrentCalls = N ---> steady write rate
  instance 3 --/                       durable buffer           scale limit = M
                                              |
                                              +--> dead-letter queue (poison messages)

Microsoft's own example uses this shape: an Azure Functions app reads messages from a Service Bus queue and performs the read and write requests against the data store. It adds that Functions can scale instances based on Service Bus backlog using target-based scaling, within your configured scaling bounds, and that you can tune trigger concurrency settings to protect the data store. Without that tuning, the worker layer can reintroduce back-end contention.

Prerequisites

• An Azure subscription and a Service Bus namespace. The Standard or Premium tier is needed if you want duplicate detection, because the Basic tier doesn't support it.
• An Azure Functions app using the Service Bus trigger.
• A downstream resource whose safe request rate you can estimate by load testing.

Step 1: Create the queue

Create a queue in the Azure portal, Azure CLI, PowerShell or an ARM template. A minimal Azure CLI sketch looks like this; replace the names with your own.

az servicebus namespace create \
  --resource-group rg-contoso-orders \
  --name sb-contoso-orders \
  --location westeurope \
  --sku Standard
 
az servicebus queue create \
  --resource-group rg-contoso-orders \
  --namespace-name sb-contoso-orders \
  --name orders

Service Bus queues offer First In, First Out delivery to competing consumers: each message is processed by only one receiver. Receivers typically get messages in the order they were added, but Microsoft cautions that adding a queue does not preserve the original order in all conditions, particularly when multiple consumers process in parallel. If you need strict ordering, use message sessions; otherwise design consumers to accept any order, which makes scaling simpler.

Step 2: Send messages from the producer

The producer should do the minimum: validate the request, send a message, and return an acknowledgement. Use the Azure.Messaging.ServiceBus library with passwordless authentication, as the Service Bus documentation recommends.

using Azure.Identity;
using Azure.Messaging.ServiceBus;
 
await using var client = new ServiceBusClient(
    "sb-contoso-orders.servicebus.windows.net",
    new DefaultAzureCredential());
 
ServiceBusSender sender = client.CreateSender("orders");
 
var message = new ServiceBusMessage(BinaryData.FromString(orderJson))
{
    MessageId = $"order-{orderId}",   // stable, business-derived identifier
    ContentType = "application/json"
};
 
await sender.SendMessageAsync(message);

Setting MessageId from the business identifier, instead of a random value, is what makes duplicate detection and idempotent processing practical later. Return 202 Accepted with a link the client can use to check progress, because the work has not happened yet.

Step 3: Consume with bounded concurrency

Create a Service Bus triggered function. The host.json concurrency settings for the Service Bus extension control how hard each instance works, and they double as the input to scaling.

{
  "version": "2.0",
  "extensions": {
    "serviceBus": {
      "maxConcurrentCalls": 8
    }
  }
}

In single-dispatch mode, where each invocation processes one message, maxConcurrentCalls is the setting target-based scaling uses as its target executions per instance. The Azure Functions documentation lists a default of 16 for the v5.x extension. Target-based scaling computes desired instances as the event source length divided by that target, and it can add up to four instances at a time.

Limit the aggregate rate

Per-instance concurrency is only half of the equation. Total downstream load is roughly instances multiplied by maxConcurrentCalls, so you must also cap the number of instances. Microsoft's pattern article is direct about this: autoscaling without bounding the consumers' aggregate downstream rate only moves the overload to downstream dependencies and reduces the effectiveness of the queue. Limit scale-out for the Functions app and choose a combination that your load tests show the back end can survive.

A few related points from the Functions documentation:

• Target-based scaling is on by default for Consumption, Flex Consumption and Elastic Premium plans, and scale limits are still honoured.
• For the most accurate scaling, use one target-based triggered function per function app.
• With the Service Bus extension, Manage rights on the resource give the most efficient scaling. With only Listen rights, scaling falls back to incremental because queue length is not available.

Step 4: Make the handler idempotent

Most queue services deliver at least once. In Service Bus peek-lock mode, if the application crashes after processing but before completing the message, the service redelivers it. Your handler therefore has to produce the same outcome when it sees the same message twice, for example by using the business identifier as a unique key in the target table.

INSERT INTO orders (order_id, payload, received_at)
VALUES (@orderId, @payload, SYSUTCDATETIME())
-- order_id is the primary key; a duplicate insert fails and is treated as already processed

Service Bus can also discard duplicates at send time. When duplicate detection is enabled on a queue, Service Bus tracks the application-controlled MessageId of messages sent within a time window, and a later message with the same MessageId is reported as accepted but dropped. The window defaults to 10 minutes, with a minimum of 20 seconds and a maximum of 7 days. Keep the window as small as practical, because larger windows affect throughput. Duplicate detection protects against sender-side retries only, so keep the idempotent handler as well. For a wider treatment of retries and duplicates, see Webhooks that don't double-charge.

Step 5: Plan for poison messages

Some messages will never succeed because they are malformed, reference missing data or hit a persistent error. Service Bus gives each queue a dead-letter sub-queue that you don't create or delete separately. Messages move there when:

• The delivery count exceeds the maximum delivery count, which defaults to 10 (reason MaxDeliveryCountExceeded).
• The message expires while dead-lettering on expiration is enabled (reason TTLExpiredException).
• The application dead-letters it explicitly.

Nothing cleans the dead-letter queue automatically. Messages stay until you retrieve and complete them. The sub-queue path is <queue path>/$deadletterqueue; with the .NET library, set ServiceBusReceiverOptions.SubQueue to SubQueue.DeadLetter. Each dead-lettered message carries DeadLetterReason and DeadLetterErrorDescription. Once you have fixed the cause, you can resubmit messages, and Service Bus Explorer in the Azure portal lets you peek, edit and resend them individually or in batches.

Verify that leveling works

Test with a burst larger than your back end could handle directly, then check these signals:

1. Producers keep returning quickly during the burst.
2. Queue depth rises and then drains back to zero after the burst ends.
3. The downstream database or API shows a steady request rate and no throttling responses.
4. The dead-letter queue stays empty for valid traffic.

Microsoft's guidance adds a warning on the other side: if the average producer rate exceeds the consumer rate, the queue keeps growing and latency increases. Monitor queue depth and scale consumers within safe limits, or shed work at the producer.

Troubleshooting

Queue depth grows without limit. Consumers are slower than producers on average. Check consumer errors and retries, raise concurrency or the scale limit only if the back end can take it, or reduce what producers accept.

The database still times out. Total concurrency is too high. Reduce maxConcurrentCalls or the scale-out limit, and recalculate instances times concurrency.

Messages land in the dead-letter queue with MaxDeliveryCountExceeded. Either the handler throws, or messages are received but never settled, for example because the receiver closed before completing them. Fix the exception, and settle each message before closing the receiver or connection.

Messages arrive out of order. Multiple parallel consumers do not preserve order. Use sessions for strict ordering, or make handlers order-independent.

Large dead-letter descriptions are rejected. Dead-letter descriptions that contain stack traces can push a message past the 256 KB limit on the Standard tier. The Premium tier has higher limits.

Checklist

• The caller can accept an asynchronous result.
• A durable queue sits between producer and consumer.
• MessageId is derived from the business identifier.
• Consumer concurrency and scale-out are both bounded and tested against the back end.
• Handlers are idempotent.
• The dead-letter queue is monitored and has an owner.
• Queue depth and age are alerted on.

For the next step in decoupling, read Transactional Outbox Pattern to publish messages reliably from a database transaction.

References

• Microsoft Learn, Queue-Based Load Leveling pattern: https://learn.microsoft.com/en-us/azure/architecture/patterns/queue-based-load-leveling
• Microsoft Learn, Service Bus queues, topics and subscriptions: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions
• Microsoft Learn, Service Bus duplicate detection: https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection
• Microsoft Learn, Service Bus dead-letter queues: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues
• Microsoft Learn, Target-based scaling in Azure Functions: https://learn.microsoft.com/en-us/azure/azure-functions/functions-target-based-scaling