Rate limiting

Introduction

This article provides details about rate limiting in Confluence to help you anticipate rate limiting and manage how your app responds.

Confluence limits the rate of REST API requests to ensure that services are reliable and responsive for customers. Rate limiting is not used to differentiate levels of service to different types of customers or apps.

Implementation overview

Rate limiting is implemented as a set of rules that consider the number of threads handling certain kinds of requests, the cost of the requests, and the resources required by the requests.

Each rule applies to a unique combination of resources such as nodes, tenants, database hosts, endpoints, or compute resources. Rules are evaluated in a sequence designed to maximize computational efficiency.

Each rule has a rate threshold and processing request counter. Request processing above the threshold is blocked, including downstream processing. Cost calculations that involve other rules can also figure into rate limiting as different requests can require different types and amounts of resources.

REST API rate limits are not published because the computation logic is evolving continuously to maximize reliability and performance for customers.

Rate limiting for Connect app migrations

All requests migrating data from server to cloud must include the Migration-App header with value true. These requests are subject to two additional migration-specific rate limits:

• A rate limit on all requests containing this header sent from a tenant
• A rate limit on the requests containing this header applied per app in a tenant

For more information about these rate limits, see the App migration platform documentation.

Rate limit responses

Rate limit detection

Apps can detect rate limits by checking if the HTTP response status code is 429. Any REST API can return a rate limit response.

Retry-After headers

429 responses may be accompanied by the Retry-After header. This header indicates how many seconds the app must wait before reissuing the request. If you reissue the request before the retry period expires, the request will fail and return the same or longer Retry-After period.

Some transient 5XX errors are accompanied by a Retry-After header. For example, a 503 response may be returned when a resource limit is reached. While these are not rate limit responses, they can be handled with similar logic as outlined below.

Retries

You can retry a failed request if all of these conditions are met:

• It is safe to do so from the perspective of the API. (For example, the API being called is idempotent).
• A small retry threshold has not been reached.
• The response indicates a retry is appropriate (Retry-After header or 429 status).

Request backoff

Apps should treat 429 responses as a signal to alleviate pressure on an endpoint and retry the request only after a delay. The best practice is to double the delay after each successive 429 response from a given endpoint. Backoff delays only need to exponentially increase to a maximum value at which point the retries can continue with the fixed maximum delay. You should also apply jitter to the delays to avoid the thundering herd problem.

The following articles provide useful insights and techniques related to retry and backoff processing:

• Exponential backoff article in Wikipedia
• Error retries and exponential backoff in AWS.

Configuration

There are several considerations that govern the rate limit response handling:

• Maximum number of retries
• Delay to first retry
• Maximum retry delay

These parameters are influenced by higher-level considerations, such as:

• Idempotency: Is it safe to retry calls to the API? Retrying on APIs with non-atomic processing can lead to side effects. This may be difficult for you to evaluate as the implementation of APIs is private, but be wary of APIs with “relative” semantics, for example, “set value” would probably be idempotent, but “add values” might not.
• Request contention: Some use cases may result in contention between requests which may also affect whether it is safe to retry requests. For example, multiple entities (users, automation rules, workflow steps, etc) might respond to a trigger with competing requests.
• Timeout: Is there a timeout associated with the request? Most UI interactions should have short timeouts. You can use longer timeouts for interactions where results are returned asynchronously so long as you provide appropriate indicators in the UI.
• Payload TTL: Will the request become stale?
• Failure impact: What is the impact of a request failure? If the impact is high, parameters can be tuned to back off more quickly, but with more retries and a greater timeout.
• Escalation: How would a failure be escalated and handled?

Response handling pseudo code

The following pseudo code illustrates the recommended response processing logic:

1
2
// Defaults may vary based on the app use case and APIs being called.
let maxRetries = 4; // Should be 0 to disable (e.g. API is not idempotent)
let lastRetryDelayMillis = 5000;
let maxRetryDelayMillis = 30000;
let jitterMultiplierRange = [0.7, 1.3];

// Re-entrant logic to send a request and process the response...
let response = await fetch(...);
if (response is OK) {
  handleSuccess(...);
} else {
  let retryDelayMillis = -1;
  if (hasHeader('Retry-After') {    
    retryDelayMillis = 1000 * headerValue('Retry-After');
  } else if (statusCode == 429) {
    retryDelayMillis = min(2 * lastRetryDelayMillis, maxRetryDelayMillis);
  }
  if (retryDelayMillis > 0 && retryCount < maxRetries) {
    retryDelayMillis += retryDelayMillis * randomInRange(jitterMultiplierRange);
    delay(retryDelayMillis);
    retryCount++;
    retryRequest(...);
  } else {
    handleFailure(...);
  }
}

Handling concurrent rate limiting

Some apps may invoke the REST API concurrently via the use of multiple threads and/or multiple execution nodes. When this is the case, developers may choose to share rate limit responses between threads and/or execution nodes such that API requests take into account rate limiting that may have occurred in other execution contexts. Distributing rate limit response data will be non-trivial, so an alternate strategy involves backing off more quickly and/or increasing the maximum number of retries. This second strategy may result in poorer performance and may need tuning if the characteristics of the app changes.

App request scheduling

These are some strategies to spread out requests and thereby lower the peaks.

App side multi-threading

A high level of concurrency in apps may slow the performance of Confluence, causing a less responsive user experience. Significant levels of concurrency will also result in a greater chance of rate limiting.

If your app makes many similar requests in a short amount of time, coordination of backoff processing may be necessary.

Although multi-threaded apps may see greater throughput for a short period, you should not attempt to use concurrency to circumvent rate limiting.

Periodically scheduled processing

When performing scheduled tasks, apply jitter to requests to avoid the thundering herd problem. For example, try to avoid performing tasks “on the hour.” This approach can be applied to many types of actions, such as sending daily email digests.

Ad-hoc scheduled processing

When you need to perform a large amount of ad-hoc processing, such as when migrating data, you should anticipate and account for rate limiting. For example, if the API calls are directed to a single tenant, it may be possible to schedule the activity at night or on a weekend to minimize customer impact while maximizing throughput.

Lowering the request cost

Consolidating requests

There are several "bulk" operations that consolidate requests. For example, Get multiple users using ids.

Many operations also enable queries to be consolidated by specifying expand query parameters.

Context parameters

Confluence provides a range of context parameters that can help minimize the number of API requests necessary. Also, note that conditions can be sent as context parameters.

Limiting requested data

As rate limiting is based on concurrency and cost, minimizing the amount of data requested will yield benefits. An obvious strategy to start with is caching. You can also save resources by specifying which fields or properties to return when using operations such as Search for issues using JQL (GET). Similarly, only request the data you need when using expand query parameters. You can use pagination in your requests to limit the number of matches required. Using webhooks to subscribe to data updates can also lower your data request volume, thereby lowering the risk of rate limiting.

Testing

Do not perform rate limit testing against Atlassian cloud tenants because this will place load on Atlassian servers and may impact customers.

The Acceptable Use Policy identifies your obligations to avoid overwhelming Atlassian infrastructure.

Additional reading

• Atlassian Acceptable Use Policy, January 24, 2020.
• Efficient Use of Atlassian Cloud REST APIs, Raimonds Simanovskis, Founder and CEO, eazyBI, 2019.
• Exponential Backoff and Jitter, Marc Brooker, 2015.

Limitations and Enhancements

The following Jira issues capture known limitation and enhancements relating to rate limiting:

• CONFCLOUD-71203: As an app developer, I need to be able to validate my app's rate limit response handling
• CONFCLOUD-71116: As an app developer, I need to know the rate limits that my app is subject to