Rate limiting

Introduction

This article provides details about rate limiting in Jira to help you anticipate rate limiting and manage how your app responds.

Jira limits the rate of REST API requests to ensure that services are reliable and responsive for customers. Rate limiting is not used to differentiate levels of service to different types of customers or apps.

Implementation overview

The implementation of rate limiting is centered around tracking the cost of processing requests against various cost budgets. Depending on the type of authentication used, each API request is associated with one of these user-based budgets:

• User: the request was made by an end user.
• App: the request was made by the app without direct user interaction.
• App + user: the request was made by the app in association with a user interaction. Each combination of app and user has its own cost budget.
• Anonymous: the request was not associated with a user or app.

This table provides a mapping of call types to cost budgets.

REST API rate limits are not published because the computation logic is evolving continuously to maximize reliability and performance for customers.

Scaling by users

It is important to understand how to design and implement your app in order to ensure it is able to scale for tenants with large numbers of users.

Let’s say we have the following conditions/assumptions:

• “App” cost budget = 10 calls per second.
• “App + user” cost budget = 10 calls per second.
• An app is installed in a tenant with 100 active users.
• The app reacts to a particular type of user action (e.g. view an issue) in such a way that it makes 1 REST API call for each action.
• Each user in the tenant performs this action once per 10 seconds.

In the above scenario, the app will be making 100 users * 1 request / 10 seconds = 10 REST API calls per second.

If the app is making the API call in such a way that it counts against the “App” cost budget, then all API calls will be evaluated against the single “App” cost budget of 10 calls per second which means it will be on the brink of being rate limited.

Alternatively, if the app is making the API call in such a way that it counts against the “App + user” cost budget, the API calls will be evaluated against 100 separate “App + user” cost budget contexts of 10 calls per second each. The app will therefore only be making 0.1 calls per second against each of these “App + user” cost budget contexts which is well within the cost budget of 10 calls per second.

The key guideline here is that in order for an app to scale by the the number of users in a tenant, it must be coded such that user activity results in API calls made against the “App + user” cost budget rather than the “App” cost budget.

Rate limit responses

Rate limit detection

Apps can detect rate limits by checking if the HTTP response status code is 429. Any REST API can return a rate limit response.

Rate limit related headers

429 responses may be accompanied by Retry-After and X-RateLimit-Reset headers.

Retry-After indicates how many seconds the app must wait before reissuing the request. If you reissue the request before the retry period expires, the request will fail and return the same or longer Retry-After period.

X-RateLimit-Reset indicates the date time when you can retry the request. It’s provided in the format: yyyy-MM-ddTHH:mmZ.

X-RateLimit-NearLimit indicates that less than 20% of any budget remains. This is a boolean value added to a successful response.

Some transient 5XX errors are accompanied by a Retry-After header. For example, a 503 response may be returned when a resource limit has been reached. While these are not rate limit responses, they can be handled with similar logic as outlined below.

Retries

You can retry a failed request if all of these conditions are met:

• It is safe to do so from the perspective of the API (for example, the API being called is idempotent).
• A small retry threshold has not been reached.
• The response indicates a retry is appropriate (Retry-After header or 429 status).

Request backoff

Apps should treat 429 responses as a signal to alleviate pressure on an endpoint. Apps should retry the request only after a delay. Best practice to double the delay after each successive 429 response from a given endpoint. Backoff delays only need to exponentially increase to a maximum value at which point the retries can continue with the fixed maximum delay. You should also apply jitter to the delays to avoid the thundering herd problem.

The following articles provide useful insights and techniques related to retry and backoff processing:

• Exponential backoff article in Wikipedia
• Error retries and exponential backoff in AWS.

Configuration

There are several considerations that govern the rate limit response handling:

• Maximum number of retries
• Delay to first retry
• Maximum retry delay

These parameters are influenced by higher-level considerations, such as:

• Idempotency: Is it safe to retry calls to the API? Retrying on APIs with non-atomic processing can lead to side effects. This may be difficult for you to evaluate as the implementation of APIs is private, but be wary of APIs with “relative” semantics, for example, “set value” would probably be idempotent, but “add values” might not.
• Request contention: Some use cases may result in contention between requests which may also affect whether it is safe to retry requests. For example, multiple entities (users, automation rules, workflow steps, etc) might respond to a trigger with competing requests.
• Timeout: Is there a timeout associated with the request? Most UI interactions should have short timeouts. You can use longer timeouts for interactions where results are returned asynchronously so long as you provide appropriate indicators in the UI.
• Payload TTL: Will the request become stale?
• Failure impact: What is the impact of a request failure? If the impact is high, parameters can be tuned to back off more quickly, but with more retries and a greater timeout.
• Escalation: How would a failure be escalated and handled?

Response handling pseudo code

The following pseudo code illustrates the recommended response processing logic:

1
2
// Defaults may vary based on the app use case and APIs being called.
let maxRetries = 4; // Should be 0 to disable (e.g. API is not idempotent)
let lastRetryDelayMillis = 5000;
let maxRetryDelayMillis = 30000;
let jitterMultiplierRange = [0.7, 1.3];

// Re-entrant logic to send a request and process the response...
let response = await fetch(...);
if (response is OK) {
  handleSuccess(...);
} else {
  let retryDelayMillis = -1;
  if (hasHeader('Retry-After') {    
    retryDelayMillis = 1000 * headerValue('Retry-After');
  } else if (statusCode == 429) {
    retryDelayMillis = min(2 * lastRetryDelayMillis, maxRetryDelayMillis);
  }
  if (retryDelayMillis > 0 && retryCount < maxRetries) {
    retryDelayMillis += retryDelayMillis * randomInRange(jitterMultiplierRange);
    delay(retryDelayMillis);
    retryCount++;
    retryRequest(...);
  } else {
    handleFailure(...);
  }
}

Handling concurrent rate limiting

Some apps may invoke the REST API concurrently via the use of multiple threads and/or multiple execution nodes. When this is the case, developers may choose to share rate limit responses between threads and/or execution nodes such that API requests take into account rate limiting that may have occurred in other execution contexts. This sharing of rate limit responses should include cost budget information in order to be effective. If the app makes impersonated REST API calls from the app's backend and also calls the API from an app iframe, then this distribution of rate limit response data would logically have to also include the app's iframe(s). Distributing rate limit response data will be non-trivial, so an alternate strategy involves backing off more quickly and/or increasing the maximum number of retries. This second strategy may result in poorer performance and may need tuning if the characteristics of the app changes.

App request scheduling

These are some strategies to spread out requests and thereby lower the peaks.

App side multi-threading

A high level of concurrency in apps may slow the performance of Jira, causing a less responsive user experience. Significant levels of concurrency will also result in a greater chance of rate limiting.

If your app makes many similar requests in a short amount of time, coordination of backoff processing may be necessary.

Although multi-threaded apps may see greater throughput for a short period, you should not attempt to use concurrency to circumvent rate limiting.

Periodically scheduled processing

When performing scheduled tasks, apply jitter to requests to avoid the thundering herd problem. For example, try to avoid performing tasks “on the hour.” This approach can be applied to many types of actions, such as sending daily email digests.

Ad-hoc scheduled processing

When you need to perform a large amount of ad-hoc processing, such as when migrating data, you should anticipate and account for rate limiting. For example, if the API calls are directed to a single tenant, it may be possible to schedule the activity at night or on a weekend to minimize customer impact while maximizing throughput.

Lowering the request cost

Consolidating requests

Consolidating requests lowers the cost by aggregating tasks and reducing overhead. For example, instead of sending a request to create a Jira issue followed by requests that set fields or add properties, a better approach is to use a single request. This case is explained further in the community post POST /rest/api/2/issue (create issue) throws error when passing entity properties.

There are several "bulk" operations that consolidate requests. For example, Bulk create issue.

Many operations also enable queries to be consolidated by specifying expand query parameters.

Context parameters

Jira provides a range of context parameters that can help minimize the number of API requests necessary. Also, note that conditions can be sent as context parameters.

Limiting requested data

As rate limiting is based on concurrency and cost, minimizing the amount of data requested will yield benefits. An obvious strategy to start with is caching. You can also save resources by specifying which fields or properties to return when using operations such as Search for issues using JQL (GET). Similarly, only request the data you need when using expand query parameters. You can use pagination in your requests to limit the number of matches required. Using webhooks to subscribe to data updates can also lower your data request volume, thereby lowering the risk of rate limiting.

Testing

Do not perform rate limit testing against Atlassian cloud tenants because this will place load on Atlassian servers and may impact customers.

The Acceptable Use Policy identifies your obligations to avoid overwhelming Atlassian infrastructure.

Additional reading

• Atlassian Acceptable Use Policy, January 24, 2020.
• Efficient Use of Atlassian Cloud REST APIs, Raimonds Simanovskis, Founder and CEO, eazyBI, 2019.
• Exponential Backoff and Jitter, Marc Brooker, 2015.

Limitations and Enhancements

The following Jira issues capture known limitation and enhancements relating to rate limiting:

• ACJIRA-2254: As an app developer, I need to be able to validate my app's rate limit response handling
• ACJIRA-2255: As an app developer, I need to know the rate limits that my app is subject to