Data residency

For more details on data residency and a glossary of terms see Manage data residency.

Atlassian data residency enables Atlassian organization administrators to specify where subsets of their product data at rest are hosted. Each realm contains geographical regions. Administrators can choose between the following realms:

Global: In-scope data is hosted within realms determined by Atlassian: data may be moved between realms as needed.
EU: In-scope data is hosted within the Dublin AWS regions.
US: In-scope data is hosted within the US East and US West AWS regions.
AU: In-scope data is hosted within the Sydney AWS region.
DE: In-scope data is hosted within the Frankfurt AWS region.
SG: In-scope data is hosted within the Singapore AWS region

Data residency and Atlassian Marketplace apps

To support all the data residency use cases, apps have to implement both the realm pinning and the real migration capabilities.

Here is how the capabilities are used:

realm pinning the URLs defined as regionBaseUrls control the location when an app is initially installed. After a customer moved a product to a specific location, when an app is installed, the corresponding location from the Connect descriptor will be selected. Even if the product has not been moved to a specific location, the regionBaseUrls is used at app installation time if the app supports the location where the product is provisioned.
realm migration the lifecycle dare-migration hook is used when a customer requests a migration for the app data because the app is not in the same location as the product. This might be the case when the app was installed in a global location because the app didn’t have a regionBaseUrls section defined in its descriptor, or didn’t support the region where the host product was provisioned. Even if the app supported the regionBaseUrls at installation time, customers might have subsequently moved the product to a different location to comply with their data residency requirements. In these cases, customers will request the app data to be moved via the customer-facing UI.

Adopting data residency is not compulsory, you can choose to maintain your app as-is. However, if you choose not to support regional data storage customers with stringent realm requirements may not be able to install your app.

Enable data residency

Realm pinning

To enable an app to use data residency:

Add a regionBaseUrls field to the app descriptor.
Add a key-value pair for the realm the app is hosted in and its corresponding baseUrl.

The regionBaseUrls field follows the same standards as the app’s main baseUrl field. The supported region keys are “EU”, “US”, "AU", "DE" and "SG".

The URLs listed in the following example are the address of the app within the given realm, and the app will store all data at rest within that realm.

1
2
{
  ...
  "regionBaseUrls": {
    "US": "https://us.your.sample.app.com",
    "EU": "https://eu.your.sample.app.com",
    "AU": "https://au.your.sample.app.com",
    "DE": "https://de.your.sample.app.com",
    "SG": "https://sg.your.sample.app.com"
  }
}

Customers who chose a single realm require an app to list that realm in its list of supported realms. For example, a customer who chooses US data residency requires a URL that specifies the US:

1
2
"regionBaseUrls": { 
  "US": "https://us.your.sample.app.com"
}

or a list of URLs that include the US:

1
2
"regionBaseUrls": { 
  "US": "https://us.your.sample.app.com",
  "EU": "https://eu.your.sample.app.com" 
}

Customers who don’t have a data residency requirement, but are provisioned in the EU, US, AU, DE or SG for performance reasons, install into the nearest supported realm - otherwise into the global realm. This may provide performance improvements for end users.

To support data resident customers, you may need to spin up stacks outside of your current infrastructure then create databases for EU, US, AU, DE or SG realms, depending on how your app is structured.

You will need to provide a data residency policy, similar to the Atlassian Data Residency policy, and update any of your other customer policies affected.

If you indicate support for a realm, it doesn't mean your app simply supports customers in that realm. It means your app only stores the relevant, in-scope data in that realm and any in-scope data that you store must be stored in that realm.

Realm migration

Connect data residency migrations is currently available in Preview.

The realm migration service and associated APIs are available for production use, and a 1-3 month deprecation period and notice will be provided for any breaking changes that may be introduced.

Supporting data residency migrations

To support data residency migrations, you must support the data residency migration hook specified in the lifecycle field in the descriptor and the subsequent hooks part of the lifecycle of a migration.

A new lifecycle can be specified as follows:

1
2
{
  ...
  "lifecycle": {
    "installed": "/installed",
    "dare-migration": "/my-cool-migration-hook"
  },
  ...
}

This makes your app eligible to receive the following compulsory hooks:

/schedule
/start
/status
/commit
/rollback

We will append this hook onto your declared dare-migration lifecycle in the descriptor, for example: /my-cool-migration-hook/schedule.

These new lifecycles don't come from the product. Instead, they come from a new service. This is to facilitate the migration process while the product is down.

The service will continue to communicate with apps by signing the JWT token with the sharedSecret. You should continue to verify the request as described on Understanding JWT for Connect apps.

Data residency migrations lifecycle

Endpoint	Description	Request payload	Response
POST `/schedule`	When a product admin schedules a migration, the service will notify eligible apps via this endpoint to schedule a migration. Use this to do any preparatory work prior to the migration.	1 2 `{ “startTime”: “2021-12-26T00:00:00.000Z”, “endTime”: “2021-12-27T00:00:00.000Z”, “location”: “EU” }`	If your app responds with a non-2xx, the app migration will be marked as a failure and won't receive further hooks (besides the rollback hook) for the same migration.
POST `/start`	Once the site is taken offline, the service will notify eligible apps that have successfully responded to the schedule hook via this endpoint to begin the migration.	1 2 `{ “startTime”: “2021-12-26T00:00:00.000Z”, “endTime”: “2021-12-27T00:00:00.000Z”, “location”: “EU” }`	If your app responds with a non-2xx, the app migration will be marked as a failure and won't receive further hooks (besides the rollback hook) for the same migration.
GET `/status`	This endpoint will be called by the service to fetch the status of an ongoing migration. The statuses the service will specifically be looking for are: `failed` `ready-to-commit` For `failed`, there will be a series of `errorResponseCodes` you may wish to report back with to provide more insight into the failure reason. This is described further under the `Error Codes` heading. A `ready-to-commit` will mean that the app has finished its migration and is awaiting a `/commit` lifecycle upon completion. Beyond that, you can use any statuses you wish. However, upon completion, any status that isn't `ready-to-commit` will receive a rollback hook. Once the migration starts, we'll also be regularly polling the apps for their status. If all apps have either `failed` or are `ready-to-commit`, then the migration can end, and the site can be brought back online.	`{}`	Failed status - service will update the app migration’s status to failed: 1 2 `{ “status”: “failed”, “errorResponseCode”: “E0004” }` App is awaiting a `/commit` hook. The service, once the migration ends, will send a `/commit` hook: 1 2 `{ “status”: “ready-to-commit” }`
POST `/commit`	Once the site is brought back online, the service will query the status of each app that has started the migration. The service will ask apps to commit via this endpoint if they’ve reported that they're `ready-to-commit`. Specifically, this is to commit the copied data to the new region.	`{}`	If your app responds with a non-2xx, the app migration will be marked as a failure and will receive a rollback hook.
POST `/rollback`	There are various use cases for this enpoint: Migration cancelled by the user via the UI No report of `ready-to-commit` by the app. Once the site is brought back online, the service will query the status of each app which has started the migration. The service will ask apps to roll back via this endpoint if they’ve reported that they aren't `ready-to-commit`. Specifically, this is intended to roll back the non-destructive copy operation to the new region.	`{}`	Not applicable.

Error Codes

To help diagnose problems with migrations, we’re adding a set of standardised error codes that your app can report back to us with when you’re reporting back with a non-2xx to the hooks or the status retrieval.

Error Code	Reason
`E0001`	The migration was not scheduled.
`E0002`	The app does not support the target region (versioning issues).
`E0003`	The app does not support migrations (versioning issues).
`E0004`	The app has too many concurrent migrations.
`E9999`	Generic failure code.

Considerations

There are a few considerations you should think about when deciding to implement data residency migrations for Connect apps.

App migrations will be performed while the site is down
App migrations will be performed at a product-level - customers cannot choose individual apps to migrate
Eligible apps will have at most 24h to migrate data
The eligibility of an app to be migrated will be decided at the scheduling time

Lifecycle of a Connect data residency migration

The product admin will schedule a migration of eligible apps via AdminHub.
Our service will notify eligible apps via the /schedule endpoint.
The site will be unavailable at the scheduled time.
Our service will notify apps that successfully responded to the /schedule endpoint to begin the migration via the /start endpoint.
The site will be brought live at the specified end time or once all app migrations are complete.
- If the migration window is over, our service will query the apps that successfully started the migration for their status via /status.
  - If the status is ready-to-commit, our service will send a /commit hook.
  - If the status is anything else, our service will send a /rollback hook.
- If all app migrations have completed (i.e. responded failed or ready-to-commit when polled or reported back early):
  - If the status is ready-to-commit, our service will send a /commit hook.
  - If the status is anything else, our service will send a /rollback hook.
Connect will then update the installed region of the apps that have finished the migration process in the specified window.

Testing Connect data residency migrations

Manual migration option

A manual alternative to perform a test migration without using the UI is also available. This options allows for more control over an app data residency migration flow.

You can manually test migrations by raising a support request.

The requirements for a manual migration are:

an app that supports data residency and migrations
the app must be installed on the site you want to trigger a migration for
the user making the request must be at least a product admin of the site or have the consent of someone who is

The manual migration testing process differs from the actual migration flow in the following ways:

the product isn't brought offline - migrations are done live during testing
you can specify any migration window, with a minimum duration of 1 hour
the product doesn't have to be pinned
the migration can go to any location, regardless of which one the product is pinned to

To perform a manual migration test:

Visit the developer support page.
Click App Migration.
Click Request for App Data Residency Migration Test.
Fill out the provided template where:
- App key refers to the addonKey specified in your descriptor.
- Premium Cloud URL refers to the host name of the site you'are trying to trigger a manual migration test for.
- Product refers to either Jira or Confluence.
- Start time refers to when the migration will begin. Uses UTC+0 time.
- End time refers to when the migration will end. If not specified, this will be set to 24 hours after the provided start time. Uses UTC+0 time.
If you are not an admin of the site, acquire consent from an admin of the site on the ticket.

This process is intended for testing purposes only and should NOT be utilised to perform migrations on your customer sites.

All apps supporting data residency migration via the dare-migration hook will receive a migration request on the specified testing site.

Lifecycle of a data-resident app

Installation

During installation time, if the product is pinned to a realm, Connect will route all traffic to that realm. If an app doesn't support that particular realm, the traffic will be routed to the global realm instead.

If the product isn't pinned to a realm, Connect will look at where the product is provisioned, known as the 'shard', and will route traffic to the corresponding realm of that shard. However, if the app doesn't support this realm, then Connect will route traffic to the global realm instead.

Uninstallation

If an app is uninstalled, information about where the installation was, will be removed. This means a subsequent reinstallation may end up in a different realm depending on whether an app adds support for a new realm or the product is now pinned to a new or different realm.

Upgrade

When an app is upgraded, the app will also continue to be in the realm of when it was installed.

Existing installations after pinning

If an administrator pins their product to a realm after an app is installed, the app will remain in its old realm.

For example, if an app is installed into the US realm, then the product is later pinned to a realm, the traffic will still continue to be routed to the US server of the app.