Last updated Nov 7, 2024

Data residency for Connect apps

Atlassian data residency enables Atlassian organization administrators to specify where subsets of their product data at rest are hosted. Each realm contains geographical regions. Administrators can choose between the following realms:

  • Global: In-scope data is hosted within realms determined by Atlassian: data may be moved between realms as needed
  • EU: In-scope data is hosted within the Frankfurt and Dublin AWS regions
  • US: In-scope data is hosted within the US East and US West AWS regions
  • AU: In-scope data is hosted within the Sydney AWS region
  • DE: In-scope data is hosted within the Frankfurt AWS region
  • SG: In-scope data is hosted within the Singapore AWS region
  • CA: In-scope data is hosted within the Canada AWS region
  • IN: In-scope data is hosted within the Mumbai AWS region
  • KR: In-scope data is hosted within the Seoul AWS region
  • JP: In-scope data is hosted within the Tokyo AWS region
  • GB: In-scope data is hosted within the London AWS region
  • CH: In-scope data is hosted within the Zurich AWS region

Data residency and Atlassian Marketplace apps

To support all the data residency use cases, apps have to implement both the realm pinning and the real migration capabilities.

Here is how the capabilities are used:

  • realm pinning the URLs defined as regionBaseUrls control the location when an app is initially installed. After a customer moved a product to a specific location, when an app is installed, the corresponding location from the Connect descriptor will be selected. Even if the product has not been moved to a specific location, the regionBaseUrls is used at app installation time if the app supports the location where the product is provisioned.
  • realm migration the lifecycle dare-migration hook is used when a customer requests a migration for the app data because the app is not in the same location as the product. This might be the case when the app was installed in a global location because the app didn’t have a regionBaseUrls section defined in its descriptor, or didn’t support the region where the host product was provisioned. Even if the app supported the regionBaseUrls at installation time, customers might have subsequently moved the product to a different location to comply with their data residency requirements. In these cases, customers will request the app data to be moved via the customer-facing UI.
  • realm persistence is an opt-in feature that allows subsequent installations of an app after an uninstallation to be installed to the location it was in before it was uninstalled. The duration of this behavior can be configured - an app's realm can be persisted for 1-30 days after uninstallation.

Adopting data residency is not compulsory, you can choose to maintain your app as-is. However, if you choose not to support regional data storage customers with stringent realm requirements may not be able to install your app.

Enable data residency

Realm pinning

To enable an app to use data residency:

  • Add a regionBaseUrls field to the app descriptor.
  • Add a key-value pair for the realm the app is hosted in and its corresponding baseUrl.

The regionBaseUrls field follows the same standards as the app’s main baseUrl field. The supported region keys are “EU”, “US”, "AU", "DE", "SG", "CA", "IN", "KR", "JP", "GB" and "CH".

The URLs listed in the following example are the address of the app within the given realm, and the app will store all data at rest within that realm.

1
2
{
  ...
  "regionBaseUrls": {
    "US": "https://us.your.sample.app.com",
    "EU": "https://eu.your.sample.app.com",
    "AU": "https://au.your.sample.app.com",
    "DE": "https://de.your.sample.app.com",
    "SG": "https://sg.your.sample.app.com",
    "CA": "https://ca.your.sample.app.com",
    "IN": "https://in.your.sample.app.com",
    "KR": "https://kr.your.sample.app.com",
    "JP": "https://jp.your.sample.app.com",
    "GB": "https://gb.your.sample.app.com",
    "CH": "https://ch.your.sample.app.com"
  }
}

Customers who chose a single realm require an app to list that realm in its list of supported realms. For example, a customer who chooses US data residency requires a URL that specifies the US:

1
2
"regionBaseUrls": { 
  "US": "https://us.your.sample.app.com"
}

or a list of URLs that include the US:

1
2
"regionBaseUrls": { 
  "US": "https://us.your.sample.app.com",
  "EU": "https://eu.your.sample.app.com" 
}

Customers who don’t have a data residency requirement, but are provisioned in a supported region for performance reasons, install into the nearest supported realm - otherwise into the global realm. This may provide performance improvements for end users.

To support data resident customers, you may need to spin up stacks outside of your current infrastructure then create databases for each realm, depending on how your app is structured.

You will need to provide a data residency policy, similar to the Atlassian Data Residency policy, and update any of your other customer policies affected.

If you indicate support for a realm, it doesn't mean your app simply supports customers in that realm. It means your app only stores the relevant, in-scope data in that realm and any in-scope data that you store must be stored in that realm.

Realm migration

Supporting data residency migrations

To support data residency migrations, you must support the data residency migration hook specified in the lifecycle field in the descriptor and the subsequent hooks part of the lifecycle of a migration.

A new lifecycle can be specified as follows:

1
2
{
  ...
  "lifecycle": {
    "installed": "/installed",
    "dare-migration": "/my-cool-migration-hook"
  },
  ...
}

This makes your app eligible to receive the following compulsory hooks:

  • /schedule
  • /start
  • /status
  • /commit
  • /rollback

We will append this hook onto your declared dare-migration lifecycle in the descriptor, for example: /my-cool-migration-hook/schedule.

These new lifecycles don't come from the product. Instead, they come from a new service. This is to facilitate the migration process while the product is down.

The service will continue to communicate with apps by signing the JWT token with the sharedSecret. You should continue to verify the request as described on Understanding JWT for Connect apps.

Data residency migrations lifecycle

EndpointDescriptionRequest payloadResponse
POST /scheduleWhen a product admin schedules a migration, the service will notify eligible apps via this endpoint to schedule a migration. Use this to do any preparatory work prior to the migration.
1
2
{
“startTime”: “2021-12-26T00:00:00.000Z”, 
“endTime”: “2021-12-27T00:00:00.000Z”, 
“location”: “EU”
}
If your app returns a non-2xx response without an errorResponseCode, we will retry up to two times. If these attempts are unsuccessful, the app migration is marked as a failure, and a rollback hook will be dispatched.
POST /startOnce the site is taken offline, the service will notify eligible apps that have successfully responded to the schedule hook via this endpoint to begin the migration.
1
2
{
 “startTime”: “2021-12-26T00:00:00.000Z”,
 “endTime”: “2021-12-27T00:00:00.000Z”,
 “location”: “EU”
}
If your app returns a non-2xx response without an errorResponseCode, we will retry up to two times. If these attempts are unsuccessful, the app migration is marked as a failure, and a rollback hook will be dispatched.
GET /statusThis endpoint will be called by the service to fetch the status of an ongoing migration. The statuses the service will specifically be looking for are:
  • failed
  • ready-to-commit
For failed, there will be a series of errorResponseCodes you may wish to report back with to provide more insight into the failure reason. This is described further under the Error Codes heading.

A ready-to-commit will mean that the app has finished its migration and is awaiting a /commit lifecycle upon completion.

Beyond that, you can use any statuses you wish. However, upon completion, any status that isn't ready-to-commit will receive a rollback hook.

Once the migration starts, we'll also be regularly polling the apps for their status. If all apps have either failed or are ready-to-commit, then the migration can end, and the site can be brought back online.

{}Failed status - service will update the app migration’s status to failed:
1
2
{
“status”: “failed”,
“errorResponseCode”: “E0004”
}
App is awaiting a /commit hook. The service, once the migration ends, will send a /commit hook:
1
2

{
“status”: “ready-to-commit”
}

POST /commitOnce the site is brought back online, the service will query the status of each app that has started the migration.

The service will ask apps to commit via this endpoint if they’ve reported that they're ready-to-commit.

Specifically, this is to commit the copied data to the new region.

{}If your app returns a non-2xx response without an errorResponseCode, we will retry up to two times. If these attempts are unsuccessful, the app migration is marked as a failure, and a rollback hook will be dispatched.
POST /rollbackThere are various use cases for this endpoint:
  • Migration cancelled by the user via the UI
  • No report of ready-to-commit by the app. Once the site is brought back online, the service will query the status of each app which has started the migration.

    The service will ask apps to roll back via this endpoint if they’ve reported that they aren't ready-to-commit.

    Specifically, this is intended to roll back the non-destructive copy operation to the new region.

  • {}Not applicable.

    Retries

    The migration hooks use a retry mechanism to handle transient errors and increase the chances of a successful migration. The hooks are retried up to 2 times after the initial call. The exemption is the status hook which is dispatched every 5 minutes.

    A retry is triggered when the app responds with a non-2xx code and no errorResponseCode. If an errorResponseCode is received, it signals a failure, stopping further retries.

    There is a 5 second pause between retries.

    Error Codes

    To help diagnose problems with migrations, we’re adding a set of standardised error codes that your app can report back to us with when you’re reporting back with a non-2xx to the hooks or the status retrieval.

    Any non-2xx response to any of the lifecycle events can optionally include any of the below error codes in its response to assist the user in understanding why the migration may have failed.

    1
    2
    {
     “errorResponseCode”: “E0004”
    }
    
    Error CodeReason
    E0001The migration was not scheduled.
    E0002The app does not support the target region (versioning issues).
    E0003The app does not support migrations (versioning issues).
    E0004The app has too many concurrent migrations.
    E9999Generic failure code.

    Considerations

    There are a few considerations you should think about when deciding to implement data residency migrations for Connect apps.

    • App migrations will be performed while the site is down
    • App migrations will be performed at a product-level - customers cannot choose individual apps to migrate
    • Eligible apps will have at most 24h to migrate data
    • The eligibility of an app to be migrated will be decided at the scheduling time

    Lifecycle of a Connect data residency migration

    1. The product admin will schedule a migration of eligible apps via AdminHub.
    2. Our service will notify eligible apps via the /schedule endpoint.
    3. The site will be unavailable at the scheduled time.
    4. Our service will notify apps that successfully responded to the /schedule endpoint to begin the migration via the /start endpoint.
    5. The site will be brought live at the specified end time or once all app migrations are complete.
      • If the migration window is over, our service will query the apps that successfully started the migration for their status via /status.
        • If the status is ready-to-commit, our service will send a /commit hook.
        • If the status is anything else, our service will send a /rollback hook.
      • If all app migrations have completed (i.e. responded failed or ready-to-commit when polled or reported back early):
        • If the status is ready-to-commit, our service will send a /commit hook.
        • If the status is anything else, our service will send a /rollback hook.
    6. Connect will then update the installed region of the apps that have finished the migration process in the specified window.

    Declaring Maximum Downtime

    Planning a migration downtime is challenging for enterprise customers, therefore a shorter app migration window encourages customers to migrate their apps. The app descriptor supports a maxMigrationDurationHours property, allowing partners to specify the maximum migration window. If this property is not declared, the maximum migration window will default to 2 hours.

    • Shorten to optimize the customer experience. Partners are encouraged to use the shortest possible migration duration they are able to support
    • Extend for up to 24 hours if a longer migration duration if required
    • Restrictions:
      • Max: 24
      • Min: 0.5
      • Increments of 0.5 only. i.e. 1, 1.5, 2 (0.7 for example would be invalid)
    1
    2
    {
      ...
      "dataResidency": {
        "maxMigrationDurationHours": 0.5,
      ...
    }
    

    If an app fails to migrate within the specified maxMigrationDuration window, the /rollback hook will be sent.

    Testing Connect data residency migrations

    Manual migration option

    A manual alternative to perform a test migration without using the UI is also available. This options allows for more control over an app data residency migration flow.

    You can manually test migrations by raising a support request.

    The requirements for a manual migration are:

    • an app that supports data residency and migrations
    • the app must be installed on the site you want to trigger a migration for
    • the user making the request must be at least a product admin of the site or have the consent of someone who is

    The manual migration testing process differs from the actual migration flow in the following ways:

    • the product isn't brought offline - migrations are done live during testing
    • you can specify any migration window, with a minimum duration of 1 hour
    • the product doesn't have to be pinned
    • the migration can go to any location, regardless of which one the product is pinned to

    To perform a manual migration test:

    1. Visit the developer support page.
    2. Click App Migration.
    3. Click Request for App Data Residency Migration Test.
    4. Fill out the provided template where:
      • App key refers to the addonKey specified in your descriptor.
      • Premium Cloud URL refers to the host name of the site you'are trying to trigger a manual migration test for.
      • Product refers to either Jira or Confluence.
      • Start time refers to when the migration will begin. Uses UTC+0 time.
      • End time refers to when the migration will end. If not specified, this will be set to 24 hours after the provided start time. Uses UTC+0 time.
    5. If you are not an admin of the site, acquire consent from an admin of the site on the ticket.

    This process is intended for testing purposes only and should NOT be utilised to perform migrations on your customer sites.

    All apps supporting data residency migration via the dare-migration hook will receive a migration request on the specified testing site.

    Realm Persistence

    Realm persistence is an opt-in feature that enables partners to ensure that any re-installations of their apps within a specific duration will persist to the same region that they were previously installed in.

    It can be enabled via setting the realmPersistenceDays property in the app descriptor, which specifies how long the previously installed region should be persisted following the app's uninstallation. realmPersistenceDays has a minimum value of 0 (for which the region will not be persisted) and a maximum value of 30 days.

    For example, if an app was installed in the EU region with the following snippet in its descriptor, all re-installations of the app will be in the EU region within 10 days of the app being uninstalled.

    1
    2
    {
      ...
      "dataResidency": {
        "realmPersistenceDays": 10,
      ...
    }
    

    Lifecycle of a data-resident app

    Installation

    During installation time, if the product is pinned to a realm, Connect will route all traffic to that realm. If an app doesn't support that particular realm, the traffic will be routed to the global realm instead.

    If the product isn't pinned to a realm, Connect will look at where the product is provisioned, known as the 'shard', and will route traffic to the corresponding realm of that shard. However, if the app doesn't support this realm, then Connect will route traffic to the global realm instead.

    Uninstallation

    If an app is uninstalled, information about where the installation was, will be removed. This means a subsequent reinstallation may end up in a different realm depending on whether an app adds support for a new realm or the product is now pinned to a new or different realm.

    Upgrade

    When an app is upgraded, the app will also continue to be in the realm of when it was installed.

    Existing installations after pinning

    If an administrator pins their product to a realm after an app is installed, the app will remain in its old realm.

    For example, if an app is installed into the US realm, then the product is later pinned to a realm, the traffic will still continue to be routed to the US server of the app.

    Rate this page: