Cloud Fortified apps program reliability requirements

Principles

The reliability requirements for Cloud Fortified apps are built around these principles:

• A consistent reliability experience for customers: The focus Atlassian has given to reliability over the past 12 months has helped lift our services' quality and reduce the number and severity of incidents. To keep this experience consistent for customers, we want to ensure app reliability matches or betters product reliability.
• Detect issues before customers: The program should drive patterns and behaviours that enable partners and Atlassian to detect issues before customers feel their impact.
• A program that scales: While the initial program may involve manual processes as we iterate the boarding experience, the goal is to scale the reliability program to cope with new apps and partners.

Goals

The reliability requirements enables Cloud Fortified apps to:

1. Benefit from a consistent measure of the reliability of their core capabilities by using SLIs and SLOs. These measures provide shared visibility and understanding in reliability-related discussions with Atlassian and ultimately with customers.
2. Reduce their Mean Time To Detect (MTTD) through automated issue detection. More issues are detected by monitoring and not by customers.
3. Reduce their Mean Time To Recover (MTTR) by using an incident process that interfaces with Atlassian's incident process.
4. Continuously raise their quality bar by having partners conduct Post Incident Reviews with actionable improvements, signed off by Atlassian.

Key concepts

Reliability metrics

Generic SLIs

When Marketplace Partners begin the onboarding process, we measure the following for your application:

1. App availability: Measures whether your app is available. Determined by periodically calling a health check URL for your app that you supply, which is expected to return a 200 (any other result counts as a failure). Atlassian will call this URL about once per minute from an automated monitoring system, so we recommend keeping it simple and lightweight. It does not need to exercise end-to-end user experiences. A simple check to confirm your app is up and responding to requests is sufficient. Consistent response times above 3s are interpreted as failures.
2. Iframe load success rate: Measures whether your app is serving iframes correctly and responsively. By default, we can only measure whether the iframe establishes a bridge connection to the host within a reasonable time. We cannot determine whether the iframe's content is as expected. So any iframe load that successfully initiates the host bridge within 12s is counted as a success. You can refine this metric by emitting your own success and failure events when your iframe finishes rendering. See "Refine your iframe success metrics" below for more info.
3. Installation callback success rate: Measures whether your app is processing installation callbacks successfully and responsively.
4. Webhook delivery success rate: Measures whether your app is processing webhooks successfully and responsively.

Target SLOs and alerts

Atlassian will detect when an SLI is in breach of its SLO and notify you.

When you sign up for the Cloud Fortified apps program, you provide Atlassian with an email address that will receive notifications on breach events as per the alert conditions described below. If your team already uses an alerting or on-call management platform such as Opsgenie, you can use these emails to trigger alerts in your system.

1. App installations are infrequent, so to avoid noisy alerts, we use a wide time window for the installation callback success rate calculations. Your metrics dashboard also includes success and failure counts so that you can check for fluctuations over shorter time periods.

Alert notifications

The notification email is sent from no-reply@am.atlassian.com. You can filter messages by this sender.

Notification data

The email HTML body contains a script tag of type application/json and ID incident-data. This HTML element contains a JSON object that holds details about the alert. Here is an example:

1
2
<script type="application/json" id="incident-data">
{
  "appKey": "com.example.appkey",
  "appDetails": {
    "name": "AppName",
    "partnerName": "Atlassian"
  },
  "incidentId": "FOotlQrY5ZQ",
  "incidentStatus": "ANOMALOUS",
  "time": "2022-03-24T15:40:00Z",
  "metricName": "Jira Installation Callback Success Rate",
  "metricUnit": "PERCENT",
  "measurementDurationSeconds": 86400,
  "sliValue": 60.0,
  "sloValue": 99.0
}
</script>

If you need it, here is a download of the JSON schema.

Impact of SLO breaches for Cloud Fortified apps

Our aim is to help Marketplace Partners improve their apps' reliability. If an app breaches its SLOs continually, it's in our interest to get it back up to Cloud Fortified app standards. Therefore, there is an escalation of actions depending on the frequency and consistency of SLO breaches:

• Notification: For each SLO breach, we notify you as described above. You can assess the breach's impact and initiate the EcoHOT process, as described below, if the app is degraded and needs an emergency response.
• Remediation: For repeated breaches, a member of the Atlassian team contacts you to identify what is causing the breaches and assist in forming a strategy to bring the app back up to scratch.
• Demotion: Apps that continuously breach SLOs and fail to fix problems are demoted to a non-Cloud Fortified designation in the marketplace. We want to avoid this scenario because it has ramifications for customer expectations.

Monitoring your SLIs

While Atlassian detects when an SLI is in breach and notifies you, you also need to consider ongoing insight into your app's metrics (for example, identifying trends that suggest impending issues or reviewing the metrics before an app is fully enrolled to identify any required remediation).

As part of your onboarding, you are granted access to the Cloud Fortified apps Statuspage, which includes an overview of the key metrics for your apps:

The dashboard provides access to 1 month of data at a 5-minute resolution, with a latency of approximately 10 minutes. Only your team and Atlassian can see metrics for your apps (different partners cannot see each others' metrics). Data collection begins when Atlassian starts processing your application to join the program.

Non-SLI metrics

In addition to the metrics tracked for SLIs, the following reliability-related metrics are tracked for Cloud Fortified apps:

• Synthetic test success rate: We recommend that partners define between 1 and 3 core capabilities (written from the perspective of the customer) for each app. Then write a synthetic test for each of those core capabilities and run it against a Developer First Rollout tenant. This check should be run on a schedule and be monitored to catch breaking changes before they hit production. Test results should be shared with Atlassian for visibility, as these tests also enable Atlassian to identify when a change being rolled out to production impacts Apps. See below for more information on implementing these checks and sharing results.
• PIR actions: Issues highlighted as part of a Production Incident Review must be resolved within the specified time. The timeframes will depend on the severity and size of the action but should be signed off by someone from the Developer Relations team.
• Deprecation: Marketplace Partners must act on deprecation notices within the deprecation period.

Data sources

SLI metrics are collected from production data, so SLOs will be assessed against Prod Groups, as shown in this diagram. At the same time, the synthetic checks are run against the Developer-First Rollout group.

Incident management

An incident means an event that disrupts or reduces the quality of a service that requires an emergency response.

Incidents can be raised by the following methods, all of which will produce an EcoHOT ticket.

The EcoHOT ticket is the source of reference for partner and Atlassian efforts. We may also start a Slack channel for faster comms between Atlassian and partners.

EcoHOT workflow

Developer Relations Incident Manager

The Developer Relations Incident Manager is an on-call position within Atlassian who is paged if the partner or Atlassian Support believes Atlassian to be the cause of the incident. The Developer Relations IM then initiates Atlassian's internal incident management process to resolve the problem, providing feedback to the partner through the EcoHOT ticket. The Developer Relations IM is also responsible for signing off any actions a partner comes up with after a Production Incident Review is conducted.

Steps to follow

Identify or implement a health check resource (required for Connect apps)

To monitor your app's availability, we need a URL we can regularly poll to ensure your app is "up." This URL:

• Must return 200 if the app is healthy. Any other result counts as a failure.
• Must not return 200 if the app is down.
• Must be unauthenticated.
• Must return in under 3s. Consistent response times above 3s are interpreted as failures.
• Must be lightweight. Atlassian will poll this URL approximately every 60s.

The URL does not need to exercise end-to-end user experiences. A simple check to confirm your app is up, responding to requests, and able to access its datastores is sufficient.

Feel free to use an existing URL that meets the above criteria (you may have such a resource, for example, for load balancer health-checking or provided by your application framework).

Production readiness documentation (required)

As part of the Cloud Fortified app approval process, we need information about your app and organization in these areas:

Marketplace Partners submit this information in the approval ticket.

Refine your iframe success rate metrics (recommended)

One of the most important measures for your app its success rate for correctly service iframes. But by default, Atlassian can only measure whether the iframe establishes a bridge connection to the host within a reasonable time. We cannot determine whether the content of the iframe is as expected. Therefore, any iframe load that successfully establishes the bridge within 12s is counted as a success. This means that if the iframe loads but displays an error to the user because, say, a product API returned an incorrect result, this event is not be recorded as a failure.

To improve the accuracy of the metric, modify your code to emit success or failure events when your iframe finishes rendering by sending a PUT request to the addons-metrics API with metricType set to IFRAME.

Implement synthetic tests (recommended)

We expect Cloud Fortified apps to make some use of Developer First Rollout instances so that you can detect unexpected behavior as early as possible when product changes are rolled out. One way to make the most of these instances is to run regular synthetic tests against them.

Synthetic tests are automated tests that simulate real user interactions to validate core app capabilities and experiences. They are usually implemented with emulated web browsers or recorded web requests. In this context, we suggest you implement automated tests that simulate users interacting with your app through Jira or Confluence, run them regularly against your Developer First Rollout instance, and publish the test result to Atlassian. This enables you to quickly spot cases where product changes degrade your app's core capabilities and gives us visibility into when a product change impacts apps.

To implement synthetic tests:

• Identify 1-3 core capabilities to validate for your app. As a general source of inspiration, you can see some core capabilities for some Atlassian products here (our synthetic tests cover a much broader spectrum, but these are the essential experiences that we monitor most closely).
• Write tests to exercise those capabilities through Jira or Confluence using your chosen framework (for example, WebDriver).
• Execute the tests on a regular schedule, for example, hourly (not just as a one-off post-deploy step). This ensures failures caused by dependencies are detected.
• Use the addons-metrics API to publish the result of the test to Atlassian. Ensure you set metricType to SYNTHETIC.

Note: The authentication on the metric API means that the publish request must emanate from the app, not from the test framework. This means that you need to add logic to your app to report synthetic test results, which is suboptimal. We're exploring ways to remove this complexity.

Review and Monitor your SLIs (required)

Once Atlassian has granted you access, validate access to and familiarize yourself with the Cloud Fortified apps Statuspage.

Confirm the metrics you're seeing match your expectations and raise any unexpected behavior (for example, suspicious or missing data) with Atlassian.

Confirm you're receiving email notifications when SLIs breach their SLOs.

If your team uses an alerting or on-call management platform, such as OpsGenie, you can use these emails to trigger alerts in your system.

Trial the EcoHOT process (required)

Once Atlassian grants you access, familiarize yourself with the EcoHOT project.

• Confirm you can raise an EcoHOT ticket and that your team can access it
• If you'd like to ensure other partners cannot see your incident ticket, set the "Restrict to" field to just your app and administrators.
• Have your operations contacts watch the incident and PIR video training material (coming soon, not required at this time)

Approximate effort

These are estimates of the effort required to fulfil the onboarding requirements for the reliability program.

Metrics publish API doc

PUT - /rest/atlassian-connect/latest/addons-metrics/${addon_key}/publish

This API is used by Fortified apps to:

• Refine iframe success rate metrics by submitting custom success/failure events.
• Publish synthetic test results

Headers

Parameters

Responses