Data Center App Performance Toolkit User Guide For Jira

This document walks you through the process of testing your app on Jira using the Data Center App Performance Toolkit. These instructions focus on producing the required performance and scale benchmarks for your Data Center app.

In this document, we cover the use of the Data Center App Performance Toolkit on two types of environments:

Development environment: Jira Data Center environment for a test run of Data Center App Performance Toolkit and development of app-specific actions. We recommend you use the AWS Quick Start for Jira Data Center with the parameters prescribed here.

1. Set up a development environment Jira Data Center on AWS.
2. Create a dataset for the development environment.
3. Run toolkit on the development environment locally.
4. Develop and test app-specific actions locally.

Enterprise-scale environment: Jira Data Center environment used to generate Data Center App Performance Toolkit test results for the Marketplace approval process. Preferably, use the AWS Quick Start for Jira Data Center with the parameters prescribed below. These parameters provision larger, more powerful infrastructure for your Jira Data Center.

5. Set up an enterprise-scale environment Jira Data Center on AWS.
6. Load an enterprise-scale dataset on your Jira Data Center deployment.
7. Set up an execution environment for the toolkit.
8. Running the test scenarios from execution environment against enterprise-scale Jira Data Center.

Development environment

Running the tests in a development environment helps familiarize you with the toolkit. It'll also provide you with a lightweight and less expensive environment for developing. Once you're ready to generate test results for the Marketplace Data Center Apps Approval process, run the toolkit in an enterprise-scale environment.

1. Setting up Jira Data Center development environment

We recommend that you set up a this development using the AWS Quick Start for Jira Data Center. All the instructions on this page are optimized for AWS. If you already have an existing Jira Data Center environment, you can also use that too (if so, skip to Create a dataset for the development environment).

Using the AWS Quick Start for Jira

If you are a new user, perform an end-to-end deployment. This involves deploying Jira into a new ASI.

If you have already deployed the ASI separately by using the ASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center), deploy Jira into your existing ASI.

To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.

AWS cost estimation for the development environment

AWS Jira Data Center development environment infrastructure costs about 10 - 15$ per working week depending on such factors like region, instance type, deployment type of DB, and other.

Quick Start parameters for development environment

All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.

Jira setup

Cluster nodes

Database

Networking (for new ASI)

Networking (for existing ASI)

Running the setup wizard

After successfully deploying the Jira Data Center on AWS, configure it as follows:

1. In the AWS console, go to Services > CloudFormation > Stack > Stack details > Select your stack.
2. On the Outputs tab, copy the value of the LoadBalancerURL key.
3. Open LoadBalancerURL in your browser. This will take you to the Jira setup wizard.

4. On the Set up application properties page, fill in the following fields:

   • Application Title: any name for your Jira Data Center deployment
   • Mode: private
   • Base URL: your stack's Elastic LoadBalancer URL

   Then select Next.

5. On the next page, fill in the Your License Key field in one of the following ways:

   • Using your existing license
   • Generating a Jira trial license
   • Contacting Atlassian to be provided two time-bomb licenses for testing. Ask for the licenses in your DCHELP ticket.

   Then select Next.

6. On the Set up administrator account page, fill in the following fields:

   • Full name: a full name of the admin user
   • Email Address: email address of the admin user
   • Username: admin (recommended)
   • Password: admin (recommended)
   • Confirm Password: admin (recommended)

   Then select Next.

7. On the Set up email notifications page, configure your email notifications, and then select Finish.

8. After going through the welcome setup, select Create new project to create a new project.

2. Generate dataset for development environment

After creating the development environment Jira Data Center, generate test dataset to run Data Center App Performance Toolkit:

• 1 Scrum software development project with 1-5 issues
• 1 Kanban software development project with 1-5 issues

3. Run toolkit on the development environment locally

1. Clone Data Center App Performance Toolkit locally.
2. Follow the README.md instructions to set up toolkit locally.
3. Navigate to dc-app-performance-toolkit/app folder.

4. Open the jira.yml file and fill in the following variables:

   • application_hostname: your_dc_jira_instance_hostname without protocol.
   • application_protocol: http or https.
   • application_port: for HTTP - 80, for HTTPS - 443, 8080, 2990 or your instance-specific port.
   • secure: True or False. Default value is True. Set False to allow insecure connections, e.g. when using self-signed SSL certificate.
   • application_postfix: it is empty by default; e.g., /jira for url like this http://localhost:2990/jira.
   • admin_login: admin user username.
   • admin_password: admin user password.
   • load_executor: executor for load tests. Valid options are jmeter (default) or locust.
   • concurrency: 2 - number of concurrent JMeter/Locust users.
   • test_duration: 5m - duration of the performance run.
   • ramp-up: 3s - amount of time it will take JMeter or Locust to add all test users to test execution.
   • total_actions_per_hour: 5450 - number of total JMeter/Locust actions per hour.
   • WEBDRIVER_VISIBLE: visibility of Chrome browser during selenium execution (False is by default).

5. Run bzt.

   1

   bzt jira.yml

6. Review the resulting table in the console log. All JMeter/Locust and Selenium actions should have 100% Success rate.
   In case some actions does not have 100% Success rate refer to the following logs in dc-app-performance-toolkit/app/results/jira/YY-MM-DD-hh-mm-ss folder:

   1
   2
   3
   4
   5
   6

   - `results_summary.log`: detailed run summary
   - `results.csv`: aggregated .csv file with all actions and timings
   - `bzt.log`: logs of the Taurus tool execution
   - `jmeter.*`: logs of the JMeter tool execution
   - `locust.*`: logs of the Locust tool execution (in case you use Locust as load_executor in jira.yml)
   - `pytest.*`: logs of Pytest-Selenium execution

4. Develop and test app-specific action locally

Data Center App Performance Toolkit has its own set of default test actions for Jira Data Center: JMeter/Locust and Selenium for load and UI tests respectively.

App-specific action - action (performance test) you have to develop to cover main use cases of your application. Performance test should focus on the common usage of your application and not to cover all possible functionality of your app. For example, application setup screen or other one-time use cases are out of scope of performance testing.

1. Define main use case of your app. Usually it is one or two main app use cases.
2. Your app adds new UI elements in Jira Data Center - Selenium app-specific action has to be developed.
3. Your app introduces new endpoint or extensively calls existing Jira Data Center API - JMeter/Locust app-specific actions has to be developed.
   JMeter and Locust actions are interchangeable, so you could select the tool you prefer:

• JMeter - UI-based performance tool.
• Locust - code-based (Python requests library) performance tool.

Custom dataset

You can filter your own app-specific issues for your app-specific actions.

1. Create app-specific issues that have specific anchor in summary, e.g. AppIssue anchor and issues summaries like AppIssue1, AppIssue2, AppIssue3.
2. Go to the search page of your Jira Data Center - JIRA_URL/issues/?jql= and select Advanced.
3. Write JQL that filter just your issues from step 1, e.g. summary ~ 'AppIssue*'.
4. Edit Jira configuration file dc-app-performance-toolkit/app/jira.yml:
   • custom_dataset_query: JQL from step 3.

Next time when you run toolkit, custom dataset issues will be stored to the dc-app-performance-toolkit/app/datasets/jira/custom-issues.csv with columns: issue_key, issue_id, project_key.

Example of app-specific Selenium action development with custom dataset

You develop an app that adds some additional fields to specific types of Jira issues. In this case, you should develop Selenium app-specific action:

1. Create app-specific Jira issues with AppIssue anchor in summary: AppIssue1, AppIssue2, etc.
2. Go to the search page of your Jira Data Center - JIRA_URL/issues/?jql= and check if JQL is correct: summary ~ 'AppIssue*'.
3. Edit dc-app-performance-toolkit/app/jira.yml configuration file and set custom_dataset_query: summary ~ 'AppIssue*'.
4. Extend example of app-specific action in dc-app-performance-toolkit/app/extension/jira/extension_ui.py.
   So, our test have to open app-specific issues and measure time to load of this app-specific issues.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import random

from selenium.webdriver.common.by import By

from selenium_ui.base_page import BasePage
from selenium_ui.conftest import print_timing
from util.conf import JIRA_SETTINGS


def app_specific_action(webdriver, datasets):
    page = BasePage(webdriver)
    app_specific_issue = random.choice(datasets['custom_issues'])
    issue_key = app_specific_issue[0]

    @print_timing("selenium_app_custom_action")
    def measure():

        @print_timing("selenium_app_custom_action:view_issue")
        def sub_measure():
            page.go_to_url(f"{JIRA_SETTINGS.server_url}/browse/{issue_key}")
            page.wait_until_visible((By.ID, "summary-val"))  # Wait for summary field visible
            page.wait_until_visible((By.ID, "ID_OF_YOUR_APP_SPECIFIC_UI_ELEMENT"))  # Wait for you app-specific UI element by ID selector
        sub_measure()
    measure()

5. In dc-app-performance-toolkit/app/selenium_ui/jira_ui.py, review and uncomment the following block of code to make newly created app-specific actions executed:

1
2

# def test_1_selenium_custom_action(webdriver, datasets, screen_shots):
#     app_specific_action(webdriver, datasets)

6. Run toolkit with bzt jira.yml command to ensure that all Selenium actions including app_specific_action are successful.

Example of app-specific Locust/JMeter action development

You develop an app that introduces new GET and POST endpoints in Jira Data Center. In this case, you should develop Locust or JMeter app-specific action.

Locust app-specific action development example

1. Extend example of app-specific action in dc-app-performance-toolkit/app/extension/jira/extension_locust.py, so that test will call the endpoint with GET request, parse response use these data to call another endpoint with POST request and measure response time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import re
from locustio.common_utils import init_logger, jira_measure

logger = init_logger(app_type='jira')


@jira_measure
def app_specific_action(locust):
    r = locust.get('/app/get_endpoint')  # call app-specific GET endpoint
    content = r.content.decode('utf-8')   # decode response content

    token_pattern_example = '"token":"(.+?)"'
    id_pattern_example = '"id":"(.+?)"'
    token = re.findall(token_pattern_example, content)  # get TOKEN from response using regexp
    id = re.findall(id_pattern_example, content)    # get ID from response using regexp

    logger.locust_info(f'token: {token}, id: {id}')  # log information for debug when verbose is true in confluence.yml file
    if 'assertion string' not in content:
        logger.error(f"'assertion string' was not found in {content}")
    assert 'assertion string' in content  # assert specific string in response content

    body = {"id": id, "token": token}  # include parsed variables to POST request body
    headers = {'content-type': 'application/json'}
    r = locust.post('/app/post_endpoint', body, headers)  # call app-specific POST endpoint
    content = r.content.decode('utf-8')
    if 'assertion string after successful POST request' not in content:
        logger.error(f"'assertion string after successful POST request' was not found in {content}")
    assert 'assertion string after successful POST request' in content  # assertion after POST request

2. In dc-app-performance-toolkit/app/jira.yml set load_executor: locust to make locust as load executor.
3. Locust uses actions percentage as relative weights, so if view_issue: 43 and standalone_extensions: 86 that means thats standalone_extension will be called twice more.
   Set standalone_extension weight in accordance with the expected frequency of your app use case compared with other base actions.
4. Run toolkit with bzt jira.yml command to ensure that all Locust actions including app_specific_action are successful.

JMeter app-specific action development example

1. Navigate to dc-app-performance-toolkit/app folder and launch JMeter by ~/.bzt/jmeter-taurus/5.2.1/bin/jmeter (it is important to launch from app folder), open dc-app-performance-toolkit/app/jmeter/jira.jmx.
2. Open Jira thread group > actions per login and navigate to standalone_extension
3. Add GET HTTP Request: right-click to standalone_extension > Add > Sampler HTTP Request, chose method GET and set endpoint in Path.
4. Add Regular Expression Extractor: right-click to to newly created HTTP Request > Add > Post processor > Regular Expression Extractor
5. Add Response Assertion: right-click to newly created HTTP Request > Add > Assertions > Response Assertion and add assertion with Contains, Matches, Equals, etc types.
6. Add POST HTTP Request: right-click to standalone_extension > Add > Sampler HTTP Request, chose method POST, set endpoint in Path and add Parameters or Body Data if needed.
7. Navigate to Global Variables and modify default values of hostname, port, protocol and postfix variables.
8. Navigate to load profile and set perc_standalone_extension default percentage to 100.
9. Right-click on View Results Tree and enable this controller.
10. Click Start button and make sure that login_and_view_dashboard and standalone_extension are successful.
11. Right-click on View Results Tree and disable this controller.
12. Click Save button.
13. To make standalone_extension executable during toolkit run edit dc-app-performance-toolkit/app/jira.yml and set execution percentage of standalone_extension accordingly to your use case frequency.
14. Run toolkit to ensure that all JMeter actions including standalone_extension are successful.

Using JMeter variables from the base script

Use or access the following variables in your standalone_extension script if needed.

• ${issue_key} - issue key being viewed or modified (e.g. ABC-123)
• ${issue_id} - issue id being viewed or modified (e.g. 693484)
• ${project_key} - project key being viewed or modified (e.g. ABC)
• ${project_id} - project id being viewed or modified (e.g. 3423)
• ${scrum_board_id} - scrum board id being viewed (e.g. 328)
• ${kanban_board_id} - kanban board id being viewed (e.g. 100)
• ${jql} - jql query being used (e.g. text ~ "qrk*" order by key)
• ${username} - the logged in username (e.g. admin)

Enterprise-scale environment

After adding your custom app-specific actions, you should now be ready to run the required tests for the Marketplace Data Center Apps Approval process. To do this, you'll need an enterprise-scale environment.

5. Set up an enterprise-scale environment Jira Data Center on AWS

We recommend that you use the AWS Quick Start for Jira Data Center to deploy a Jira Data Center enterprise-scale environment. This Quick Start will allow you to deploy Jira Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.

The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Jira into this new VPC. Deploying Jira with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.

Using the AWS Quick Start for Jira

If you are a new user, perform an end-to-end deployment. This involves deploying Jira into a new ASI.

If you have already deployed the ASI separately by using the ASI Quick StartASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center development environment), deploy Jira into your existing ASI.

To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.

AWS cost estimation

AWS Pricing Calculator provides an estimate of usage charges for AWS services based on certain information you provide. Monthly charges will be based on your actual usage of AWS services and may vary from the estimates the Calculator has provided.

*The prices below are approximate and may vary depending on such factors like region, instance type, deployment type of DB, and other.

Stop Jira cluster nodes

To reduce AWS infrastructure costs you could stop Jira nodes when the cluster is standing idle.
Jira node might be stopped by using Suspending and Resuming Scaling Processes.

To stop one node within the Jira cluster, follow the instructions below:

1. Go to EC2 Auto Scaling Groups and open the necessary group to which belongs the node you want to stop.
2. Press Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and add HealthCheck to the Suspended Processes. Amazon EC2 Auto Scaling stops marking instances unhealthy as a result of EC2 and Elastic Load Balancing health checks.
3. Go to Instances and stop Jira node.

To return Jira node into a working state follow the instructions:

1. Go to Instances and start Jira node, wait a few minutes for Jira node to become responsible.
2. Go to EC2 Auto Scaling Groups and open the necessary group to which belongs the node you want to start.
3. Press Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and remove HealthCheck from Suspended Processes of Auto Scaling Group.

Quick Start parameters

All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.

Jira setup

Cluster nodes

Database

Networking (for new ASI)

Networking (for existing ASI)

Running the setup wizard

After successfully deploying Jira Data Center in AWS, you'll need to configure it:

1. In the AWS console, go to Services > CloudFormation > Stack > Stack details > Select your stack.
2. On the Outputs tab, copy the value of the LoadBalancerURL key.
3. Open LoadBalancerURL in your browser. This will take you to the Jira setup wizard.
4. On the Set up application properties page, populate the following fields:
   • Application Title: any name for your Jira Data Center deployment
   • Mode: Private
   • Base URL: your stack's Elastic LoadBalancer URL Click Next.
5. On the next page, populate the Your License Key field by either:
   • Using your existing license, or
   • Generating a Jira trial license, or
   • Contacting Atlassian to be provided two time-bomb licenses for testing. Ask for it in your DCHELP ticket. Click Next.
6. On the Set up administrator account page, populate the following fields:
   • Full name: any full name of the admin user
   • Email Address: email address of the admin user
   • Username: admin (recommended)
   • Password: admin (recommended)
   • Confirm Password: admin (recommended) Click Next.
7. On the Set up email notifications page, configure your email notifications, and then click Finish.
8. After going through the welcome setup, click Create new project to create a new project.

6. Preloading your Jira deployment with an enterprise-scale dataset

Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.

Pre-loading the dataset is a three-step process:

1. Importing the main dataset. To help you out, we provide an enterprise-scale dataset you can import either via the populate_db.sh script or restore from xml backup file.
2. Restoring attachments. We also provide attachments, which you can pre-load via an upload_attachments.sh script.
3. Re-indexing Jira Data Center. For more information, go to Re-indexing Jira.

The following subsections explain each step in greater detail.

Importing the main dataset

You can load this dataset directly into the database (via a populate_db.sh script), or import it via XML.

Option 1 (recommended): Loading the dataset via populate_db.sh script (~1 hour)

To populate the database with SQL:

1. In the AWS console, go to Services > EC2 > Instances.
2. On the Description tab, do the following:
   • Copy the Public IP of the Bastion instance.
   • Copy the Private IP of the Jira node instance.

3. Using SSH, connect to the Jira node via the Bastion instance:

   For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS:

   1
   2
   3
   4
   5

   ssh-add path_to_your_private_key_pem
   export BASTION_IP=bastion_instance_public_ip
   export NODE_IP=node_private_ip
   export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
   ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

   For more information, go to Connecting your nodes over SSH.

4. Download the populate_db.sh script and make it executable:

   1

   wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/jira/populate_db.sh && chmod +x populate_db.sh

5. Review the following Variables section of the script:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10

   INSTALL_PSQL_CMD="amazon-linux-extras install -y postgresql10"
   DB_CONFIG="/var/atlassian/application-data/jira/dbconfig.xml"
   JIRA_CURRENT_DIR="/opt/atlassian/jira-software/current"
   CATALINA_PID_FILE="${JIRA_CURRENT_DIR}/work/catalina.pid"
   JIRA_DB_NAME="jira"
   JIRA_DB_USER="postgres"
   JIRA_DB_PASS="Password1!"
   JIRA_SETENV_FILE="${JIRA_CURRENT_DIR}/bin/setenv.sh"
   JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version"
   DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"

6. Run the script:

   1

   ./populate_db.sh | tee -a populate_db.log

Option 2: Loading the dataset through XML import (~4 hours)

We recommend that you only use this method if you are having problems with the populate_db.sh script.

1. In the AWS console, go to Services > EC2 > Instances.
2. On the Description tab, do the following:
   • Copy the Public IP of the Bastion instance.
   • Copy the Private IP of the Jira node instance.

3. Using SSH, connect to the Jira node via the Bastion instance:

   For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS:

   1
   2
   3
   4
   5

   ssh-add path_to_your_private_key_pem
   export BASTION_IP=bastion_instance_public_ip
   export NODE_IP=node_private_ip
   export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
   ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

   For more information, go to Connecting your nodes over SSH.

4. Download the xml_backup.zip file corresponding to your Jira version.

   1
   2

   JIRA_VERSION=$(sudo su jira -c "cat /media/atl/jira/shared/jira-software.version")
   sudo su jira -c "wget https://centaurus-datasets.s3.amazonaws.com/jira/${JIRA_VERSION}/large/xml_backup.zip -O /media/atl/jira/shared/import/xml_backup.zip"

5. From a different computer, log in as a user with the Jira System Administrators global permission.

6. Go to > System > Restore System. from the menu.
7. Populate the File name field with xml_backup.zip.
8. Click Restore and wait until the import is completed.

Restoring attachments (~2 hours)

After Importing the main dataset, you'll now have to pre-load an enterprise-scale set of attachments.

1. Using SSH, connect to the Jira node via the Bastion instance:

   For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS:

   1
   2
   3
   4
   5

   ssh-add path_to_your_private_key_pem
   export BASTION_IP=bastion_instance_public_ip
   export NODE_IP=node_private_ip
   export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
   ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@$BASTION_IP" ec2-user@${NODE_IP}

   For more information, go to Connecting your nodes over SSH.

2. Download the upload_attachments.sh script and make it executable:

   1

   wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/jira/upload_attachments.sh && chmod +x upload_attachments.sh

3. Review the following Variables section of the script:

   1
   2
   3
   4
   5

   DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
   ATTACHMENTS_TAR="attachments.tar.gz"
   ATTACHMENTS_DIR="attachments"
   TMP_DIR="/tmp"
   EFS_DIR="/media/atl/jira/shared/data"

4. Run the script:

   1

   ./upload_attachments.sh | tee -a upload_attachments.log

Re-indexing Jira Data Center (~30 min)

For more information, go to Re-indexing Jira.

1. Log in as a user with the Jira System Administrators global permission.
2. Go to > System > Indexing.
3. Select the Full re-index option.
4. Click Re-Index and wait until re-indexing is completed.
5. Take a screenshot of the acknowledgment screen displaying the re-index time and Lucene index timing.
6. Attach the screenshot to your DCHELP ticket.

Jira will be unavailable for some time during the re-indexing process. When finished, the Acknowledge button will be available on the re-indexing page.

7. Setting up an execution environment

For generating performance results suitable for Marketplace approval process use dedicated execution environment. This is a separate AWS EC2 instance to run the toolkit from. Running toolkit from dedicated instance but not from local machine eliminates network fluctuations and guarantees stable CPU and memory performance.

1. Launch AWS EC2 instance. Instance type: c5.2xlarge, OS: select from Quick Start Ubuntu Server 18.04 LTS.

2. Connect to the instance using SSH or the AWS Systems Manager Sessions Manager.

   1

   ssh -i path_to_pem_file ubuntu@INSTANCE_PUBLIC_IP

3. Install Docker. Setup manage Docker as a non-root user.

4. Go to GitHub and create a fork of dc-app-performance-toolkit.
5. Clone the fork locally, then edit the jira.yml configuration file. Set enterprise-scale Jira Data Center parameters:

1
2
3
4
5
6
7
8
9
10
11
12

    application_hostname: test_jira_instance.atlassian.com   # Jira DC hostname without protocol and port e.g. test-jira.atlassian.com or localhost
    application_protocol: http      # http or https
    application_port: 80            # 80, 443, 8080, 2990, etc
    secure: True                    # Set False to allow insecure connections, e.g. when using self-signed SSL certificate
    application_postfix:            # e.g. /jira in case of url like http://localhost:2990/jira
    admin_login: admin
    admin_password: admin
    load_executor: jmeter           # jmeter and locust are supported. jmeter by default.
    concurrency: 200                # number of concurrent virtual users for jmeter or locust scenario
    test_duration: 45m
    ramp-up: 3m                     # time to spin all concurrent users
    total_actions_per_hour: 54500   # number of total JMeter/Locust actions per hour

1. Push your changes to the forked repository.
2. Connect to the AWS EC2 instance and clone forked repository.

You'll need to run the toolkit for each test scenario in the next section.

8. Running the test scenarios from execution environment against enterprise-scale Jira Data Center

Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:

• Performance regression
• Scalability testing

Each scenario will involve multiple test runs. The following subsections explain both in greater detail.

Scenario 1: Performance regression

This scenario helps to identify basic performance issues without a need to spin up a multi-node Jira DC. Make sure the app does not have any performance impact when it is not exercised.

Run 1 (~50 min)

To receive performance baseline results without an app installed:

1. Use SSH to connect to execution environment.

2. Run toolkit with docker:

   1
   2

   cd dc-app-performance-toolkit
   docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt jira.yml

3. View the following main results of the run in the dc-app-performance-toolkit/app/results/jira/YY-MM-DD-hh-mm-ss folder:

   • results_summary.log: detailed run summary
   • results.csv: aggregated .csv file with all actions and timings
   • bzt.log: logs of the Taurus tool execution
   • jmeter.*: logs of the JMeter tool execution
   • pytest.*: logs of Pytest-Selenium execution

Run 2 (~50 min + Lucene Index timing test)

If you are submitting a Jira app, you are required to conduct a Lucene Index timing test. This involves conducting a foreground re-index on a single-node Data Center deployment (with your app installed) and a dataset that has 1M issues.

Benchmark your re-index time with your app installed:

1. Install the app you want to test.
2. Go to > System > Indexing.
3. Select the Full re-index option.
4. Click Re-Index and wait until re-indexing is completed.
5. Take a screenshot of the acknowledgment screen displaying the re-index time and Lucene index timing.
6. Attach the screenshot to your DCHELP ticket.

Performance results generation with the app installed:

1
2

 cd dc-app-performance-toolkit
 docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt jira.yml

Generating a performance regression report

To generate a performance regression report:

1. Use SSH to connect to execution environment.
2. Install the virtualenv as described in dc-app-performance-toolkit/README.md
3. Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
4. Edit the performance_profile.yml file:
   • Under runName: "without app", in the fullPath key, insert the full path to results directory of Run 1.
   • Under runName: "with app", in the fullPath key, insert the full path to results directory of Run 2.

5. Run the following command:

   1

   python csv_chart_generator.py performance_profile.yml

6. In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and performance scenario summary report.

Analyzing report

Once completed, you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see an impact (>20%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.

Scenario 2: Scalability testing

The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.

For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Jira DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Jira DC app in a cluster.

Run 3 (~50 min)

To receive scalability benchmark results for one-node Jira DC with app-specific actions, run bzt:

1. Apply app-specific code changes to a new branch of forked repo.
2. Use SSH to connect to execution environment.
3. Pull cloned fork repo branch with app-specific actions.

4. Run toolkit with docker:

   1
   2

    cd dc-app-performance-toolkit
    docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt jira.yml

Run 4 (~50 min)

To receive scalability benchmark results for two-node Jira DC with app-specific actions:

1. In the AWS console, go to CloudFormation > Stack details > Select your stack.
2. On the Update tab, select Use current template, and then click Next.
3. Enter 2 in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.
4. Click Next > Next > Update stack and wait until stack is updated.

5. Make sure that Jira index successfully synchronized to the second node. To do that, use SSH to connect to the second node via Bastion (where NODE_IP is the IP of the second node):

   1
   2
   3
   4
   5

   ssh-add path_to_your_private_key_pem
   export BASTION_IP=bastion_instance_public_ip
   export NODE_IP=node_private_ip
   export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
   ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@$BASTION_IP" ec2-user@${NODE_IP}

6. Once you're in the second node, download the index-sync.sh file. Then, make it executable and run it:

   1
   2

   wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/jira/index-sync.sh && chmod +x index-sync.sh
   ./index-sync.sh | tee -a index-sync.log

   Index synchronizing time is about 5-10 minutes. When index synchronizing is successfully completed, the following lines will be displayed in console output:

   1
   2
   3
   4
   5

   IndexCopyService] Index restore started. Total 0 issues on instance before loading Snapshot file: IndexSnapshot_10203.tar.sz
   Recovering search indexes - 60% complete... Recovered added and updated issues
   Recovering search indexes - 80% complete... Cleaned removed issues
   Recovering search indexes - 100% complete... Recovered all indexes
   IndexCopyService] Index restore complete. Total N issues on instance

7. Run toolkit with docker:

   1
   2

    cd dc-app-performance-toolkit
    docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt jira.yml

Run 5 (~50 min)

To receive scalability benchmark results for four-node Jira DC with app-specific actions:

1. Scale your Jira Data Center deployment to 3 nodes as described in Run 4.
2. Check Index is synchronized to new nodes the same way as in Run 4.
3. Scale your Jira Data Center deployment to 4 nodes as described in Run 4.
4. Check Index is synchronized to new nodes the same way as in Run 4.

5. Run toolkit with docker:

   1
   2

    cd dc-app-performance-toolkit
    docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt jira.yml

Generating a report for scalability scenario

To generate a scalability report:

1. Use SSH to connect to execution environment.
2. Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
3. Edit the scale_profile.yml file:
   • For runName: "Node 1", in the fullPath key, insert the full path to results directory of Run 3.
   • For runName: "Node 2", in the fullPath key, insert the full path to results directory of Run 4.
   • For runName: "Node 4", in the fullPath key, insert the full path to results directory of Run 5.

4. Run the following command from the virtualenv:

   1

   python csv_chart_generator.py scale_profile.yml

5. In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and summary report.

Analyzing report

Once completed, you will be able to review action timings on Jira Data Center with different numbers of nodes. If you see a significant variation in any action timings between configurations, we recommend taking a look into the app implementation to understand the root cause of this delta.

After completing all your tests, delete your Jira Data Center stacks.

Attaching testing results to DCHELP ticket

1. Use scp command to copy dc-app-performance-toolkit/app/results folder to your local machine.
2. Make sure you have five run results folders and two reports (remove all unsuccessful attempts).
3. Zip dc-app-performance-toolkit/app/results folder and attach archive to DCHELP ticket.

Support

In case of technical questions, issues or problems with DC Apps Performance Toolkit, contact us for support in the community Slack #data-center-app-performance-toolkit channel.