To use the Data Center App Performance Toolkit, you'll need to first clone its repo.
1
git clone git@github.com:atlassian/dc-app-performance-toolkit.git
Follow installation instructions described in the dc-app-performance-toolkit/README.md
file.
If you need performance testing results at a production level, follow instructions in this chapter to set up Jira Data Center with the corresponding dataset.
For spiking, testing, or developing, your local Jira instance would work well. Thus, you can skip this chapter and proceed with Testing scenarios. Still, script adjustments for your local dataset may be required.
We recommend that you use the AWS Quick Start for Jira Data Center to deploy a Jira Data Center testing environment. This Quick Start will allow you to deploy Jira Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.
The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Jira into this new VPC. Deploying Jira with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.
If you are a new user, perform an end-to-end deployment. This involves deploying Jira into a new ASI.
If you have already deployed the ASI separately by using the ASI Quick StartASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center), deploy Jira into your existing ASI.
To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.
All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.
Jira setup
Parameter | Recommended Value |
---|---|
Jira Product | Software |
Jira Version | 8.0.3 or 7.13.6 or 8.5.0 |
The Data Center App Performance Toolkit officially supports:
Cluster nodes
Parameter | Recommended Value |
---|---|
Cluster node instance type | c5.4xlarge |
Maximum number of cluster nodes | 1 |
Minimum number of cluster nodes | 1 |
We recommend c5.4xlarge to strike the balance between cost and hardware we see in the field for our enterprise customers. This differs from our public recommendation on c4.8xlarge for production instances but is representative for a lot of our Jira Data Center customers.
The Data Center App Performance Toolkit framework is also set up for concurrency we expect on this instance size. As such, underprovisioning will likely show a larger performance impact than expected.
Database
Parameter | Recommended Value |
---|---|
Database instance class | db.m4.large |
RDS Provisioned IOPS | 1000 |
Master (admin) password | Password1! |
Enable RDS Multi-AZ deployment | true |
Application user database password | Password1! |
Networking (for new ASI)
Parameter | Recommended Value |
---|---|
Trusted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Availability Zones | Select two availability zones in your region. Both zones must support EFS (see Supported AWS regions for details). |
Permitted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Make instance internet facing | true |
Key Name | The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info. |
Networking (for existing ASI)
Parameter | Recommended Value |
---|---|
Make instance internet facing | true |
Permitted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Key Name | The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info. |
After successfully deploying Jira Data Center in AWS, you'll need to configure it:
Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.
Data dimensions | Value for an enterprise-scale dataset |
---|---|
Attachments | ~2 000 000 |
Comments | ~6 000 000 |
Components | ~2 500 |
Custom fields | ~800 |
Groups | ~1 000 |
Issue security levels | 10 |
Issue types | ~300 |
Issues | ~1 000 000 |
Priorities | 5 |
Projects | 500 |
Resolutions | 34 |
Screen schemes | ~200 |
Screens | ~200 |
Statuses | ~400 |
Users | ~21 000 |
Versions | ~20 000 |
Workflows | 50 |
Pre-loading the dataset is a three-step process:
The following subsections explain each step in greater detail.
You can load this dataset directly into the database (via a populate_db.sh script), or import it via XML.
To populate the database with SQL:
Using SSH, connect to the Jira node via the Bastion instance:
For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS:
1 2 3 4
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Download the populate_db.sh script and make it executable:
1
wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/jira/populate_db.sh && chmod +x populate_db.sh
Review the following Variables section
of the script:
1 2 3 4 5 6 7 8 9 10
INSTALL_PSQL_CMD="amazon-linux-extras install -y postgresql10"
DB_CONFIG="/var/atlassian/application-data/jira/dbconfig.xml"
JIRA_CURRENT_DIR="/opt/atlassian/jira-software/current"
CATALINA_PID_FILE="${JIRA_CURRENT_DIR}/work/catalina.pid"
JIRA_DB_NAME="jira"
JIRA_DB_USER="postgres"
JIRA_DB_PASS="Password1!"
JIRA_SETENV_FILE="${JIRA_CURRENT_DIR}/bin/setenv.sh"
JIRA_VERSION_FILE="/media/atl/jira/shared/jira-software.version"
DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
Run the script:
1
./populate_db.sh | tee -a populate_db.log
We recommend that you only use this method if you are having problems with the populate_db.sh script.
Using SSH, connect to the Jira node via the Bastion instance:
For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS:
1 2 3 4
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" eec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Download the xml_backup.zip file corresponding to your Jira version.
1 2
JIRA_VERSION=$(sudo su jira -c "cat /media/atl/jira/shared/jira-software.version")
sudo su jira -c "wget https://centaurus-datasets.s3.amazonaws.com/jira/${JIRA_VERSION}/large/xml_backup.zip -O /media/atl/jira/shared/import/xml_backup.zip"
From a different computer, log in as a user with the Jira System Administrators global permission.
xml_backup.zip
.After Importing the main dataset, you'll now have to pre-load an enterprise-scale set of attachments.
Using SSH, connect to the Jira node via the Bastion instance:
For Windows, use Putty to connect to the Jira node over SSH. For Linux or MacOS:
1 2 3 4
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Download the upload_attachments.sh script and make it executable:
1
wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/jira/upload_attachments.sh && chmod +x upload_attachments.sh
Review the following Variables section
of the script:
1 2 3 4 5
DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/jira"
ATTACHMENTS_TAR="attachments.tar.gz"
ATTACHMENTS_DIR="attachments"
TMP_DIR="/tmp"
EFS_DIR="/media/atl/jira/shared/data"
Run the script:
1
./upload_attachments.sh | tee -a upload_attachments.log
For more information, go to Re-indexing Jira.
Jira will be unavailable for some time during the re-indexing process. When finished, the Acknowledge button will be available on the re-indexing page.
Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:
Each scenario will involve multiple test runs. The following subsections explain both in greater detail.
This scenario helps to identify basic performance issues without a need to spin up a multi-node Jira DC. Make sure the app does not have any performance impact when it is not exercised.
To receive performance baseline results without an app installed:
dc-app-performance-toolkit/app folder
.jira.yml
file and fill in the following variables:application_hostname
: your_dc_jira_instance_hostname without protocolapplication_protocol
: HTTP or HTTPSapplication_port
: for HTTP - 80, for HTTPS - 443, or your instance-specific port. The self-signed certificate is not supported.admin_login
: admin user usernameadmin_password
: admin user passwordconcurrency
: number of concurrent users for JMeter scenario - 200 by defaulttest_duration
: duration of the performance run - 45min by defaultRun bzt.
1
bzt jira.yml
View the following main results of the run in the dc-app-performance-toolkit/app/results/jira/YY-MM-DD-hh-mm-ss
folder:
results.csv
: aggregated .csv file with all actions and timingsbzt.log
: logs of the Taurus tool executionjmeter.*
: logs of the JMeter tool executionpytest.*
: logs of Pytest-Selenium executionTo receive performance results with an app installed:
Run bzt.
1
bzt jira.yml
To generate a performance regression report:
dc-app-performance-toolkit/app/reports_generation
folder.performance_profile.yml
file:Run the following command:
1
python csv_chart_generator.py performance_profile.yml
In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss
folder, view the .csv
file (with consolidated scenario results) and the .png
file.
Once completed, you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see a significant impact (>10%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.
The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.
For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Jira DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Jira DC app in a cluster.
Extension scripts, which extend the base JMeter (jira.jmx
) and Selenium (jira-ui.py
) scripts, are located in a separate folder (dc-app-performance-toolkit/extension/jira
). You can modify these scripts to include their app-specific actions.
JMeter is written in XML and requires JMeter GUI to view and make changes. You can launch JMeter GUI by running the ~/.bzt/jmeter-taurus/<jmeter_version>/bin/jmeter
command.
Make sure you run this command inside the dc-app-performance-toolkit/app directory
. The main jmeter/jira.jmx
file contains relative paths to other scripts and will throw errors if run and loaded elsewhere.
Here's a snippet of the base JMeter script (jira.jmx
):
For every base action, there is an extension script executed after the base script. In most cases, you should modify only the extension.jmx
file. For example, if there are additional REST APIs introduced as part of viewing an issue, you can include these calls in the extension.jmx
file under the view issue transaction.
Here's a snippet of the extension JMeter script (extension.jmx
).
This ensures that these APIs are called as part of the view issue transaction with minimal intrusion (for example, no additional logins). For a fairer comparison, you have to keep the same number of base transactions before and after the plugin is installed.
When debugging, if you want to only test transactions in the extend_view_issue
action, you can comment out other transactions in the jira.yml
config file and set the percentage of the base execution to 100. Alternatively, you can change percentages of others to 0.
1 2 3 4 5
# perc_create_issue: 4
# perc_search_jql: 16
perc_view_issue: 100
# perc_view_project_summary: 4
# perc_view_dashboard: 8
You can run your script independently of the base action under a specific workload if, for example, your plugin introduces a separate URL and has no correlation to the base transactions.
In such a case, you extend the extend_standalone_extension
controller, which is also located in the extension.jmx
file. With this option, you can define the execution percentage by the perc_standalone_extension
parameter in the jira.yml
config file.
The following configuration ensures that extend_standalone_extension controller is executed 10% of the total transactions.
1
perc_standalone_extension: 10
Use or access the following variables of the extension script from the base script. They can also be inherited.
${issue_key}
- issue key being viewed or modified (e.g. ABC-123)${issue_id}
- issue id being viewed or modified (e.g. 693484)${project_key}
- project key being viewed or modified (e.g. ABC)${project_id}
- project id being viewed or modified (e.g. 3423)${scrum_board_id}
- scrum board id being viewed (e.g. 328)${kanban_board_id}
- kanban board id being viewed (e.g. 100)${jql}
- jql query being used (e.g. text ~ "qrk*" order by key)${username}
- the logged in username (e.g. admin)In addition to JMeter, you can extend Selenium scripts to measure the end-to-end browser timings.
We use Pytest to drive Selenium tests. The jira-ui.py
executor script is located in the app/selenium_ui/
folder. This file contains all browser actions, defined by the test_ functions
. These actions are executed one by one during the testing.
In the jira-ui.py
script, view the following block of code:
1 2
# def test_1_selenium_custom_action(webdriver, datasets, screen_shots):
# custom_action(webdriver, datasets)
This is a placeholder to add an extension action. The custom action can be moved to a different line, depending on the required workflow, as long as it is between the login (test_0_selenium_a_login
) and logout (test_2_selenium_z_log_out
) actions.
To implement the custom_action function, modify the extension_ui.py
file in the extension/jira/
directory. The following is an example of the custom_action
function, where Selenium navigates to a URL, clicks on an element, and waits until an element is visible:
1 2 3 4 5 6 7 8
def custom_action(webdriver, datasets):
@print_timing
def measure(webdriver, interaction):
@print_timing
def measure(webdriver, interaction):
webdriver.get(f'{APPLICATION_URL}/plugins/servlet/some-app/reporter')
WebDriverWait(webdriver, timeout).until(EC.visibility_of_element_located((By.ID, 'plugin-element')))
measure(webdriver, 'selenium_app_custom_action:view_report')
To view more examples, see the modules.py
file in the selenium_ui/jira
directory.
To ensure that the test runs without errors in parallel, run your extension scripts with the base scripts as a sanity check.
To receive scalability benchmark results for one-node Jira DC with app-specific actions, run bzt
:
1
bzt jira.yml
To receive scalability benchmark results for two-node Jira DC with app-specific actions:
2
in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.Make sure that Jira index successfully synchronized to the second node. To do that, use SSH to connect to the second node via Bastion (where NODE_IP
is the IP of the second node):
1 2 3 4
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
ssh -o "proxycommand ssh -W %h:%p ec2-user@$BASTION_IP" ec2-user@${NODE_IP}
Once you're in the second node, download the index-sync.sh file. Then, make it executable and run it:
1 2
wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/jira/index-sync.sh && chmod +x index-sync.sh
./index-sync.sh | tee -a index-sync.log
Index synchronizing time is about 5-10 minutes. When index synchronizing is successfully completed, the following lines will be displayed in console output:
1 2 3 4 5
IndexCopyService] Index restore started. Total 0 issues on instance before loading Snapshot file: IndexSnapshot_10203.tar.sz
Recovering search indexes - 60% complete... Recovered added and updated issues
Recovering search indexes - 80% complete... Cleaned removed issues
Recovering search indexes - 100% complete... Recovered all indexes
IndexCopyService] Index restore complete. Total N issues on instance
Run bzt.
1
bzt jira.yml
To receive scalability benchmark results for four-node Jira DC with app-specific actions:
Run bzt.
1
bzt jira.yml
To generate a scalability report:
dc-app-performance-toolkit/app/reports_generation
folder.scale_profile.yml
file:Run the following command:
1
python csv_chart_generator.py scale_profile.yml
In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss
folder, view the .csv
file (with consolidated scenario results) and the .png
file.
Once completed, you will be able to review action timings on Jira Data Center with different numbers of nodes. If you see a significant variation in any action timings between configurations, we recommend taking a look into the app implementation to understand the root cause of this delta.
After completing all your tests, delete your Jira Data Center stacks.