Rate this page:
This document walks you through the process of testing your app on Crowd using the Data Center App Performance Toolkit. These instructions focus on producing the required performance and scale benchmarks for your Data Center app.
In this document, we cover the use of the Data Center App Performance Toolkit on Enterprise-scale environment.
Enterprise-scale environment: Crowd Data Center environment used to generate Data Center App Performance Toolkit test results for the Marketplace approval process. Preferably, use the AWS Quick Start for Crowd Data Center with the parameters prescribed below. These parameters provision larger, more powerful infrastructure for your Crowd Data Center.
We recommend that you use the AWS Quick Start for Crowd Data Center (How to deploy tab) to deploy a Crowd Data Center enterprise-scale environment. This Quick Start will allow you to deploy Crowd Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.
The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Crowd into this new VPC. Deploying Crowd with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.
If you are a new user, perform an end-to-end deployment. This involves deploying Crowd into a new ASI:
Navigate to AWS Quick Start for Crowd Data Center > How to deploy tab > Deploy into a new ASI link.
If you have already deployed the ASI separately by using the ASI Quick StartASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, Confluence or Crowd Data Center development environment) with ASI, deploy Crowd into your existing ASI:
Navigate to AWS Quick Start for Crowd Data Center > How to deploy tab > Deploy into your existing ASI link.
You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional price for using this Quick Start. For more information, go to aws.amazon.com/pricing.
To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.
AWS Pricing Calculator provides an estimate of usage charges for AWS services based on certain information you provide. Monthly charges will be based on your actual usage of AWS services and may vary from the estimates the Calculator has provided.
*The prices below are approximate and may vary depending on such factors like region, instance type, deployment type of DB, and other.
Stack | Estimated hourly cost ($) |
---|---|
One Node Crowd DC | 0.4 - 0.6 |
Two Nodes Crowd DC | 0.6 - 0.8 |
Four Nodes Crowd DC | 0.9 - 1.4 |
To reduce AWS infrastructure costs you could stop cluster nodes when the cluster is standing idle.
Cluster node might be stopped by using Suspending and Resuming Scaling Processes.
To stop one node within the cluster, follow the instructions below:
Edit
on Advanced configuration
) and add HealthCheck
to the Suspended Processes
. Amazon EC2 Auto Scaling stops marking instances unhealthy as a result of EC2 and Elastic Load Balancing health checks.To return node into a working state follow the instructions:
Edit
on Advanced configuration
) and remove HealthCheck
from Suspended Processes
of Auto Scaling Group.To reduce AWS infrastructure costs database could be stopped when the cluster is standing idle. Keep in mind that database would be automatically started in 7 days.
To stop database:
To start database:
All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.
Crowd setup
Parameter | Recommended Value |
---|---|
Version | The Data Center App Performance Toolkit officially supports 5.0.2 |
Cluster nodes
Parameter | Recommended Value |
---|---|
Cluster node instance type | c5.xlarge |
Maximum number of cluster nodes | 1 |
Minimum number of cluster nodes | 1 |
Cluster node instance volume size | 100 |
Database
Parameter | Recommended Value |
---|---|
The database engine to deploy with | PostgresSQL |
The database engine version to use | 11 |
Database instance class | db.m5.large |
RDS Provisioned IOPS | 1000 |
Master (admin) password | Password1! |
Enable RDS Multi-AZ deployment | false |
Application user database password | Password1! |
Database storage | 200 |
The Master (admin) password will be used later when restoring the SQL database dataset. If password value is not set to default, you'll need to change DB_PASS
value manually in the restore database dump script (later in Preloading your Crowd deployment with an enterprise-scale dataset).
Networking (for new ASI)
Parameter | Recommended Value |
---|---|
Trusted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Availability Zones | Select two availability zones in your region |
Permitted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Make instance internet facing | true |
Key Name | The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info. |
Networking (for existing ASI)
Parameter | Recommended Value |
---|---|
Make instance internet facing | true |
Permitted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Key Name | The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info. |
After successfully deploying Crowd Data Center in AWS, you'll need to configure it:
Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.
Data dimensions | Value for an enterprise-scale dataset |
---|---|
Users | ~100 000 |
Groups | ~15 |
All the datasets use the standard admin
/admin
credentials.
Importing the main dataset. To help you out, we provide an enterprise-scale dataset you can import either via the populate_db.sh script or restore from xml backup file.
The following subsections explain dataset import process in greater detail.
You can load this dataset directly into the database (via a populate_db.sh script), or import it via XML.
To populate the database with SQL:
In the AWS console, go to Services > EC2 > Instances.
On the Description tab, do the following:
Using SSH, connect to the Crowd node via the Bastion instance:
For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):
1 2ssh-add path_to_your_private_key_pem export BASTION_IP=bastion_instance_public_ip export NODE_IP=node_private_ip export SSH_OPTS1='-o ServerAliveInterval=60' export SSH_OPTS2='-o ServerAliveCountMax=30' ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Download the populate_db.sh script and make it executable:
1 2wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/crowd/populate_db.sh && chmod +x populate_db.sh
Review the following Variables section
of the script:
1 2DB_CONFIG="/usr/lib/systemd/system/crowd.service" CROWD_DB_NAME="crowd" CROWD_DB_USER="postgres" CROWD_DB_PASSWORD="Password1!"
Run the script:
1 2./populate_db.sh 2>&1 | tee -a populate_db.log
Do not close or interrupt the session. It will take about an hour to restore SQL database. When SQL restoring is finished, an admin user will have admin
/admin
credentials.
In case of a failure, check the Variables
section and run the script one more time.
We recommend that you only use this method if you are having problems with the populate_db.sh script.
In the AWS console, go to Services > EC2 > Instances.
On the Description tab, do the following:
Using SSH, connect to the Crowd node via the Bastion instance:
For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):
1 2ssh-add path_to_your_private_key_pem export BASTION_IP=bastion_instance_public_ip export NODE_IP=node_private_ip export SSH_OPTS1='-o ServerAliveInterval=60' export SSH_OPTS2='-o ServerAliveCountMax=30' ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Download the db.xml file corresponding to your Crowd version.
1 2CROWD_VERSION=$(sudo su crowd -c "cat /media/atl/crowd/shared/crowd.version") sudo su crowd -c "wget https://centaurus-datasets.s3.amazonaws.com/crowd/${CROWD_VERSION}/large/db.xml -O /media/atl/crowd/shared/db.xml"
Log in as a user with the Crowd System Administrators global permission.
Go to > Restore. from the menu.
Populate the Restore file path field with /media/atl/crowd/shared/db.xml
.
Click Submit and wait until the import is completed.
After Preloading your Crowd deployment with an enterprise-scale dataset, the admin user will have admin
/admin
credentials.
It's recommended to change default password from UI account page for security reasons.
Data Center App Performance Toolkit has its own set of default JMeter test actions for Crowd Data Center.
App-specific action - action (performance test) you have to develop to cover main use cases of your application. Performance test should focus on the common usage of your application and not to cover all possible functionality of your app. For example, application setup screen or other one-time use cases are out of scope of performance testing.
JMeter app-specific actions development
Set up local environment for toolkit using the README.
Check that crowd.yml
file has correct settings of application_hostname
, application_protocol
, application_port
, application_postfix
, etc.
Navigate to dc-app-performance-toolkit/app
folder and run from virtualenv(as described in dc-app-performance-toolkit/README.md
):
python util/jmeter/start_jmeter_ui.py --app crowd
Open Crowd
thread group and add new transaction controller.
Open newly added transaction controller, and add new HTTP requests (based on your app use cases) into it.
Run toolkit locally from dc-app-performance-toolkit/app
folder with the command
bzt crowd.yml
Make sure that execution is successful.
For generating performance results suitable for Marketplace approval process use dedicated execution environment. This is a separate AWS EC2 instance to run the toolkit from. Running the toolkit from a dedicated instance but not from a local machine eliminates network fluctuations and guarantees stable CPU and memory performance.
crowd.yml
configuration file. Set enterprise-scale Crowd Data Center parameters:Do not push to the fork real application_hostname
, admin_login
and admin_password
values for security reasons.
Instead, set those values directly in .yml
file on execution environment instance.
1 2application_hostname: test_crowd_instance.atlassian.com # Crowd DC hostname without protocol and port e.g. test-crowd.atlassian.com or localhost application_protocol: http # http or https application_port: 80 # 80, 443, 8080, 4990, etc secure: True # Set False to allow insecure connections, e.g. when using self-signed SSL certificate application_postfix: # e.g. /crowd in case of url like http://localhost:4990/crowd admin_login: admin admin_password: admin application_name: crowd application_password: 1111 load_executor: jmeter concurrency: 1000 # number of concurrent threads to authenticate random users test_duration: 45m
Push your changes to the forked repository.
Ubuntu Server 20.04 LTS
.c5.2xlarge
30
GiBConnect to the instance using SSH or the AWS Systems Manager Sessions Manager.
1 2ssh -i path_to_pem_file ubuntu@INSTANCE_PUBLIC_IP
Install Docker. Setup manage Docker as a non-root user.
Clone forked repository.
You'll need to run the toolkit for each test scenario in the next section.
Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:
Each scenario will involve multiple test runs. The following subsections explain both in greater detail.
This scenario helps to identify basic performance issues without a need to spin up a multi-node Crowd DC. Make sure the app does not have any performance impact when it is not exercised.
To receive performance baseline results without an app installed and without app-specific actions (use code from master
branch):
Use SSH to connect to execution environment.
Run toolkit with docker from the execution environment instance:
1 2cd dc-app-performance-toolkit docker pull atlassian/dcapt docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml
View the following main results of the run in the dc-app-performance-toolkit/app/results/crowd/YY-MM-DD-hh-mm-ss
folder:
results_summary.log
: detailed run summaryresults.csv
: aggregated .csv file with all actions and timingsbzt.log
: logs of the Taurus tool executionjmeter.*
: logs of the JMeter tool executionReview results_summary.log
file under artifacts dir location. Make sure that overall status is OK
before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above.
Performance results generation with the app installed (still use master branch):
Run toolkit with docker from the execution environment instance:
1 2cd dc-app-performance-toolkit docker pull atlassian/dcapt docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml
Review results_summary.log
file under artifacts dir location. Make sure that overall status is OK
before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above.
To generate a performance regression report:
virtualenv
as described in dc-app-performance-toolkit/README.md
ubuntu
) to access Docker generated reports:
1 2sudo chown -R ubuntu:ubuntu /home/ubuntu/dc-app-performance-toolkit/app/results
dc-app-performance-toolkit/app/reports_generation
folder.performance_profile.yml
file:
1 2python csv_chart_generator.py performance_profile.yml
dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss
folder, view the .csv
file (with consolidated scenario results), the .png
chart file and performance scenario summary report.Use scp command to copy report artifacts from execution env to local drive:
1 2export EXEC_ENV_PUBLIC_IP=execution_environment_ec2_instance_public_ip scp -r -i path_to_exec_env_pem ubuntu@$EXEC_ENV_PUBLIC_IP:/home/ubuntu/dc-app-performance-toolkit/app/results/reports ./reports
./reports
folder you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see an impact (>20%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.
For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Crowd DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Crowd DC app in a cluster.
To receive scalability benchmark results for one-node Crowd DC with app-specific actions:
Apply app-specific code changes to a new branch of forked repo.
Use SSH to connect to execution environment.
Pull cloned fork repo branch with app-specific actions.
Run toolkit with docker from the execution environment instance:
1 2cd dc-app-performance-toolkit docker pull atlassian/dcapt docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml
Review results_summary.log
file under artifacts dir location. Make sure that overall status is OK
before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above.
Before scaling your DC make sure that AWS vCPU limit is not lower than needed number. Use vCPU limits calculator to see current limit. The same article has instructions on how to increase limit if needed.
To receive scalability benchmark results for two-node Crowd DC with app-specific actions:
2
in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.In case if you got error during update - BastionPrivIp cannot be updated
.
Please use those steps for a workaround:
2
in the Desired capacity, Minimum capacity and Maximum capacity fields.Edit run parameters for 2 nodes run. To do it, left uncommented only 2 nodes scenario parameters in crowd.yml
file.
1 2# 1 node scenario parameters # ramp-up: 20s # time to spin all concurrent threads # total_actions_per_hour: 180000 # number of total JMeter actions per hour # 2 nodes scenario parameters ramp-up: 10s # time to spin all concurrent threads total_actions_per_hour: 360000 # number of total JMeter actions per hour # 4 nodes scenario parameters # ramp-up: 5s # time to spin all concurrent threads # total_actions_per_hour: 720000 # number of total JMeter actions per hour
Run toolkit with docker from the execution environment instance:
1 2cd dc-app-performance-toolkit docker pull atlassian/dcapt docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml
Review results_summary.log
file under artifacts dir location. Make sure that overall status is OK
before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above.
Before scaling your DC make sure that AWS vCPU limit is not lower than needed number. Use vCPU limits calculator to see current limit. The same article has instructions on how to increase limit if needed.
To receive scalability benchmark results for four-node Crowd DC with app-specific actions:
Scale your Crowd Data Center deployment to 4 nodes as described in Run 4.
Edit run parameters for 4 nodes run. To do it, left uncommented only 4 nodes scenario parameters crowd.yml
file.
1 2# 1 node scenario parameters # ramp-up: 20s # time to spin all concurrent threads # total_actions_per_hour: 180000 # number of total JMeter actions per hour # 2 nodes scenario parameters # ramp-up: 10s # time to spin all concurrent threads # total_actions_per_hour: 360000 # number of total JMeter actions per hour # 4 nodes scenario parameters ramp-up: 5s # time to spin all concurrent threads total_actions_per_hour: 720000 # number of total JMeter actions per hour
Run toolkit with docker from the execution environment instance:
1 2cd dc-app-performance-toolkit docker pull atlassian/dcapt docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml
Review results_summary.log
file under artifacts dir location. Make sure that overall status is OK
before moving to the next steps. For an enterprise-scale environment run, the acceptable success rate for actions is 95% and above.
To generate a scalability report:
ubuntu
) to access Docker generated reports:
1 2sudo chown -R ubuntu:ubuntu /home/ubuntu/dc-app-performance-toolkit/app/results
dc-app-performance-toolkit/app/reports_generation
folder.scale_profile.yml
file:
virtualenv
(as described in dc-app-performance-toolkit/README.md
):
1 2python csv_chart_generator.py scale_profile.yml
dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss
folder, view the .csv
file (with consolidated scenario results), the .png
chart file and summary report.Use scp command to copy report artifacts from execution env to local drive:
1 2export EXEC_ENV_PUBLIC_IP=execution_environment_ec2_instance_public_ip scp -r -i path_to_exec_env_pem ubuntu@$EXEC_ENV_PUBLIC_IP:/home/ubuntu/dc-app-performance-toolkit/app/results/reports ./reports
./reports
folder you will be able to review action timings on Crowd Data Center with different numbers of nodes. If you see a significant variation in any action timings between configurations, we recommend taking a look into the app implementation to understand the root cause of this delta.After completing all your tests, delete your Crowd Data Center stacks.
Do not forget to attach performance testing results to your ECOHELP ticket.
profile.csv
, profile.png
, profile_summary.log
and profile run result archives. Archives
should contain all raw data created during the run: bzt.log
, selenium/jmeter/locust logs, .csv and .yml files, etc.In case of technical questions, issues or problems with DC Apps Performance Toolkit, contact us for support in the community Slack #data-center-app-performance-toolkit channel.
Rate this page: