Data Center App Performance Toolkit User Guide For Crowd

This document walks you through the process of testing your app on Crowd using the Data Center App Performance Toolkit. These instructions focus on producing the required performance and scale benchmarks for your Data Center app.

In this document, we cover the use of the Data Center App Performance Toolkit on Enterprise-scale environment.

Enterprise-scale environment: Crowd Data Center environment used to generate Data Center App Performance Toolkit test results for the Marketplace approval process. Preferably, use the AWS Quick Start for Crowd Data Center with the parameters prescribed below. These parameters provision larger, more powerful infrastructure for your Crowd Data Center.

1. Set up an enterprise-scale environment Crowd Data Center on AWS.
2. Load an enterprise-scale dataset on your Crowd Data Center deployment.
3. App-specific actions development.
4. Set up an execution environment for the toolkit.
5. Running the test scenarios from execution environment against enterprise-scale Crowd Data Center.

1. Set up an enterprise-scale environment Crowd Data Center on AWS

We recommend that you use the AWS Quick Start for Crowd Data Center (How to deploy tab) to deploy a Crowd Data Center enterprise-scale environment. This Quick Start will allow you to deploy Crowd Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.

The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Crowd into this new VPC. Deploying Crowd with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.

Using the AWS Quick Start for Crowd

If you are a new user, perform an end-to-end deployment. This involves deploying Crowd into a new ASI:

Navigate to AWS Quick Start for Crowd Data Center > How to deploy tab > Deploy into a new ASI link.

If you have already deployed the ASI separately by using the ASI Quick StartASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, Confluence or Crowd Data Center development environment) with ASI, deploy Crowd into your existing ASI:

Navigate to AWS Quick Start for Crowd Data Center > How to deploy tab > Deploy into your existing ASI link.

To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.

AWS cost estimation

AWS Pricing Calculator provides an estimate of usage charges for AWS services based on certain information you provide. Monthly charges will be based on your actual usage of AWS services and may vary from the estimates the Calculator has provided.

*The prices below are approximate and may vary depending on such factors like region, instance type, deployment type of DB, and other.

Stop cluster nodes

To reduce AWS infrastructure costs you could stop cluster nodes when the cluster is standing idle.
Cluster node might be stopped by using Suspending and Resuming Scaling Processes.

To stop one node within the cluster, follow the instructions below:

1. In the AWS console, go to Services > EC2 > Auto Scaling Groups and open the necessary group to which belongs the node you want to stop.
2. Click Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and add HealthCheck to the Suspended Processes. Amazon EC2 Auto Scaling stops marking instances unhealthy as a result of EC2 and Elastic Load Balancing health checks.
3. Go to EC2 Instances, select instance, click Instance state > Stop instance.

To return node into a working state follow the instructions:

1. Go to EC2 Instances, select instance, click Instance state > Start instance, wait a few minutes for node to become available.
2. Go to EC2 Auto Scaling Groups and open the necessary group to which belongs the node you want to start.
3. Press Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and remove HealthCheck from Suspended Processes of Auto Scaling Group.

Stop database

To reduce AWS infrastructure costs database could be stopped when the cluster is standing idle. Keep in mind that database would be automatically started in 7 days.

To stop database:

1. In the AWS console, go to Services > RDS > Databases.
2. Select cluster database.
3. Click on Actions > Stop.

To start database:

1. In the AWS console, go to Services > RDS > Databases.
2. Select cluster database.
3. Click on Actions > Start.

Quick Start parameters

All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.

Crowd setup

Cluster nodes

Database

Networking (for new ASI)

Networking (for existing ASI)

Running the setup wizard

After successfully deploying Crowd Data Center in AWS, you'll need to configure it:

1. In the AWS console, go to Services > CloudFormation > Stack > Stack details > Select your stack.
2. On the Outputs tab, copy the value of the LoadBalancerURL key.
3. Open LoadBalancerURL in your browser. This will take you to the Crowd setup wizard.
4. On the License page, populate the License Key field by either:
   • Using your existing license, or
   • Generating a Crowd trial license, or
   • Contacting Atlassian to be provided two time-bomb licenses for testing.
     Click Continue.
5. On the Crowd installation page choose New Installation and click Continue.
6. On the Database configuration page, leave all fields default and click Continue.
7. On the Options page, populate the following fields:
   • Deployment title: any instance title
   • Session timeout: 30 (recommended). The number of minutes a session lasts before expiring. Must be greater than 0.
   • Base Url: review and confirm the Crowd instance base url.
     Click Continue.
8. On the Internal directory page, populate the following fields and press Continue:
   • Name: a short, recognisable name that characterises this user directory.
   • Password encryption: chose ATLASSIAN-SECURITY from the dropdown list (recommended)
     Click Continue.
9. On the Default administrator page, fill the following fields:
   • Email Address: email address of the admin user
   • Username: admin (recommended)
   • Password: admin (recommended)
   • Confirm Password: admin (recommended)
   • First name: admin user first name
   • Last name: admin user last name
     Click Continue.
10. On the Integrated applications page leave Open ID server unchecked and click Continue.

2. Preloading your Crowd deployment with an enterprise-scale dataset

Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.

Pre-loading the dataset:

Importing the main dataset. To help you out, we provide an enterprise-scale dataset you can import either via the populate_db.sh script or restore from xml backup file.

The following subsections explain dataset import process in greater detail.

Importing the main dataset

You can load this dataset directly into the database (via a populate_db.sh script), or import it via XML.

Option 1 (recommended): Loading the dataset via populate_db.sh script (~15 minutes)

To populate the database with SQL:

1. In the AWS console, go to Services > EC2 > Instances.
2. On the Description tab, do the following:
   • Copy the Public IP of the Bastion instance.
   • Copy the Private IP of the Crowd node instance.

3. Using SSH, connect to the Crowd node via the Bastion instance:

   For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):

   1
   2
   3
   4
   5
   6

   ssh-add path_to_your_private_key_pem
   export BASTION_IP=bastion_instance_public_ip
   export NODE_IP=node_private_ip
   export SSH_OPTS1='-o ServerAliveInterval=60'
   export SSH_OPTS2='-o ServerAliveCountMax=30'
   ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

   For more information, go to Connecting your nodes over SSH.

4. Download the populate_db.sh script and make it executable:

   1

   wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/crowd/populate_db.sh && chmod +x populate_db.sh

5. Review the following Variables section of the script:

   1
   2
   3
   4

   DB_CONFIG="/usr/lib/systemd/system/crowd.service"
   CROWD_DB_NAME="crowd"
   CROWD_DB_USER="postgres"
   CROWD_DB_PASSWORD="Password1!"

6. Run the script:

   1

   ./populate_db.sh 2>&1 | tee -a populate_db.log

Option 2: Loading the dataset through XML import (~30 minutes)

We recommend that you only use this method if you are having problems with the populate_db.sh script.

1. In the AWS console, go to Services > EC2 > Instances.
2. On the Description tab, do the following:
   • Copy the Public IP of the Bastion instance.
   • Copy the Private IP of the Crowd node instance.

3. Using SSH, connect to the Crowd node via the Bastion instance:

   For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):

   1
   2
   3
   4
   5
   6

   ssh-add path_to_your_private_key_pem
   export BASTION_IP=bastion_instance_public_ip
   export NODE_IP=node_private_ip
   export SSH_OPTS1='-o ServerAliveInterval=60'
   export SSH_OPTS2='-o ServerAliveCountMax=30'
   ssh ${SSH_OPTS1} ${SSH_OPTS2} -o "proxycommand ssh -W %h:%p ${SSH_OPTS1} ${SSH_OPTS2} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

   For more information, go to Connecting your nodes over SSH.

4. Download the db.xml file corresponding to your Crowd version.

   1
   2

   CROWD_VERSION=$(sudo su crowd -c "cat /media/atl/crowd/shared/crowd.version")
   sudo su crowd -c "wget https://centaurus-datasets.s3.amazonaws.com/crowd/${CROWD_VERSION}/large/db.xml -O /media/atl/crowd/shared/db.xml"

5. Log in as a user with the Crowd System Administrators global permission.

6. Go to > Restore. from the menu.
7. Populate the Restore file path field with /media/atl/crowd/shared/db.xml.
8. Click Submit and wait until the import is completed.

3. App-specific actions development

Data Center App Performance Toolkit has its own set of default JMeter test actions for Crowd Data Center.

App-specific action - action (performance test) you have to develop to cover main use cases of your application. Performance test should focus on the common usage of your application and not to cover all possible functionality of your app. For example, application setup screen or other one-time use cases are out of scope of performance testing.

JMeter app-specific actions development

1. Set up local environment for toolkit using the README.
2. Check that crowd.yml file has correct settings of application_hostname, application_protocol, application_port, application_postfix, etc.

3. Navigate to dc-app-performance-toolkit/app folder and run from virtualenv(as described in dc-app-performance-toolkit/README.md):

   python util/jmeter/start_jmeter_ui.py --app crowd

4. Open Crowd thread group and add new transaction controller.

5. Open newly added transaction controller, and add new HTTP requests (based on your app use cases) into it.
6. Run toolkit locally from dc-app-performance-toolkit/app folder with the command
   bzt crowd.yml
   Make sure that execution is successful.

4. Setting up an execution environment

For generating performance results suitable for Marketplace approval process use dedicated execution environment. This is a separate AWS EC2 instance to run the toolkit from. Running the toolkit from a dedicated instance but not from a local machine eliminates network fluctuations and guarantees stable CPU and memory performance.

1. Go to GitHub and create a fork of dc-app-performance-toolkit.

2. Clone the fork locally, then edit the crowd.yml configuration file. Set enterprise-scale Crowd Data Center parameters:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12

    application_hostname: test_crowd_instance.atlassian.com    # Crowd DC hostname without protocol and port e.g. test-crowd.atlassian.com or localhost
    application_protocol: http      # http or https
    application_port: 80            # 80, 443, 8080, 4990, etc
    secure: True                    # Set False to allow insecure connections, e.g. when using self-signed SSL certificate
    application_postfix:            # e.g. /crowd in case of url like http://localhost:4990/crowd
    admin_login: admin
    admin_password: admin
    application_name: crowd
    application_password: 1111
    load_executor: jmeter            
    concurrency: 1000               # number of concurrent threads to authenticate random users
    test_duration: 45m

3. Push your changes to the forked repository.

4. Launch AWS EC2 instance.
   • OS: select from Quick Start Ubuntu Server 18.04 LTS.
   • Instance type: c5.2xlarge
   • Storage size: 30 GiB

5. Connect to the instance using SSH or the AWS Systems Manager Sessions Manager.

   1

   ssh -i path_to_pem_file ubuntu@INSTANCE_PUBLIC_IP

6. Install Docker. Setup manage Docker as a non-root user.

7. Clone forked repository.

You'll need to run the toolkit for each test scenario in the next section.

5. Running the test scenarios from execution environment against enterprise-scale Crowd Data Center

Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:

• Performance regression
• Scalability testing

Each scenario will involve multiple test runs. The following subsections explain both in greater detail.

Scenario 1: Performance regression

This scenario helps to identify basic performance issues without a need to spin up a multi-node Crowd DC. Make sure the app does not have any performance impact when it is not exercised.

Run 1 (~50 min)

To receive performance baseline results without an app installed and without app-specific actions (use code from master branch):

1. Use SSH to connect to execution environment.

2. Run toolkit with docker from the execution environment instance:

   1
   2
   3

   cd dc-app-performance-toolkit
   docker pull atlassian/dcapt
   docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml

3. View the following main results of the run in the dc-app-performance-toolkit/app/results/crowd/YY-MM-DD-hh-mm-ss folder:

   • results_summary.log: detailed run summary
   • results.csv: aggregated .csv file with all actions and timings
   • bzt.log: logs of the Taurus tool execution
   • jmeter.*: logs of the JMeter tool execution

Run 2

Performance results generation with the app installed (still use master branch):

1. Run toolkit with docker from the execution environment instance:

   1
   2
   3

   cd dc-app-performance-toolkit
   docker pull atlassian/dcapt
   docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml

Generating a performance regression report

To generate a performance regression report:

1. Use SSH to connect to execution environment.
2. Install and activate the virtualenv as described in dc-app-performance-toolkit/README.md
3. Allow current user (for execution environment default user is ubuntu) to access Docker generated reports:

   1

   sudo chown -R ubuntu:ubuntu /home/ubuntu/dc-app-performance-toolkit/app/results

4. Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
5. Edit the performance_profile.yml file:
   • Under runName: "without app", in the fullPath key, insert the full path to results directory of Run 1.
   • Under runName: "with app", in the fullPath key, insert the full path to results directory of Run 2.
6. Run the following command:

   1

   python csv_chart_generator.py performance_profile.yml

7. In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and performance scenario summary report.

Analyzing report

Use scp command to copy report artifacts from execution env to local drive:

1. From local machine terminal (Git bash terminal for Windows) run command:

   1
   2

   export EXEC_ENV_PUBLIC_IP=execution_environment_ec2_instance_public_ip
   scp -r -i path_to_exec_env_pem ubuntu@$EXEC_ENV_PUBLIC_IP:/home/ubuntu/dc-app-performance-toolkit/app/results/reports ./reports

2. Once completed, in the ./reports folder you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see an impact (>20%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.

Scenario 2: Scalability testing

The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.

For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Crowd DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Crowd DC app in a cluster.

Run 3 (~50 min)

To receive scalability benchmark results for one-node Crowd DC with app-specific actions:

1. Apply app-specific code changes to a new branch of forked repo.
2. Use SSH to connect to execution environment.
3. Pull cloned fork repo branch with app-specific actions.

4. Run toolkit with docker from the execution environment instance:

   1
   2
   3

   cd dc-app-performance-toolkit
   docker pull atlassian/dcapt
   docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml

Run 4 (~50 min)

To receive scalability benchmark results for two-node Crowd DC with app-specific actions:

1. In the AWS console, go to CloudFormation > Stack details > Select your stack.
2. On the Update tab, select Use current template, and then click Next.
3. Enter 2 in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.
4. Click Next > Next > Update stack and wait until stack is updated.

5. Edit run parameters for 2 nodes run. To do it, left uncommented only 2 nodes scenario parameters in crowd.yml file.

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11

   # 1 node scenario parameters
   # ramp-up: 20s                    # time to spin all concurrent threads
   # total_actions_per_hour: 180000  # number of total JMeter actions per hour
   
   # 2 nodes scenario parameters
     ramp-up: 10s                    # time to spin all concurrent threads
     total_actions_per_hour: 360000  # number of total JMeter actions per hour
   
   # 4 nodes scenario parameters
   # ramp-up: 5s                     # time to spin all concurrent threads
   # total_actions_per_hour: 720000  # number of total JMeter actions per hour

6. Run toolkit with docker from the execution environment instance:

   1
   2
   3

   cd dc-app-performance-toolkit
   docker pull atlassian/dcapt
   docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml

Run 5 (~50 min)

To receive scalability benchmark results for four-node Crowd DC with app-specific actions:

1. Scale your Crowd Data Center deployment to 4 nodes as described in Run 4.

2. Edit run parameters for 4 nodes run. To do it, left uncommented only 4 nodes scenario parameters crowd.yml file.

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11

   # 1 node scenario parameters
   # ramp-up: 20s                    # time to spin all concurrent threads
   # total_actions_per_hour: 180000  # number of total JMeter actions per hour
   
   # 2 nodes scenario parameters
   # ramp-up: 10s                    # time to spin all concurrent threads
   # total_actions_per_hour: 360000  # number of total JMeter actions per hour
   
   # 4 nodes scenario parameters
   ramp-up: 5s                     # time to spin all concurrent threads
   total_actions_per_hour: 720000  # number of total JMeter actions per hour

3. Run toolkit with docker from the execution environment instance:

   1
   2
   3

   cd dc-app-performance-toolkit
   docker pull atlassian/dcapt
   docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt crowd.yml

Generating a report for scalability scenario

To generate a scalability report:

1. Use SSH to connect to execution environment.
2. Allow current user (for execution environment default user is ubuntu) to access Docker generated reports:

   1

   sudo chown -R ubuntu:ubuntu /home/ubuntu/dc-app-performance-toolkit/app/results

3. Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
4. Edit the scale_profile.yml file:
   • For runName: "Node 1", in the fullPath key, insert the full path to results directory of Run 3.
   • For runName: "Node 2", in the fullPath key, insert the full path to results directory of Run 4.
   • For runName: "Node 4", in the fullPath key, insert the full path to results directory of Run 5.
5. Run the following command from the activated virtualenv (as described in dc-app-performance-toolkit/README.md):

   1

   python csv_chart_generator.py scale_profile.yml

6. In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and summary report.

Analyzing report

Use scp command to copy report artifacts from execution env to local drive:

1. From local terminal (Git bash terminal for Windows) run command:

   1
   2

   export EXEC_ENV_PUBLIC_IP=execution_environment_ec2_instance_public_ip
   scp -r -i path_to_exec_env_pem ubuntu@$EXEC_ENV_PUBLIC_IP:/home/ubuntu/dc-app-performance-toolkit/app/results/reports ./reports

2. Once completed, in the ./reports folder you will be able to review action timings on Crowd Data Center with different numbers of nodes. If you see a significant variation in any action timings between configurations, we recommend taking a look into the app implementation to understand the root cause of this delta.

Attaching testing results to DCHELP ticket

1. Make sure you have two reports folders: one with performance profile and second with scale profile results. Each folder should have profile.csv, profile.png, profile_summary.log and profile run result archives.
2. Attach two reports folders to your DCHELP ticket.

Support

In case of technical questions, issues or problems with DC Apps Performance Toolkit, contact us for support in the community Slack #data-center-app-performance-toolkit channel.