Rate this page:
This document walks you through the process of testing your app on Bitbucket using the Data Center App Performance Toolkit. These instructions focus on producing the required performance and scale benchmarks for your Data Center app.
To use the Data Center App Performance Toolkit, you'll need to:
We recommend that you use the AWS Quick Start for Bitbucket Data Center to deploy a Bitbucket Data Center testing environment. This Quick Start will allow you to deploy Bitbucket Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.
The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Bitbucket into this new VPC. Deploying Bitbucket with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.
If you are a new user, perform an end-to-end deployment. This involves deploying Bitbucket into a new ASI.
If you have already deployed the ASI separately by using the ASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center), deploy Bitbucket into your existing ASI.
To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.
AWS Pricing Calculator provides an estimate of usage charges for AWS services based on certain information you provide. Monthly charges will be based on your actual usage of AWS services, and may vary from the estimates the Calculator has provided.
*The prices below are approximate and may vary depending on factors such as (region, instance type, deployment type of DB, etc.)
Stack | Estimated hourly cost ($) |
---|---|
One Node Bitbucket DC | 1.4 - 2.0 |
Two Nodes Bitbucket DC | 1.7 - 2.5 |
Four Nodes Bitbucket DC | 2.4 - 3.6 |
To reduce AWS infrastructure costs you could stop cluster nodes when the cluster is standing idle.
Cluster node might be stopped by using Suspending and Resuming Scaling Processes.
To stop one node within the cluster, follow the instructions below:
Edit
on Advanced configuration
) and add HealthCheck
to the Suspended Processes
. Amazon EC2 Auto Scaling stops marking instances unhealthy as a result of EC2 and Elastic Load Balancing health checks.To return node into a working state follow the instructions:
Edit
on Advanced configuration
) and remove HealthCheck
from Suspended Processes
of Auto Scaling Group.To reduce AWS infrastructure costs database could be stopped when the cluster is standing idle. Keep in mind that database would be automatically started in 7 days.
To stop database:
To start database:
All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.
Bitbucket setup
Parameter | Recommended Value |
---|---|
Version | The Data Center App Performance Toolkit officially supports 7.6.2 , 6.10.7 (Long Term Support releases) and 7.0.5 Platform Release |
Cluster nodes
Parameter | Recommended Value |
---|---|
Bitbucket cluster node instance type | c5.2xlarge |
Maximum number of cluster nodes | 1 |
Minimum number of cluster nodes | 1 |
Cluster node instance volume size | 50 |
We recommend c5.2xlarge to strike the balance between cost and hardware we see in the field for our enterprise customers. More info could be found in public recommendations.
The Data Center App Performance Toolkit framework is also set up for concurrency we expect on this instance size. As such, underprovisioning will likely show a larger performance impact than expected.
File server
Parameter | Recommended Value |
---|---|
File server instance type | m4.xlarge |
Home directory size | 1000 |
Database
Parameter | Recommended Value |
---|---|
Database instance class | db.m4.large |
RDS Provisioned IOPS | 1000 |
Master password | Password1! |
Enable RDS Multi-AZ deployment | false |
Bitbucket database password | Password1! |
Database storage | 100 |
Elasticsearch
Parameter | Recommended Value |
---|---|
Elasticsearch instance type | m4.xlarge.elasticsearch |
Elasticsearch disk-space per node (GB) | 1000 |
Networking (for new ASI)
Parameter | Recommended Value |
---|---|
Trusted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Availability Zones | Select two availability zones in your region |
Permitted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Make instance internet facing | true |
Key Name | The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info. |
Networking (for existing ASI)
Parameter | Recommended Value |
---|---|
Make instance internet facing | true |
Permitted IP range | 0.0.0.0/0 (for public access) or your own trusted IP range |
Key Name | The EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info. |
After successfully deploying Bitbucket Data Center in AWS, you'll need to configure it:
Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.
Data dimensions | Value for an enterprise-scale dataset |
---|---|
Projects | ~25 000 |
Repositories | ~52 000 |
Users | ~25 000 |
Pull Requests | ~ 1 000 000 |
Total files number | ~750 000 |
Pre-loading the dataset is a three-step process:
The following subsections explain each step in greater detail.
You can load this dataset directly into the database (via a populate_db.sh script).
To populate the database with SQL:
Using SSH, connect to the Bitbucket node via the Bastion instance:
For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):
1 2 3 4 5
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Stop Bitbucket Server:
1
sudo systemctl stop bitbucket
In a new terminal session connect to the Bitbucket NFS Server over SSH:
For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):
1 2 3 4 5
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NFS_SERVER_IP=nfs_server_private_ip
export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NFS_SERVER_IP}
For more information, go to Connecting your nodes over SSH.
Download the populate_db.sh script and make it executable:
1
wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/bitbucket/populate_db.sh && chmod +x populate_db.sh
Review the following Variables section
of the script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
INSTALL_PSQL_CMD="amazon-linux-extras install -y postgresql10"
DB_CONFIG="/media/atl/bitbucket/shared/bitbucket.properties"
# Depending on BITBUCKET installation directory
BITBUCKET_CURRENT_DIR="/opt/atlassian/bitbucket/current/"
BITBUCKET_VERSION_FILE="/media/atl/bitbucket/shared/bitbucket.version"
# DB admin user name, password and DB name
BITBUCKET_DB_NAME="bitbucket"
BITBUCKET_DB_USER="postgres"
BITBUCKET_DB_PASS="Password1!"
# Datasets AWS bucket and db dump name
DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/bitbucket"
DATASETS_SIZE="large"
Run the script:
1
./populate_db.sh | tee -a populate_db.log
After Importing the main dataset, you'll now have to pre-load an enterprise-scale set of attachments.
Using SSH, connect to the Bitbucket NFS Server via the Bastion instance:
For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):
1 2 3 4 5
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NFS_SERVER_IP=nfs_server_private_ip
export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@$BASTION_IP" ec2-user@${NFS_SERVER_IP}
For more information, go to Connecting your nodes over SSH.
Download the upload_attachments.sh script and make it executable:
1
wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/bitbucket/upload_attachments.sh && chmod +x upload_attachments.sh
Review the following Variables section
of the script:
1 2 3 4 5 6
DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/bitbucket"
ATTACHMENTS_TAR="attachments.tar.gz"
DATASETS_SIZE="large"
ATTACHMENTS_TAR_URL="${DATASETS_AWS_BUCKET}/${BITBUCKET_VERSION}/${DATASETS_SIZE}/${ATTACHMENTS_TAR}"
NFS_DIR="/media/atl/bitbucket/shared"
ATTACHMENT_DIR_DATA="data"
Run the script:
1
./upload_attachments.sh | tee -a upload_attachments.log
Using SSH, connect to the Bitbucket node via the Bastion instance:
For Linux or MacOS run following commands in terminal (for Windows use Git Bash terminal):
1 2 3 4 5
ssh-add path_to_your_private_key_pem
export BASTION_IP=bastion_instance_public_ip
export NODE_IP=node_private_ip
export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}
For more information, go to Connecting your nodes over SSH.
Start Bitbucket Server:
1
sudo systemctl start bitbucket
Wait 10-15 minutes until Bitbucket Server is started.
If your app does not use Bitbucket search functionality just skip this section.
Otherwise, if your app is depending on Bitbucket search functionality you need to wait until Elasticsearch index is finished. Bitbucket-project index and bitbucket-repository index usually take about 10 hours on a User Guide recommended configuration, bitbucket-search index (search by repositories content) could take up to a couple of days.
To check status of indexing:
Navigate to LoadBalancerURL/rest/indexing/latest/status
page:
"status":"INDEXING"
- index is in progress
"status":"IDLE"
- index is finished
For generating performance results suitable for Marketplace approval process use dedicated execution environment. This is a separate AWS EC2 instance to run the toolkit from. Running toolkit from dedicated instance but not from local machine eliminates network fluctuations and guarantees stable CPU and memory performance.
Ubuntu Server 18.04 LTS
.c5.2xlarge
30
GiBConnect to the instance using SSH or the AWS Systems Manager Sessions Manager.
1
ssh -i path_to_pem_file ubuntu@INSTANCE_PUBLIC_IP
Install Docker. Setup manage Docker as a non-root user.
bitbucket.yml
configuration file. Set enterprise-scale Jira Data Center parameters: 1 2 3 4 5 6 7 8 9 10 11 12
application_hostname: test_bitbucket_instance.atlassian.com # Bitbucket DC hostname without protocol and port e.g. test-bitbucket.atlassian.com or localhost
application_protocol: http # http or https
application_port: 80 # 80, 443, 8080, 7990 etc
secure: True # Set False to allow insecure connections, e.g. when using self-signed SSL certificate
application_postfix: # e.g. /bitbucket in case of url like http://localhost:7990/bitbucket
admin_login: admin
admin_password: admin
load_executor: jmeter # only jmeter executor is supported
concurrency: 20 # number of concurrent virtual users for jmeter scenario
test_duration: 50m
ramp-up: 10m # time to spin all concurrent users
total_actions_per_hour: 32700 # number of total JMeter actions per hour
You'll need to run the toolkit for each test scenario in the next section.
Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:
Each scenario will involve multiple test runs. The following subsections explain both in greater detail.
This scenario helps to identify basic performance issues without a need to spin up a multi-node Bitbucket DC. Make sure the app does not have any performance impact when it is not exercised.
To receive performance baseline results without an app installed:
Run toolkit with docker:
1 2 3
cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml
View the following main results of the run in the dc-app-performance-toolkit/app/results/jira/YY-MM-DD-hh-mm-ss
folder:
results_summary.log
: detailed run summaryresults.csv
: aggregated .csv file with all actions and timingsbzt.log
: logs of the Taurus tool executionjmeter.*
: logs of the JMeter tool executionpytest.*
: logs of Pytest-Selenium executionTo receive performance results with an app installed:
Run bzt.
1 2 3
cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml
To generate a performance regression report:
virtualenv
as described in dc-app-performance-toolkit/README.md
dc-app-performance-toolkit/app/reports_generation
folder.performance_profile.yml
file:Run the following command:
1
python csv_chart_generator.py performance_profile.yml
In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss
folder, view the .csv
file (with consolidated scenario results), the .png
chart file and performance scenario summary report.
Once completed, you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see an impact (>20%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.
The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.
For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Bitbucket DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Bitbucket DC app in a cluster.
Extension scripts, which extend the base Selenium (bitbucket-ui.py
) scripts, are located in a separate folder (dc-app-performance-toolkit/extension/bitbucket
). You can modify these scripts to include their app-specific actions.
You can extend Selenium scripts to measure the end-to-end browser timings.
We use Pytest to drive Selenium tests. The bitbucket-ui.py
executor script is located in the app/selenium_ui/
folder. This file contains all browser actions, defined by the test_ functions
. These actions are executed one by one during the testing.
You develop an app that adds additional UI elements to a repository page, in this case you should edit dc-app-performance-toolkit/extension/bitbucket/extension_ui.py
:
Code example.
In the bitbucket-ui.py
script, view the following block of code:
1 2
# def test_1_selenium_custom_action(webdriver, datasets, screen_shots):
# app_specific_action(webdriver, datasets)
To view more examples, see the modules.py
file in the selenium_ui/bitbucket
directory.
To ensure that the test runs without errors in parallel, run your extension scripts with the base scripts as a sanity check.
To receive scalability benchmark results for one-node Bitbucket DC with app-specific actions, run bzt
:
Run toolkit with docker:
1 2 3
cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml
To receive scalability benchmark results for two-node Bitbucket DC with app-specific actions:
2
in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.Run toolkit with docker:
1 2 3
cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml
To receive scalability benchmark results for four-node Bitbucket DC with app-specific actions:
Run bzt.
1 2 3
cd dc-app-performance-toolkit
docker pull atlassian/dcapt
docker run --shm-size=4g -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml
To generate a scalability report:
dc-app-performance-toolkit/app/reports_generation
folder.scale_profile.yml
file:Run the following command from the virtualenv
(as described in dc-app-performance-toolkit/README.md
):
1
python csv_chart_generator.py scale_profile.yml
In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss
folder, view the .csv
file (with consolidated scenario results), the .png
chart file and summary report.
dc-app-performance-toolkit/app/results
folder to your local machine.dc-app-performance-toolkit/app/results
folder and attach archive to DCHELP ticket.In case of technical questions, issues or problems with DC Apps Performance Toolkit, contact us for support in the community Slack #data-center-app-performance-toolkit channel.
Rate this page: