Atlassian Marketplace Developer

Atlassian Marketplace Developer

Last updatedOct 12, 2020

Rate this page:

Data Center App Performance Toolkit User Guide For Bitbucket

This document walks you through the process of testing your app on Bitbucket using the Data Center App Performance Toolkit. These instructions focus on producing the required performance and scale benchmarks for your Data Center app.

To use the Data Center App Performance Toolkit, you'll need to:

  1. Set up Bitbucket Data Center on AWS.
  2. Load an enterprise-scale dataset on your Bitbucket Data Center deployment.
  3. Set up an execution environment for the toolkit.
  4. Run all the testing scenarios in the toolkit.

For simple spikes or tests, you can skip steps 1-2 and target any Bitbucket test instance. When you set up your execution environment, you may need to edit the scripts according to your test instance's data set.


1. Setting up Bitbucket Data Center

We recommend that you use the AWS Quick Start for Bitbucket Data Center to deploy a Bitbucket Data Center testing environment. This Quick Start will allow you to deploy Bitbucket Data Center with a new Atlassian Standard Infrastructure (ASI) or into an existing one.

The ASI is a Virtual Private Cloud (VPC) consisting of subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components required by all Atlassian applications, and then deploys Bitbucket into this new VPC. Deploying Bitbucket with a new ASI takes around 50 minutes. With an existing one, it'll take around 30 minutes.

Using the AWS Quick Start for Bitbucket

If you are a new user, perform an end-to-end deployment. This involves deploying Bitbucket into a new ASI.

If you have already deployed the ASI separately by using the ASI Quick Start or by deploying another Atlassian product (Jira, Bitbucket, or Confluence Data Center), deploy Bitbucket into your existing ASI.

You are responsible for the cost of the AWS services used while running this Quick Start reference deployment. There is no additional price for using this Quick Start. For more information, go to aws.amazon.com/pricing.

To reduce costs, we recommend you to keep your deployment up and running only during the performance runs.

AWS cost estimation

AWS Pricing Calculator provides an estimate of usage charges for AWS services based on certain information you provide. Monthly charges will be based on your actual usage of AWS services, and may vary from the estimates the Calculator has provided.

*The prices below are approximate and may vary depending on factors such as (region, instance type, deployment type of DB, etc.)

StackEstimated hourly cost ($)
One Node Bitbucket DC1.4 - 2.0
Two Nodes Bitbucket DC1.7 - 2.5
Four Nodes Bitbucket DC2.4 - 3.6

Stop Bitbucket cluster nodes

To reduce AWS infrastructure costs you could stop Bitbucket nodes when the cluster is standing idle.
Bitbucket node might be stopped by using Suspending and Resuming Scaling Processes.

To stop one node within the Bitbucket cluster follow the instructions: 1. Go to EC2 Auto Scaling Groups and open the necessary group to which belongs the node you want to stop. 1. Press Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and add HealthCheck to the Suspended Processes. Amazon EC2 Auto Scaling stops marking instances unhealthy as a result of EC2 and Elastic Load Balancing health checks. 1. Go to Instances and stop Bitbucket node.

To return Bitbucket node into a working state follow the instructions:
1. Go to Instances and start Bitbucket node, wait a few minutes for Bitbucket node to become responsible. 1. Go to EC2 Auto Scaling Groups and open the necessary group to which belongs the node you want to start. 1. Press Edit (in case you have New EC2 experience UI mode enabled, press Edit on Advanced configuration) and remove HealthCheck from Suspended Processes of Auto Scaling Group.

Quick Start parameters

All important parameters are listed and described in this section. For all other remaining parameters, we recommend using the Quick Start defaults.

Bitbucket setup

ParameterRecommended Value
VersionThe Data Center App Performance Toolkit officially supports 6.10.5 (Long Term Support releases) or 7.0.5

Cluster nodes

ParameterRecommended Value
Bitbucket cluster node instance typec5.2xlarge
Maximum number of cluster nodes1
Minimum number of cluster nodes1
Cluster node instance volume size50

We recommend c5.2xlarge to strike the balance between cost and hardware we see in the field for our enterprise customers. More info could be found in public recommendations.

The Data Center App Performance Toolkit framework is also set up for concurrency we expect on this instance size. As such, underprovisioning will likely show a larger performance impact than expected.

File server

ParameterRecommended Value
File server instance typem4.xlarge
Home directory size1000

Database

ParameterRecommended Value
Database instance classdb.m4.large
RDS Provisioned IOPS1000
Master passwordPassword1!
Enable RDS Multi-AZ deploymentfalse
Bitbucket database passwordPassword1!
Database storage100

The Master (admin) password will be used later when restoring the SQL database dataset. If password value is not set to default, you'll need to change DB_PASS value manually in the restore database dump script (later in Preloading your Bitbucket deployment with an enterprise-scale dataset).

Elasticsearch

ParameterRecommended Value
Elasticsearch instance typem4.xlarge.elasticsearch
Elasticsearch disk-space per node (GB)1000

Networking (for new ASI)

ParameterRecommended Value
Trusted IP range0.0.0.0/0 (for public access) or your own trusted IP range
Availability ZonesSelect two availability zones in your region
Permitted IP range0.0.0.0/0 (for public access) or your own trusted IP range
Make instance internet facingtrue
Key NameThe EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info.

Networking (for existing ASI)

ParameterRecommended Value
Make instance internet facingtrue
Permitted IP range0.0.0.0/0 (for public access) or your own trusted IP range
Key NameThe EC2 Key Pair to allow SSH access. See Amazon EC2 Key Pairs for more info.

Running the setup wizard

After successfully deploying Bitbucket Data Center in AWS, you'll need to configure it:

  1. In the AWS console, go to Services > CloudFormation > Stack > Stack details > Select your stack.
  2. On the Outputs tab, copy the value of the LoadBalancerURL key.
  3. Open LoadBalancerURL in your browser. This will take you to the Bitbucket setup wizard.
  4. On the Bitbucket setup page, populate the following fields:
    • Application title: any name for your Bitbucket Data Center deployment
    • Base URL: your stack's Elastic LoadBalancer URL
    • License key: select new evaluation license or existing license checkbox Click Next.
  5. On the Administrator account setup page, populate the following fields:
    • Username: admin (recommended)
    • Full name: any full name of the admin user
    • Email address: email address of the admin user
    • Password: admin (recommended)
    • Confirm Password: admin (recommended) Click Go to Bitbucket.

After Preloading your Bitbucket deployment with an enterprise-scale dataset, the admin user will have admin/admin credentials.


2. Preloading your Bitbucket deployment with an enterprise-scale dataset

Data dimensions and values for an enterprise-scale dataset are listed and described in the following table.

Data dimensionsValue for an enterprise-scale dataset
Projects~25 000
Repositories~52 000
Users~25 000
Pull Requests~ 1 000 000
Total files number~750 000

All the datasets use the standard admin/admin credentials.

Pre-loading the dataset is a three-step process:

  1. Importing the main dataset. To help you out, we provide an enterprise-scale dataset you can import either via the populate_db.sh script or restore from xml backup file.
  2. Restoring attachments. We also provide attachments, which you can pre-load via an upload_attachments.sh script.

The following subsections explain each step in greater detail.

Importing the main dataset

You can load this dataset directly into the database (via a populate_db.sh script).

Loading the dataset via populate_db.sh script (~2 hours)

We recommend doing this via the CLI.

To populate the database with SQL:

  1. In the AWS console, go to Services > EC2 > Instances.
  2. On the Description tab, do the following:
    • Copy the Public IP of the Bastion instance.
    • Copy the Private IP of the Bitbucket node instance.
    • Copy the Private IP of the Bitbucket NFS Server instance.
  3. Using SSH, connect to the Bitbucket node via the Bastion instance:

    For Windows, use Putty to connect to the Bitbucket node over SSH. For Linux or MacOS:

    1
    2
    3
    4
    5
    ssh-add path_to_your_private_key_pem
    export BASTION_IP=bastion_instance_public_ip
    export NODE_IP=node_private_ip
    export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
    ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

    For more information, go to Connecting your nodes over SSH.

  4. Stop Bitbucket Server:

    1
    sudo systemctl stop bitbucket
  5. In a new terminal session connect to the Bitbucket NFS Server over SSH:

    For Windows, use Putty to connect to the Bitbucket node over SSH. For Linux or MacOS:

    1
    2
    3
    4
    5
    ssh-add path_to_your_private_key_pem
    export BASTION_IP=bastion_instance_public_ip
    export NFS_SERVER_IP=nfs_server_private_ip
    export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
    ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NFS_SERVER_IP}

    For more information, go to Connecting your nodes over SSH.

  6. Download the populate_db.sh script and make it executable:

    1
    wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/bitbucket/populate_db.sh && chmod +x populate_db.sh
  7. Review the following Variables section of the script:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    INSTALL_PSQL_CMD="amazon-linux-extras install -y postgresql10"
    DB_CONFIG="/media/atl/bitbucket/shared/bitbucket.properties"
    
    # Depending on BITBUCKET installation directory
    BITBUCKET_CURRENT_DIR="/opt/atlassian/bitbucket/current/"
    BITBUCKET_VERSION_FILE="/media/atl/bitbucket/shared/bitbucket.version"
    
    # DB admin user name, password and DB name
    BITBUCKET_DB_NAME="bitbucket"
    BITBUCKET_DB_USER="postgres"
    BITBUCKET_DB_PASS="Password1!"
    
    # Datasets AWS bucket and db dump name
    DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/bitbucket"
    DATASETS_SIZE="large"
  8. Run the script:

    1
    ./populate_db.sh | tee -a populate_db.log

Do not close or interrupt the session. It will take about an hour to restore SQL database. When SQL restoring is finished, an admin user will have admin/admin credentials.

In case of a failure, check the Variables section and run the script one more time.

Restoring attachments (~2 hours)

After Importing the main dataset, you'll now have to pre-load an enterprise-scale set of attachments.

  1. Using SSH, connect to the Bitbucket NFS Server via the Bastion instance:

    For Windows, use Putty to connect to the Bitbucket node over SSH. For Linux or MacOS:

    1
    2
    3
    4
    5
    ssh-add path_to_your_private_key_pem
    export BASTION_IP=bastion_instance_public_ip
    export NFS_SERVER_IP=nfs_server_private_ip
    export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
    ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@$BASTION_IP" ec2-user@${NFS_SERVER_IP}

    For more information, go to Connecting your nodes over SSH.

  2. Download the upload_attachments.sh script and make it executable:

    1
    wget https://raw.githubusercontent.com/atlassian/dc-app-performance-toolkit/master/app/util/bitbucket/upload_attachments.sh && chmod +x upload_attachments.sh
  3. Review the following Variables section of the script:

    1
    2
    3
    4
    5
    6
    DATASETS_AWS_BUCKET="https://centaurus-datasets.s3.amazonaws.com/bitbucket"
    ATTACHMENTS_TAR="attachments.tar.gz"
    DATASETS_SIZE="large"
    ATTACHMENTS_TAR_URL="${DATASETS_AWS_BUCKET}/${BITBUCKET_VERSION}/${DATASETS_SIZE}/${ATTACHMENTS_TAR}"
    NFS_DIR="/media/atl/bitbucket/shared"
    ATTACHMENT_DIR_DATA="data"
  4. Run the script:

    1
    ./upload_attachments.sh | tee -a upload_attachments.log

Do not close or interrupt the session. It will take about two hours to upload attachments.

Start Bitbucket Server

  1. Using SSH, connect to the Bitbucket node via the Bastion instance:

    For Windows, use Putty to connect to the Bitbucket node over SSH. For Linux or MacOS:

    1
    2
    3
    4
    5
    ssh-add path_to_your_private_key_pem
    export BASTION_IP=bastion_instance_public_ip
    export NODE_IP=node_private_ip
    export SSH_OPTS='-o ServerAliveInterval=60 -o ServerAliveCountMax=30'
    ssh ${SSH_OPTS} -o "proxycommand ssh -W %h:%p ${SSH_OPTS} ec2-user@${BASTION_IP}" ec2-user@${NODE_IP}

    For more information, go to Connecting your nodes over SSH.

  2. Start Bitbucket Server:

    1
    sudo systemctl start bitbucket
  3. Wait 10-15 minutes until Bitbucket Server is started.

Elasticsearch Index

If your app does not use Bitbucket search functionality just skip this section.

Otherwise, if your app is depending on Bitbucket search functionality you need to wait until Elasticsearch index is finished. Bitbucket-project index and bitbucket-repository index usually take about 10 hours on a User Guide recommended configuration, bitbucket-search index (search by repositories content) could take up to a couple of days.

To check status of indexing:

  1. Open LoadBalancerURL in your browser.
  2. Login with admin user.
  3. Navigate to LoadBalancerURL/rest/indexing/latest/status page:

    "status":"INDEXING" - index is in progress

    "status":"IDLE" - index is finished

In case of any difficulties with Index generation, contact us for support in the community Slack #data-center-app-performance-toolkit channel.


3. Setting up an execution environment

For generating performance results suitable for Marketplace approval process use dedicated execution environment. This is a separate AWS EC2 instance to run the toolkit from. Running toolkit from dedicated instance but not from local machine eliminates network fluctuations and guarantees stable CPU and memory performance.

  1. Launch AWS EC2 instance. Instance type: c5.2xlarge, OS: select from Quick Start Ubuntu Server 18.04 LTS.
  2. Connect to the instance using SSH or the AWS Systems Manager Sessions Manager.

    1
    ssh -i path_to_pem_file ubuntu@INSTANCE_PUBLIC_IP
  3. Install Docker. Setup manage Docker as a non-root user.

  4. Go to GitHub and create a fork of dc-app-performance-toolkit.
  5. Clone the fork locally, then edit the bitbucket.yml configuration file. Set enterprise-scale Jira Data Center parameters:
1
2
3
4
5
6
7
8
9
10
11
12
    application_hostname: test_bitbucket_instance.atlassian.com   # Bitbucket DC hostname without protocol and port e.g. test-bitbucket.atlassian.com or localhost
    application_protocol: http      # http or https
    application_port: 80            # 80, 443, 8080, 7990 etc
    secure: True                    # Set False to allow insecure connections, e.g. when using self-signed SSL certificate
    application_postfix:            # e.g. /bitbucket in case of url like http://localhost:7990/bitbucket
    admin_login: admin
    admin_password: admin
    load_executor: jmeter           # only jmeter executor is supported
    concurrency: 20                 # number of concurrent virtual users for jmeter scenario
    test_duration: 50m
    ramp-up: 10m                    # time to spin all concurrent users
    total_actions_per_hour: 32700   # number of total JMeter actions per hour
  1. Push your changes to the forked repository.
  2. Connect to the AWS EC2 instance and clone forked repository.

At this stage app-specific actions are not needed yet. Use code from master branch with your bitbucket.yml changes.

You'll need to run the toolkit for each test scenario in the next section.


4. Running the test scenarios on your execution environment

Using the Data Center App Performance Toolkit for Performance and scale testing your Data Center app involves two test scenarios:

Each scenario will involve multiple test runs. The following subsections explain both in greater detail.

Scenario 1: Performance regression

This scenario helps to identify basic performance issues without a need to spin up a multi-node Bitbucket DC. Make sure the app does not have any performance impact when it is not exercised.

Run 1 (~1 hour)

To receive performance baseline results without an app installed:

  1. Use SSH to connect to execution environment.
  2. Run toolkit with docker:

    1
    2
    cd dc-app-performance-toolkit
    docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml
  3. View the following main results of the run in the dc-app-performance-toolkit/app/results/jira/YY-MM-DD-hh-mm-ss folder:

    • results_summary.log: detailed run summary
    • results.csv: aggregated .csv file with all actions and timings
    • bzt.log: logs of the Taurus tool execution
    • jmeter.*: logs of the JMeter tool execution
    • pytest.*: logs of Pytest-Selenium execution

Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps.

Run 2 (~1 hour)

To receive performance results with an app installed:

  1. Install the app you want to test.
  2. Run bzt.

    1
    2
     cd dc-app-performance-toolkit
     docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml

Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps.

Generating a performance regression report

To generate a performance regression report:

  1. Use SSH to connect to execution environment.
  2. Install the virtualenv as described in dc-app-performance-toolkit/README.md
  3. Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
  4. Edit the performance_profile.yml file:
    • Under runName: "without app", in the fullPath key, insert the full path to results directory of Run 1.
    • Under runName: "with app", in the fullPath key, insert the full path to results directory of Run 2.
  5. Run the following command:

    1
    python csv_chart_generator.py performance_profile.yml
  6. In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and performance scenario summary report.

Analyzing report

Once completed, you will be able to review the action timings with and without your app to see its impact on the performance of the instance. If you see an impact (>20%) on any action timing, we recommend taking a look into the app implementation to understand the root cause of this delta.

Scenario 2: Scalability testing

The purpose of scalability testing is to reflect the impact on the customer experience when operating across multiple nodes. For this, you have to run scale testing on your app.

For many apps and extensions to Atlassian products, there should not be a significant performance difference between operating on a single node or across many nodes in Bitbucket DC deployment. To demonstrate performance impacts of operating your app at scale, we recommend testing your Bitbucket DC app in a cluster.

Extending the base action

Extension scripts, which extend the base Selenium (bitbucket-ui.py) scripts, are located in a separate folder (dc-app-performance-toolkit/extension/bitbucket). You can modify these scripts to include their app-specific actions.

Modifying Selenium

You can extend Selenium scripts to measure the end-to-end browser timings.

We use Pytest to drive Selenium tests. The bitbucket-ui.py executor script is located in the app/selenium_ui/ folder. This file contains all browser actions, defined by the test_ functions. These actions are executed one by one during the testing.

Example of app-specific Selenium action development

You develop an app that adds additional UI elements to a repository page, in this case you should edit dc-app-performance-toolkit/extension/bitbucket/extension_ui.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from selenium.webdriver.common.by import By
from selenium_ui.conftest import print_timing
from util.conf import BITBUCKET_SETTINGS

from selenium_ui.base_page import BasePage


def app_specific_action(webdriver, datasets):
    page = BasePage(webdriver)
    repo = datasets['repos']
    repo_slug = repo[0]
    project_key = repo[1]

    @print_timing("selenium_app_custom_action")
    def measure():

        @print_timing("selenium_app_custom_action:view_repo_page")
        def sub_measure():
            page.go_to_url(f"{BITBUCKET_SETTINGS.server_url}/projects/{project_key}/repos/{repo_slug}/browse")
            page.wait_until_visible((By.CSS_SELECTOR, '.aui-navgroup-vertical>.aui-navgroup-inner')) # Wait for repo navigation panel is visible 
            page.wait_until_visible((By.ID, 'ID_OF_YOUR_APP_SPECIFIC_UI_ELEMENT'))  # Wait for you app-specific UI element by ID selector
        sub_measure()
    measure()

In the bitbucket-ui.py script, view the following block of code:

1
2
# def test_1_selenium_custom_action(webdriver, datasets, screen_shots):
#     app_specific_action(webdriver, datasets)

To view more examples, see the modules.py file in the selenium_ui/bitbucket directory.

Running tests with your modification

To ensure that the test runs without errors in parallel, run your extension scripts with the base scripts as a sanity check.

Run 3 (~1 hour)

To receive scalability benchmark results for one-node Bitbucket DC with app-specific actions, run bzt:

  1. Apply app-specific code changes to a new branch of forked repo.
  2. Use SSH to connect to execution environment.
  3. Pull cloned fork repo branch with app-specific actions.
  4. Run toolkit with docker:

    1
    2
     cd dc-app-performance-toolkit
     docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml

Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps.

Run 4 (~1 hour)

To receive scalability benchmark results for two-node Bitbucket DC with app-specific actions:

  1. In the AWS console, go to CloudFormation > Stack details > Select your stack.
  2. On the Update tab, select Use current template, and then click Next.
  3. Enter 2 in the Maximum number of cluster nodes and the Minimum number of cluster nodes fields.
  4. Click Next > Next > Update stack and wait until stack is updated.
  5. Run toolkit with docker:

    1
    2
     cd dc-app-performance-toolkit
     docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml

Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps.

Run 5 (~1 hour)

To receive scalability benchmark results for four-node Bitbucket DC with app-specific actions:

  1. Scale your Bitbucket Data Center deployment to 4 nodes the same way as in Run 4.
  2. Run bzt.

    1
    2
     cd dc-app-performance-toolkit
     docker run --shm-size=4g  -v "$PWD:/dc-app-performance-toolkit" atlassian/dcapt bitbucket.yml

Review results_summary.log file under artifacts dir location. Make sure that overall status is OK before moving to the next steps.

Generating a report for scalability scenario

To generate a scalability report:

  1. Use SSH to connect to execution environment.
  2. Navigate to the dc-app-performance-toolkit/app/reports_generation folder.
  3. Edit the scale_profile.yml file:
    • For runName: "Node 1", in the fullPath key, insert the full path to results directory of Run 3.
    • For runName: "Node 2", in the fullPath key, insert the full path to results directory of Run 4.
    • For runName: "Node 4", in the fullPath key, insert the full path to results directory of Run 5.
  4. Run the following command from the virtualenv:

    1
    python csv_chart_generator.py scale_profile.yml
  5. In the dc-app-performance-toolkit/app/results/reports/YY-MM-DD-hh-mm-ss folder, view the .csv file (with consolidated scenario results), the .png chart file and summary report.

Analyzing report

  1. Use scp command to copy dc-app-performance-toolkit/app/results folder to your local machine.
  2. Make sure you have five run results folders and two reports (remove all unsuccessful attempts).
  3. Zip dc-app-performance-toolkit/app/results folder and attach archive to DCHELP ticket.

Support

In case of technical questions, issues or problems with DC Apps Performance Toolkit, contact us for support in the community Slack #data-center-app-performance-toolkit channel.

Rate this page: