Bitbucket Dynamic Pipelines provider

The bitbucket:dynamicPipelinesProvider module defines a Dynamic Pipelines provider that can be configured in the context of a repository or a workspace. Dynamic Pipelines provider generates pipeline definition at runtime using dynamic logic, as opposed to the static Pipelines configuration file.

System hierarchy

For the purpose of this explanation, the term “resources” refers to either a Repository or a Workspace; Projects are not currently supported, but will be added in the future. Resources exist hierarchically, e.g. a Repository exists “under” a Workspace. Dynamic pipelines leverages this hierarchy in order to enable flexibility to be added to pipeline workflows at every level.

For any given Workspace, there can be many bitbucket:dynamicPipelinesProvider modules installed. However, each resource within that workspace can only have a single dynamicPipelinesProvider selected to execute for that resource.

For example:

• You may have four seperate repositories within a workspace - Repos A, B, C, & D
  • Those repositories may all have unique bitbucket:dynamicPipelinesProvider modules configured for each of them (Providers A, B, C, & D), or they may share/reuse providers in any configuration the user wants.
  • However, each individual repository can only have a single bitbucket:dynamicPipelinesProvider module configured for it.
• In addition, a bitbucket:dynamicPipelinesProvider module can also be configured against other resources above those repositories in the hierarchy (e.g. the parent workspace).
  • Just like at the repository level, only a single bitbucket:dynamicPipelinesProvider can be configured for those resources (e.g. the parent workspace).
• At runtime, these bitbucket:dynamicPipelinesProvider modules at different levels of the hierarchy create a chain, all of which will be executed for the respective pipeline run.
  • This allows multiple levels of dynamic execution to occur, in a specific order, one at each level of the hierarchy.

Each Dynamic Pipelines provider acts as a transforming function of the pipeline configuration: it takes a valid pipeline configuration along with pipeline metadata as input, and generates a valid pipeline configuration as output. This “Pipeline-In|Pipeline-Out” design is what enables multiple providers to be chained together up the hierarchy.

This chain will always be executed in a “bottom up” order, starting with the static .yml configuration stored in the repository, then the repository level bitbucket:dynamicPipelinesProvider and finally, the workspace level bitbucket:dynamicPipelinesProvider. This ensures that policies and rules put in place “higher” in the hierarchy are able to take precedence over ones implemented “lower” in the hierarchy.

• initial workflow is read from the static bitbucket-pipelines.yml file.
• if shared workflow is used, import statements are resolved from the static bitbucket-pipelines.yml files of the external repositories.
• if a repository-level dynamic pipeline is configured, it is executed and transforms the workflow.
• if a workspace-level dynamic pipeline is configured, it is executed and transforms the workflow.

The output of this chain is the final pipeline workflow which is then consumed by Bitbucket Pipelines to generate and run a pipeline using that definition.

Properties

Example

The snippet below defines a Dynamic Pipelines provider.

1
2
modules:
  bitbucket:dynamicPipelinesProvider:
    - key: my-bitbucket-dynamic-pipelines-provider
      name: My first Dynamic Pipelines provider
      description: This is my first provider
      function: dynamic-pipelines-provider
  function:
    - key: dynamic-pipelines-provider
      handler: index.dynamicPipelinesProviderFunction

Invocation scenarios

Whenever a bitbucket:dynamicPipelinesProvider module is invoked, an event payload will be provided as an argument to the Forge function. This payload provides the identifiers of the workspace, repository, pipeline configuration, and some additional metadata.

A bitbucket:dynamicPipelinesProvider module is invoked by Bitbucket Pipelines in two scenarios:

• To generate a list of available pipeline definitions that a user can run against a particular branch or tag.
• To generate the configuration for a pipeline in order to run it in response to a particular event.

The only responsibility of the function registered against the bitbucket:dynamicPipelinesProvider module is that it returns a well-defined response payload every time the Bitbucket Forge infrastructure invokes it.

This result produced by each invocation of the bitbucket:dynamicPipelinesProvider module is handled and stored by the Bitbucket infrastructure, there is no requirement for the Forge app to store it, or send the response back to a specific Bitbucket API. The Dynamic Pipelines provider is only expected to calculate and return result in the correct response format.

Get pipeline definitions

This scenario is triggered, for example, when user opens the Run pipeline dialog in the UI and selects a branch.

The Dynamic Pipelines provider is expected to return all pipeline definitions applicable for the provided context. From this set of results, the user can then select a definition to run.

Invocation payload

Example

Request to get pipeline definitions with a single resolved default definition:

1
2
{
  "workspace": "{308ce669-576d-4fa7-b4d4-15e8171f0c8c}",
  "repository": "{2fc883c6-3dee-4e73-a354-ccee4bd77938}",
  "creator": "{cda97709-b1a9-4544-9a8d-750f1b3a12c4}",
  "trigger": "definitions",
  "target": {
    "type": "pipeline_commit_target",
    "commit": {
      "hash": "e34ad324d1e4334777dca3e56c04eba36c2a68c6"
    }
  },
  "pipelines_configuration": {
    "pipelines": {
      "default": [
        {
          "step": {
            "name": "Foo",
            "script": [
              "echo Bar"
            ]
          }
        }
      ]
    }
  }
}

PipelineTarget

Commit

PullRequest

Examples

Commit target:

1
2
{
  "type": "pipeline_commit_target",
  "commit": {
    "hash": "ca0beaeaf9c4d6ef07e9abf2de0d262014f23dd5"
  }
}

Ref target:

1
2
{
  "type": "pipeline_ref_target",
  "commit": {
    "hash": "ca0beaeaf9c4d6ef07e9abf2de0d262014f23dd5"
  },
  "ref_type": "branch",
  "ref_name": "master"
}

Pull request target:

1
2
{
  "type": "pipeline_pullrequest_target",
  "commit": {
    "hash": "6486ba02730e5a90394cbd5d3f27aa84ccabb1e4"
  },
  "source": "feature",
  "destination": "master",
  "destination_commit": {
    "hash": "ca0beaeaf9c4d6ef07e9abf2de0d262014f23dd5"
  },
  "pullrequest": {
    "id": 239
  }
}

PipelinesConfiguration

This is a complex type which covers all the features and properties supported in the Bitbucket Pipelines YML configuration file. Simply speaking, this is a JSON version of what you can possibly configure in bitbucket-pipelines.yml file, with some minor nuances.

Image

The parameters of the Docker image.

AwsImageProperties

The properties specific to images stored in Amazon Elastic Container Registry (AWS ECR).

Examples

Simple image:

1
2
{
  "name": "atlassian/default-image:3",
  "run-as-user": 1001
}

Image with basic authentication:

1
2
{
  "name": "my.docker.registry/my-project/my-repo/test-image:latest",
  "username": "$DOCKER_REGISTRY_USERNAME",
  "password": "$DOCKER_REGISTRY_PASSWORD"
}

AWS ECR image with credentials:

1
2
{
  "name": "aws_account_id.dkr.ecr.us-west-2.amazonaws.com/my-repository:tag",
  "aws": {
    "access-key": "$AWS_ACCESS_KEY",
    "secret-key": "$AWS_SECRET_KEY"
  }
}

AWS ECR image with OpenID Connect:

1
2
{
  "name": "aws_account_id.dkr.ecr.us-west-2.amazonaws.com/my-repository:tag",
  "aws": {
    "oidc-role": "arn:aws:iam::aws_account_id:role/pipelines-ecr-access"
  }
}

Options

Global options allow to override the default values applied to all steps in all declared pipelines.

Example

1
2
{
  "docker": true,
  "max-time": 60,
  "size": "2x"
}

Clone

Settings for cloning a repository into a container.

SparseCheckout

The properties of Git sparse checkout mode.

Example

1
2
{
  "depth": "3",
  "enabled": true,
  "lfs": true,
  "skip-ssl-verify": true
}

Definitions

The definitions of caches and services used in the declared pipelines.

Cache

A definition of a custom cache.

CacheKey

Describes the set of files to generate the cache key.

Service

A definition of a custom service.

Examples

Definitions of caches:

1
2
{
  "caches": {
    "simple-cache": {
      "path": "/cache/a"
    },
    "cache-with-key": {
      "key": {
        "files": [
          "dependencies.lock"
        ]
      },
      "path": "/cache/b"
    }
  }
}

Definitions of services:

1
2
{
  "services": {
    "service-a": {
      "image": {
        "name": "my.docker.registry/service-a-image:latest",
        "username": "$DOCKER_REGISTRY_USERNAME",
        "password": "$DOCKER_REGISTRY_PASSWORD",
        "run-as-user": 1111
      },
      "memory": 128,
      "type": "docker",
      "variables": {
        "FOO": "BAR",
        "BAZ": "QUX"
      }
    },
    "service-b": {
      "image": {
        "name": "sample/service-b-image:latest"
      },
      "variables": {
        "HELLO": "WORLD"
      }
    }
  }
}

Pipelines

The pipeline definitions, grouped by the trigger type.

Pipeline

Pipeline definition is an array of objects wrapped into a single-value map with their type as the key:

• step — individual pipeline step
• parallel — declares steps in the parallel group to run concurrently
• stage — a logical group of steps to run sequentially

Example

A pipeline with a step, followed by a stage with two steps, followed by a parallel group with two steps:

1
2
[
  {
    "step": {
      "name": "Just a step",
      "script": [
        "echo This is just a step"
      ]
    }
  },
  {
    "stage": {
      "name": "A stage",
      "steps": [
        {
          "step": {
            "name": "Step 1 in stage",
            "script": [
              "echo This is the first step of a stage"
            ]
          }
        },
        {
          "step": {
            "name": "Step 2 in stage",
            "script": [
              "echo This is the second step of a stage"
            ]
          }
        }
      ]
    }
  },
  {
    "parallel": {
      "steps": [
        {
          "step": {
            "name": "Parallel step 1",
            "script": [
              "echo This is the first step of a parallel group"
            ]
          }
        },
        {
          "step": {
            "name": "Parallel step 2",
            "script": [
              "echo This is the second step of a parallel group"
            ]
          }
        }
      ]
    }
  }
]

CustomPipeline

Custom pipeline definition additionally allows another named item to be declared in the array before all steps and/or stages of the pipeline:

• variables — variables for the custom pipeline

CustomPipelineVariables

An array of variables for the custom pipeline, each item of which is of CustomPipelineVariable type.

CustomPipelineVariable

Properties of the custom pipeline variable.

Example

A custom pipeline with variables declaration and a step that uses these variables:

1
2
[
  {
    "variables": [
      {
        "name": "Var"
      },
      {
        "name": "VarWithDefaultValue",
        "default": "Fallback value when user didn't provide one",
        "description": "Example of a variable with default value"
      },
      {
        "name": "VarWithAllowedValues",
        "default": "foo",
        "allowed-values": [
          "foo",
          "bar",
          "baz"
        ]
      }
    ]
  },
  {
    "step": {
      "name": "Print variables",
      "script": [
        "echo Var: $Var",
        "echo VarWithDefaultValue: $VarWithDefaultValue",
        "echo VarWithAllowedValues: $VarWithAllowedValues"
      ]
    }
  }
]

Step

Condition

The condition to execute a step or a stage.

ChangesetsCondition

Artifacts

Pipe

Examples

A manually triggered step which uses custom caches and services:

1
2
{
  "name": "Manually triggered deployment step with caches and services",
  "trigger": "manual",
  "deployment": "test",
  "caches": [
    "simple-cache",
    "cache-with-key"
  ],
  "services": [
    "service-a",
    "service-b"
  ],
  "script": [
    "echo Deploying to test environment..."
  ]
}

A step with custom size which makes use of a pipe and after-script:

1
2
{
  "step": {
    "name": "A step with custom size, a pipe call, and after-script",
    "size": "2x",
    "script": [
      "echo I will call a pipe now...",
      {
        "pipe": "atlassian/slack-notify:2.2.0",
        "variables": {
          "FOO": "BAR"
        }
      }
    ],
    "after-script": [
      "echo This step is about to finish"
    ]
  }
}

A step which conditionally runs if certain files have been updated:

1
2
{
  "step": {
    "name": "Conditional step",
    "condition": {
      "changesets": {
        "includePaths": [
          "foo/*.bar",
          "qux/**"
        ]
      }
    },
    "script": [
      "echo Some interesting files have been changed"
    ]
  }
}

Parallel

Example

1
2
{
  "parallel": {
    "fail-fast": true,
    "steps": [
      {
        "step": {
          "name": "Parallel step 1",
          "script": [
            "echo Run first parallel operation..."
          ]
        }
      },
      {
        "step": {
          "name": "Parallel step 2",
          "script": [
            "echo Run second parallel operation..."
          ]
        }
      }
    ]
  }
}

Stage

Example

A conditional deployment stage with two steps:

1
2
{
  "name": "Conditional deployment stage",
  "deployment": "staging",
  "trigger": "automatic",
  "condition": {
    "changesets": {
      "includePaths": [
        "staging/*.conf",
        "global/**"
      ]
    }
  },
  "steps": [
    {
      "step": {
        "name": "Stage step 1",
        "script": [
          "echo Run first stage operation..."
        ]
      }
    },
    {
      "step": {
        "name": "Stage step 2",
        "script": [
          "echo Run second stage operation..."
        ]
      }
    }
  ]
}

Labels

Labels represent additional free-form key-value data supplied in the configuration YAML.

The following restrictions apply to the labels:

• key and value must be no longer than 128 characters each
• there can be no more than 10 labels
• labels can be nested with a limit of 5 levels deep

Example

A 3 levels deep map declaring 4 labels.

1
2
{
  "label": "foo",
  "bar": {
    "sub_label": "baz",
    "qux": {
      "sub_sub_label": "quux"
    },
    "another_sub_label": "corge"
  }
}

Response payload

The Dynamic Pipelines provider is expected to modify (or retain) the configuration passed in the pipelines_configuration parameter and then return the outcome back. Returning exactly what was passed to the dynamic pipeline provider is a valid response, and is the expected behaviour when no modifications are required.

Get configuration for a pipeline

This scenario is triggered in response to an event, such as:

• commit pushed
• pull request created or updated
• user requested to run specific pipeline manually
• a pipeline was requested to run on user-configured schedule

The Dynamic Pipelines provider is expected to return a pipeline configuration containing a definition matching the provided selector. In certain cases Dynamic Pipelines provider may return a configuration with no matching definitions.

Invocation payload

Example

Request to get definition of a pipeline in response to a commit pushed to master branch of the repository and with a single matched branch-specific mas* pipeline definition from the previously discovered configuration (e.g. in the static bitbucket-pipelines.yml file):

1
2
{
  "workspace": "{308ce669-576d-4fa7-b4d4-15e8171f0c8c}",
  "repository": "{2fc883c6-3dee-4e73-a354-ccee4bd77938}",
  "creator": "{cda97709-b1a9-4544-9a8d-750f1b3a12c4}",
  "trigger": "push",
  "build_number": 239,
  "pipeline": "{0ac5e5c0-a233-4cb5-88b9-bcffc5fa4925}",
  "created_on": "2024-04-22T02:42:19.807Z",
  "target": {
    "type": "pipeline_ref_target",
    "selector": {
      "type": "branches",
      "pattern": "mas*"
    },
    "commit": {
      "hash": "e34ad324d1e4334777dca3e56c04eba36c2a68c6"
    },
    "ref_type": "branch",
    "ref_name": "master"
  },
  "pipelines_configuration": {
    "pipelines": {
      "branches": {
        "mas*": [
          {
            "step": {
              "name": "A step",
              "script": [
                "echo Running job..."
              ]
            }
          }
        ]
      }
    }
  }
}

PipelineTarget with selector

PipelineSelector

ImportedFrom

Example

Commit target with a selector of a custom build-and-test pipeline definition and resolved import metadata:

1
2
{
  "type": "pipeline_commit_target",
  "selector": {
    "type": "custom",
    "pattern": "build-and-test",
    "imported_from": {
      "repo_uuid": "{9d6433db-fbd2-4fcf-a7e0-621526b3161e}",
      "revision": "master",
      "pipelineName": "generic-build"
    }
  },
  "commit": {
    "hash": "82dec1c48c8d3aa75e9e246646894ed952216f3d"
  }
}

Filtered PipelinesConfiguration

The only difference is that in the request to the Dynamic Pipelines provider the definitions are filtered down to just the one matching the specified target, or none in case no pipelines matched that target.

Response payload

The Dynamic Pipelines provider is expected to modify (or retain) the configuration passed in the pipelines_configuration property and then return the outcome back.

Alternatively, provider may return an error message in the error property. In this case, the pipeline run will be considered failed, and the error message will be surfaced as the cause of the failure.

Response target behaviours

In order for a returned PipelinesConfiguration to be executed, it must contain a pipeline definition that meets the criteria necessary for it to match the target specified in the initial request. If this does not occur, there are two potential outcomes, depending on the trigger that initiated the process.

If the trigger == 'push' and the returned PipelinesConfiguration does not match the target specified in the initial request, the pipelines system will simply ignore the response and no pipeline run will be generated.

If the trigger == 'manual' or trigger == 'schedule' and the returned PipelinesConfiguration does not match the target specified in the initial request, the pipelines system would generate and display a “failed” pipeline run.

ProviderError

If provided, the error message will be shown to the user in the UI on the pipeline result screen.

Examples

A response with a modified pipeline definition:

1
2
{
  "pipelines_configuration": {
    "image": {
      "name": "enforced/image:latest"
    },
    "pipelines": {
      "branches": {
        "master": [
          {
            "step": {
              "name": "Updated step",
              "script": [
                "echo Running updated job..."
              ]
            }
          }
        ]
      }
    }
  }
}

A response with an error:

1
2
{
  "error": {
    "key": "provider.error.example",
    "message": "This error message was provided by Dynamic Pipelines provider"
  }
}