Creating Templates

Sovereign uses a templating system to make it easier to turn data into Envoy configuration.

Using templates you can represent your envoy configuration dynamically.

This page focuses more on getting a static template up and running before moving onto dynamic config.

Discovery Types

A template must be configured for each discovery type that you want to provide from your sovereign server.

As of this writing, some of the known xDS resource types are:

• clusters
• endpoints
• listeners
• routes
• secrets
• scoped_routes
• extension_configs
• runtime

However, sovereign allows you to define a template for any resource type, even ones that are not specific to Envoy. In this way, as long as you know the resource type that Envoy will ask for, you can serve configuration for it from sovereign.

Writing templates

Writing a template is similar to how you would write the bootstrap configuration for envoy.

The key difference is that instead of providing static configuration that envoy reads when it starts, you are now providing lists of resources that envoy will fetch dynamically either all at once or one-by-one using a resource name.

Start by creating a directory at templates/default/ where each template will be stored.

1
2
resources:
{% if "my_cluster" in resource_names %}
- '@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: my_cluster
  connect_timeout: 0.25s
  type: STRICT_DNS
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
  load_assignment:
    cluster_name: my_cluster
    endpoints:
      - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: edgediag.services.atlassian.com
                  port_value: 443
{% endif %}

Notice in the template that it checks if "my_cluster" is contained in resource_names. This is a special variable that is included in all templates to inform you what resources the proxy has requested. For clusters (and listeners) this check is actually irrelevant because envoy requests all resources for those two discovery types. For routes, secrets, endpoints, and other types, envoy will request specific resources.

Once you've written your first template, you can repeat this process for all other discovery types that you wish to support.

Typing your templates

In the above examples, we used YAML and Python dictionaries to render envoy configuration. While this is quick and easy, it has one significant downside which is that there's no way of knowing whether it is valid envoy configuration until it is given to envoy and either rejected or accepted.

You could go ahead and create your own datastructures or classes to handle this, however I have already gone ahead and compiled the envoy protobufs into python objects in an open-source library called envoy_data_plane.

Expanding on the above Python example, you can add this dependency to your project and then use it in templates like so:

1
2
from datetime import timedelta
from envoy_data_plane.envoy.config.core.v3 import (
    Address,
    SocketAddress,
    TransportSocket,
)
from envoy_data_plane.envoy.config.cluster.v3 import (
    Cluster,
    ClusterDiscoveryType,
)
from envoy_data_plane.envoy.config.endpoint.v3 import (
    Endpoint,
    LbEndpoint,
    LocalityLbEndpoints,
    ClusterLoadAssignment,
)
from envoy_data_plane.envoy.extensions.transport_sockets.tls.v3 import (
    CommonTlsContext,
    UpstreamTlsContext,
)


class TypedConfig(dict):
    _serialized_on_wire = True

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if "type" in self:
            self["@type"] = self.pop("type")

    def to_dict(self, *args, **kwargs):
        return self


def endpoint(address: str, port: int) -> LbEndpoint:
    return LbEndpoint(
        endpoint=Endpoint(
            address=Address(
                socket_address=SocketAddress(
                    address=address,
                    port_value=port
                )
            )
        )
    )

def call(discovery_request, resource_names, **kwargs):
    if "my_cluster" in resource_names:
        resource = Cluster(
            name="my_cluster",
            connect_timeout=timedelta(milliseconds=250),
            type=ClusterDiscoveryType.STRICT_DNS,
            transport_socket=TransportSocket(
                name="envoy.transport_sockets.tls",
                typed_config=TypedConfig(  # type: ignore
                    UpstreamTlsContext(
                        common_tls_context=CommonTlsContext(),
                    ).to_dict(),
                    type="type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                ),
            ),
            load_assignment=ClusterLoadAssignment(
                cluster_name="my_cluster",
                endpoints=[
                    LocalityLbEndpoints(
                        lb_endpoints=[endpoint("edgediag.services.atlassian.com", 443)]
                    )
                ]
            )
        ).to_dict()
        resource['@type'] = 'type.googleapis.com/envoy.config.cluster.v3.Cluster'
        yield resource

This will produce the same output as the un-typed Python example, but with the added benefit of raising an exception if some of the arguments provided are not valid for the field, or if some field has a typo.

This comes at the slight cost of having to build and validate the object and all its fields which may have a slight performance overhead. The code itself is also longer and could be considered more complex, but that is a natural consequence of it being more rigorous.

Adding the templates to configuration

Once you have finalized the contents of your templates, they can be added to Sovereign via it's main configuration file that we created in the previous section:

1
2
templates:
  default:
    type: clusters
    spec: 
      protocol: file
      path: templates/default/clusters.yaml

Versioning templates

In the previous configuration examples you'll notice a key called default.
This means that if the Envoy version does not match a specifically configured version, Sovereign will use the default templates to generate configuration for the Node.

If your fleet of Envoys contains multiple different versions, and the same template wouldn't work across all of them due to backward compatibility issues, you can configure templates for each version of Envoy that you need to support

1
2
templates:
  # Sovereign will match all patch versions of an envoy release when given a short version as follows
  1.13: # Versions 1.13.0, 1.13.1, etc
    type: clusters
    spec: 
      protocol: file
      path: templates/v13/clusters.yaml
  
  1.12: # Versions 1.12.0, 1.12.1, 1.12.2, etc
    type: clusters
    spec:
      protocol: file
      path: templates/v12/clusters.yaml
  
  # Everything that doesn't match will use this
  default:
    type: clusters
    spec:
      protocol: file
      path: templates/default/clusters.yaml

Now that we have the clusters discovery endpoint implemented, we can start to make it dynamic by adding context.