clusterdock package

clusterdock.models module

This module contains the main abstractions used by clusterdock topologies to bring up clusters.

class clusterdock.models.Cluster(*nodes)[source]

The central abstraction for interacting with Docker container clusters. No Docker behavior is actually invoked until the start method is called.

Parameters:*nodes – One or more clusterdock.models.Node instances.
execute(command, **kwargs)[source]
Execute a command on every clusterdock.models.Node within the

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

start(network, pull_images=False, update_etc_hosts=True)[source]

Start the cluster.

  • network (str) – Name of the Docker network to use for the cluster.
  • pull_images (bool, optional) – Pull every Docker image needed by every clusterdock.models.Node instance, even if it exists locally. Default: False
  • update_etc_hosts (bool) – Update the /etc/hosts file on the host with the hostname and IP address of the container. Default: True
class clusterdock.models.Node(hostname, group, image, ports=None, volumes=None, devices=None, environment=None, **create_container_kwargs)[source]

Class representing a single cluster host.

  • hostname (str) – Hostname of the node.
  • group (str) – clusterdock.models.NodeGroup to which the node should belong.
  • image (str) – Docker image with which to start the container.
  • ports (list, optional) – A list of container ports to expose to the host. Elements of the list could be integers (in which case a random port on the host will be chosen by the Docker daemon) or dictionaries (with the key being the host port and the value being the container port). Default: None
  • volumes (list, optional) – A list of volumes to create for the node. Elements of the list could be dictionaries of bind volumes (i.e. key: the absolute path on the host, value: the absolute path in the container) or strings representing the names of Docker images from which to get volumes. As an example, [{'/var/www': '/var/www'}, 'my_super_secret_image'] would create a bind mount of /var/www on the host and use any volumes from my_super_secret_image. Default: None
  • devices (list, optional) – Devices on the host to expose to the node. Default: None
  • **create_container_kwargs – Any other keyword arguments to pass directly to docker.api.container.create_container().
DEFAULT_CREATE_CONTAINER_KWARGS = {'detach': True, 'volumes': []}
DEFAULT_CREATE_HOST_CONFIG_KWARGS = {'cap_add': ['ALL'], 'security_opt': ['seccomp=unconfined']}
commit(repository, tag=None, push=False, **kwargs)[source]

Commit the Node’s Docker container to a Docker image.

  • repository (str) – The Docker repository to commit the image to.
  • tag (str, optional) – Docker image tag. Default: None
  • push (bool, optional) – Push the image to Docker repository. Default: False
  • **kwargs – Additional keyword arguments to pass to docker.models.Containers.Container.commit()
execute(command, user='root', quiet=False, detach=False)[source]

Execute a command on the node.

  • command (str) – Command to execute.
  • user (str, optional) – User with which to execute the command. Default: root
  • quiet (bool, optional) – Run the command without showing any output. Default: False
  • detach (bool, optional) – Run the command in detached mode. Default: False

A collections.namedtuple instance with exit_code and output attributes.


Get file from the node.

Parameters:path (str) – Absolute path to file.
Returns:A str containing the contents of the file.
put_file(path, contents)[source]

Put file on the node.

  • path (str) – Absolute path to file.
  • contents – The contents of the file in str or bytes form.
start(network, cluster_name=None, pull_images=False)[source]

Start the node.

  • network (str) – Docker network to which to attach the container.
  • cluster_name (str, optional) – Cluster name to use for the Node. Default: None
  • pull_images (bool, optional) – Pull every Docker image needed by this node instance, even if it exists locally. Default: False

Stop the node and optionally removing the Docker container.

Parameters:remove (bool, optional) – Remove underlying Docker container. Default: True
class clusterdock.models.NodeGroup(name, *nodes)[source]

Abstraction representing a collection of Nodes that it could be useful to interact with enmasse. For example, a typical HDFS cluster could be seen as consisting of a 1 node group consisting of hosts with NameNodes and an n-1 node group of hosts with DataNodes.

  • name (str) – The name by which to refer to the group.
  • *nodes – One or more clusterdock.models.Node instances.
execute(command, **kwargs)[source]
Execute a command on every clusterdock.models.Node within the

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

clusterdock.utils module

Various utilities to be used by other modules.

clusterdock.utils.DEFAULT_TIMEOUT = 60
clusterdock.utils.DEFAULT_TIME_BETWEEN_CHECKS = 1
class clusterdock.utils.Version(version)[source]

Maven version string abstraction.

Use this class to enable correct comparison of Maven versioned projects. For our purposes, any version is equivalent to any other version that has the same 4-digit version number (i.e. == ==

Parameters:version (str) or (int) or (float) – Version string (e.g. ‘’).
class clusterdock.utils.VersionSplit(name, delimiter1, version, delimiter2, specifier)[source]

Util function to hold various parts of a version.

  • name (str) –
  • delimiter1 (str) –
  • version (str) –
  • delimiter2 (str) –
  • specifier (str) –

Generate a random cluster name.


Generate a clusterdock meta data label in json format. Meta data such as: clusterdock package name, version, location of clusterdock install, etc.

cluster_name (str, optional): Cluster name to attach to meta data label.
Default: None
(json): clusterdock meta data label

Get running Docker container for a given hostname.


Get Docker containers.

Parameters:clusterdock (bool, optional) – clusterdock containers only. Default: False
Returns:List of containers.
Return type:(list)

Join a URL from a list of parts. See for examples of why urllib.parse.urljoin is insufficient for what we want to do.

clusterdock.utils.max_len_list_dict_item(list_dict, attr)[source]

Returns max length of a given attribute from a list of dict items.

clusterdock.utils.nested_get(dict_, keys)[source]

Utility function that returns the value of a sequence of nested keys.


>>> details = {'name': {'first': {'english': 'Dima'}}}
>>> nested_get(details, ['name', 'first', 'english'])
  • dict (dict) – Dictionary to access.
  • keys (list) – A list of keys to access in a nested manner.

The value.

clusterdock.utils.print_topology_meta(topology_name, quiet=False)[source]

Given a toplogy name, relative to current directory, print its meta info.


Convert a version tuple or string to a string. Will return major.minor.release kind of format.


Convert a version string or tuple to a tuple. Will return (major, minor, release) kind of format.

clusterdock.utils.wait_for_condition(condition, condition_args=None, condition_kwargs=None, time_between_checks=1, timeout=60, time_to_success=0, success=None, failure=None)[source]

Wait until a condition is satisfied (or timeout).

  • condition – Callable to evaluate.
  • condition_args (optional) – A list of args to pass to the condition. Default: None
  • condition_kwargs (optional) – A dictionary of kwargs to pass to the condition. Default: None
  • time_between_checks (int, optional) – Seconds between condition checks. Default: DEFAULT_TIME_BETWEEN_CHECKS
  • timeout (int, optional) – Seconds to wait before timing out. Default: DEFAULT_TIMEOUT
  • time_to_success (int, optional) – Seconds for the condition to hold true before it is considered satisfied. Default: 0
  • success (optional) – Callable to invoke when condition succeeds. A time variable will be passed as an argument, so can be used. Default: None
  • failure (optional) – Callable to invoke when timeout occurs. timeout will be passed as an argument. Default: None