clusterdock package

clusterdock.models module

This module contains the main abstractions used by clusterdock topologies to bring up clusters.

class clusterdock.models.Cluster(*nodes)

The central abstraction for interacting with Docker container clusters. No Docker behavior is actually invoked until the start method is called.

Parameters:*nodes – One or more clusterdock.models.Node instances.

execute(command, **kwargs)

Execute a command on every clusterdock.models.Node within the

clusterdock.models.Cluster.

Parameters:

• command (str) – Command to execute.
• **kwargs – Additional keyword arguments to pass to clusterdock.models.Node.execute().

Returns:

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

start(network, pull_images=False, update_etc_hosts=True)

Start the cluster.

Parameters:

• network (str) – Name of the Docker network to use for the cluster.
• pull_images (bool, optional) – Pull every Docker image needed by every clusterdock.models.Node instance, even if it exists locally. Default: False
• update_etc_hosts (bool) – Update the /etc/hosts file on the host with the hostname and IP address of the container. Default: True

class clusterdock.models.Node(hostname, group, image, ports=None, volumes=None, devices=None, environment=None, **create_container_kwargs)

Class representing a single cluster host.

Parameters:

• hostname (str) – Hostname of the node.
• group (str) – clusterdock.models.NodeGroup to which the node should belong.
• image (str) – Docker image with which to start the container.
• ports (list, optional) – A list of container ports to expose to the host. Elements of the list could be integers (in which case a random port on the host will be chosen by the Docker daemon) or dictionaries (with the key being the host port and the value being the container port). Default: None
• volumes (list, optional) – A list of volumes to create for the node. Elements of the list could be dictionaries of bind volumes (i.e. key: the absolute path on the host, value: the absolute path in the container) or strings representing the names of Docker images from which to get volumes. As an example, [{'/var/www': '/var/www'}, 'my_super_secret_image'] would create a bind mount of /var/www on the host and use any volumes from my_super_secret_image. Default: None
• devices (list, optional) – Devices on the host to expose to the node. Default: None
• **create_container_kwargs – Any other keyword arguments to pass directly to docker.api.container.create_container().

DEFAULT_CREATE_CONTAINER_KWARGS = {'detach': True, 'volumes': []}

DEFAULT_CREATE_HOST_CONFIG_KWARGS = {'cap_add': ['ALL'], 'security_opt': ['seccomp=unconfined']}

commit(repository, tag=None, push=False, **kwargs)

Commit the Node’s Docker container to a Docker image.

Parameters:

• repository (str) – The Docker repository to commit the image to.
• tag (str, optional) – Docker image tag. Default: None
• push (bool, optional) – Push the image to Docker repository. Default: False
• **kwargs – Additional keyword arguments to pass to docker.models.Containers.Container.commit()

execute(command, user='root', quiet=False, detach=False)

Execute a command on the node.

Parameters:

• command (str) – Command to execute.
• user (str, optional) – User with which to execute the command. Default: root
• quiet (bool, optional) – Run the command without showing any output. Default: False
• detach (bool, optional) – Run the command in detached mode. Default: False

Returns:

A collections.namedtuple instance with exit_code and output attributes.

get_file(path)

Get file from the node.

Parameters:path (str) – Absolute path to file. Returns:A str containing the contents of the file.

put_file(path, contents)

Put file on the node.

Parameters:

• path (str) – Absolute path to file.
• contents – The contents of the file in str or bytes form.

start(network, cluster_name=None, pull_images=False)

Start the node.

Parameters:

• network (str) – Docker network to which to attach the container.
• cluster_name (str, optional) – Cluster name to use for the Node. Default: None
• pull_images (bool, optional) – Pull every Docker image needed by this node instance, even if it exists locally. Default: False

stop(remove=True)

Stop the node and optionally removing the Docker container.

Parameters:remove (bool, optional) – Remove underlying Docker container. Default: True

class clusterdock.models.NodeGroup(name, *nodes)

Abstraction representing a collection of Nodes that it could be useful to interact with enmasse. For example, a typical HDFS cluster could be seen as consisting of a 1 node group consisting of hosts with NameNodes and an n-1 node group of hosts with DataNodes.

Parameters:

• name (str) – The name by which to refer to the group.
• *nodes – One or more clusterdock.models.Node instances.

execute(command, **kwargs)

Execute a command on every clusterdock.models.Node within the

clusterdock.models.NodeGroup.

Parameters:

• command (str) – Command to execute.
• **kwargs – Additional keyword arguments to pass to clusterdock.models.Node.execute().

Returns:

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

clusterdock.utils module

Various utilities to be used by other modules.

clusterdock.utils.DEFAULT_TIMEOUT = 60

clusterdock.utils.DEFAULT_TIME_BETWEEN_CHECKS = 1

class clusterdock.utils.Version(version)

Maven version string abstraction.

Use this class to enable correct comparison of Maven versioned projects. For our purposes, any version is equivalent to any other version that has the same 4-digit version number (i.e. 3.0.0.0-SNAPSHOT == 3.0.0.0-RC2 == 3.0.0.0).

Parameters:version (str) or (int) or (float) – Version string (e.g. ‘2.5.0.0-SNAPSHOT’).

class clusterdock.utils.VersionSplit(name, delimiter1, version, delimiter2, specifier)

Util function to hold various parts of a version.

Parameters:

• name (str) –
• delimiter1 (str) –
• version (str) –
• delimiter2 (str) –
• specifier (str) –

delimiter1

delimiter2

name

specifier

version

clusterdock.utils.generate_cluster_name()

Generate a random cluster name.

clusterdock.utils.get_clusterdock_label(cluster_name=None)

Generate a clusterdock meta data label in json format. Meta data such as: clusterdock package name, version, location of clusterdock install, etc.

Args:

cluster_name (str, optional): Cluster name to attach to meta data label.

Default: None

Returns:

(json): clusterdock meta data label

clusterdock.utils.get_container(hostname)

Get running Docker container for a given hostname.

clusterdock.utils.get_containers(clusterdock=False)

Get Docker containers.

Parameters:clusterdock (bool, optional) – clusterdock containers only. Default: False Returns:List of containers. Return type:(list)

clusterdock.utils.join_url_parts(*parts)

Join a URL from a list of parts. See http://stackoverflow.com/questions/24814657 for examples of why urllib.parse.urljoin is insufficient for what we want to do.

clusterdock.utils.max_len_list_dict_item(list_dict, attr)

Returns max length of a given attribute from a list of dict items.

clusterdock.utils.nested_get(dict_, keys)

Utility function that returns the value of a sequence of nested keys.

Example

>>> details = {'name': {'first': {'english': 'Dima'}}}
>>> nested_get(details, ['name', 'first', 'english'])
'Dima'

Parameters:

• dict (dict) – Dictionary to access.
• keys (list) – A list of keys to access in a nested manner.

Returns:

The value.

clusterdock.utils.print_topology_meta(topology_name, quiet=False)

Given a toplogy name, relative to current directory, print its meta info.

clusterdock.utils.version_str(version)

Convert a version tuple or string to a string. Will return major.minor.release kind of format.

clusterdock.utils.version_tuple(version)

Convert a version string or tuple to a tuple. Will return (major, minor, release) kind of format.

clusterdock.utils.wait_for_condition(condition, condition_args=None, condition_kwargs=None, time_between_checks=1, timeout=60, time_to_success=0, success=None, failure=None)

Wait until a condition is satisfied (or timeout).

Parameters:

• condition – Callable to evaluate.
• condition_args (optional) – A list of args to pass to the condition. Default: None
• condition_kwargs (optional) – A dictionary of kwargs to pass to the condition. Default: None
• time_between_checks (int, optional) – Seconds between condition checks. Default: DEFAULT_TIME_BETWEEN_CHECKS
• timeout (int, optional) – Seconds to wait before timing out. Default: DEFAULT_TIMEOUT
• time_to_success (int, optional) – Seconds for the condition to hold true before it is considered satisfied. Default: 0
• success (optional) – Callable to invoke when condition succeeds. A time variable will be passed as an argument, so can be used. Default: None
• failure (optional) – Callable to invoke when timeout occurs. timeout will be passed as an argument. Default: None

Raises:

TimeoutError