clusterdock package

clusterdock.models module

This module contains the main abstractions used by clusterdock topologies to bring up clusters.

class clusterdock.models.Cluster(*nodes)

The central abstraction for interacting with Docker container clusters. No Docker behavior is actually invoked until the start method is called.

Parameters:*nodes – One or more clusterdock.models.Node instances.

execute(command, **kwargs)

Execute a command on every clusterdock.models.Node within the

clusterdock.models.Cluster.

Parameters:

• command (str) – Command to execute.
• **kwargs – Additional keyword arguments to pass to clusterdock.models.Node.execute().

Returns:

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

start(network, pull_images=False, update_etc_hosts=True)

Start the cluster.

Parameters:

• network (str) – Name of the Docker network to use for the cluster.
• pull_images (bool, optional) – Pull every Docker image needed by every clusterdock.models.Node instance, even if it exists locally. Default: False
• update_etc_hosts (bool) – Update the /etc/hosts file on the host with the hostname and IP address of the container. Default: True

class clusterdock.models.Node(hostname, group, image, ports=None, volumes=None, devices=None, **create_container_kwargs)

Class representing a single cluster host.

Parameters:

• hostname (str) – Hostname of the node.
• group (str) – clusterdock.models.NodeGroup to which the node should belong.
• image (str) – Docker image with which to start the container.
• ports (list, optional) – A list of container ports to expose to the host. Elements of the list could be integers (in which case a random port on the host will be chosen by the Docker daemon) or dictionaries (with the key being the host port and the value being the container port). Default: None
• volumes (list, optional) – A list of volumes to create for the node. Elements of the list could be dictionaries of bind volumes (i.e. key: the absolute path on the host, value: the absolute path in the container) or strings representing the names of Docker images from which to get volumes. As an example, [{'/var/www': '/var/www'}, 'my_super_secret_image'] would create a bind mount of /var/www on the host and use any volumes from my_super_secret_image. Default: None
• devices (list, optional) – Devices on the host to expose to the node. Default: None
• **create_container_kwargs – Any other keyword arguments to pass directly to docker.api.container.create_container().

DEFAULT_CREATE_CONTAINER_KWARGS = {'detach': True, 'volumes': ['/etc/localtime']}

DEFAULT_CREATE_HOST_CONFIG_KWARGS = {'security_opt': ['seccomp=unconfined'], 'cap_add': ['ALL'], 'binds': {'/etc/localtime': {'mode': 'rw', 'bind': '/etc/localtime'}}}

commit(repository, tag=None, push=False, **kwargs)

Commit the Node’s Docker container to a Docker image.

Parameters:

• repository (str) – The Docker repository to commit the image to.
• tag (str, optional) – Docker image tag. Default: None
• push (bool, optional) – Push the image to Docker repository. Default: False
• **kwargs – Additional keyword arguments to pass to docker.models.Containers.Container.commit()

execute(command, user='root', quiet=False, detach=False)

Execute a command on the node.

Parameters:

• command (str) – Command to execute.
• user (str, optional) – User with which to execute the command. Default: root
• quiet (bool, optional) – Run the command without showing any output. Default: False
• detach (bool, optional) – Run the command in detached mode. Default: False

Returns:

A collections.namedtuple instance with exit_code and output attributes.

get_file(path)

Get file from the node.

Parameters:path (str) – Absolute path to file. Returns:A str containing the contents of the file.

put_file(path, contents)

Put file on the node.

Parameters:

• path (str) – Absolute path to file.
• contents (str) – The contents of the file.

start(network, cluster_name=None)

Start the node.

Parameters:

• network (str) – Docker network to which to attach the container.
• cluster_name (str, optional) – Cluster name to use for the Node. Default: None

stop(remove=True)

Stop the node and optionally removing the Docker container.

Parameters:remove (bool, optional) – Remove underlying Docker container. Default: True

class clusterdock.models.NodeGroup(name, *nodes)

Abstraction representing a collection of Nodes that it could be useful to interact with enmasse. For example, a typical HDFS cluster could be seen as consisting of a 1 node group consisting of hosts with NameNodes and an n-1 node group of hosts with DataNodes.

Parameters:

• name (str) – The name by which to refer to the group.
• *nodes – One or more clusterdock.models.Node instances.

execute(command, **kwargs)

Execute a command on every clusterdock.models.Node within the

clusterdock.models.NodeGroup.

Parameters:

• command (str) – Command to execute.
• **kwargs – Additional keyword arguments to pass to clusterdock.models.Node.execute().

Returns:

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

clusterdock.utils module

Various utilities to be used by other modules.

clusterdock.utils.DEFAULT_TIMEOUT = 60

clusterdock.utils.DEFAULT_TIME_BETWEEN_CHECKS = 1

clusterdock.utils.generate_cluster_name()

Generate a random cluster name.

clusterdock.utils.get_clusterdock_label(cluster_name=None)

Generate a clusterdock meta data label in json format. Meta data such as: clusterdock package name, version, location of clusterdock install, etc.

Args:

cluster_name (str, optional): Cluster name to attach to meta data label.

Default: None

Returns:

(json): clusterdock meta data label

clusterdock.utils.get_container(hostname)

Get running Docker container for a given hostname.

clusterdock.utils.get_containers(clusterdock=False)

Get Docker containers.

Parameters:clusterdock (bool, optional) – clusterdock containers only. Default: False Returns:List of containers. Return type:(list)

clusterdock.utils.join_url_parts(*parts)

Join a URL from a list of parts. See http://stackoverflow.com/questions/24814657 for examples of why urllib.parse.urljoin is insufficient for what we want to do.

clusterdock.utils.max_len_list_dict_item(list_dict, attr)

Returns max length of a given attribute from a list of dict items.

clusterdock.utils.nested_get(dict_, keys)

Utility function that returns the value of a sequence of nested keys.

Example

>>> details = {'name': {'first': {'english': 'Dima'}}}
>>> nested_get(details, ['name', 'first', 'english'])
'Dima'

Parameters:

• dict (dict) – Dictionary to access.
• keys (list) – A list of keys to access in a nested manner.

Returns:

The value.

clusterdock.utils.print_topology_meta(topology_name, quiet=False)

Given a toplogy name, relative to current directory, print its meta info.

clusterdock.utils.version_str(version)

Convert a version tuple or string to a string. Will return major.minor.release kind of format.

clusterdock.utils.version_tuple(version)

Convert a version string or tuple to a tuple. Will return (major, minor, release) kind of format.

clusterdock.utils.wait_for_condition(condition, condition_args=None, condition_kwargs=None, time_between_checks=1, timeout=60, time_to_success=0, success=None, failure=None)

Wait until a condition is satisfied (or timeout).

Parameters:

• condition – Callable to evaluate.
• condition_args (optional) – A list of args to pass to the condition. Default: None
• condition_kwargs (optional) – A dictionary of kwargs to pass to the condition. Default: None
• time_between_checks (int, optional) – Seconds between condition checks. Default: DEFAULT_TIME_BETWEEN_CHECKS
• timeout (int, optional) – Seconds to wait before timing out. Default: DEFAULT_TIMEOUT
• time_to_success (int, optional) – Seconds for the condition to hold true before it is considered satisfied. Default: 0
• success (optional) – Callable to invoke when condition succeeds. A time variable will be passed as an argument, so can be used. Default: None
• failure (optional) – Callable to invoke when timeout occurs. timeout will be passed as an argument. Default: None

Raises:

TimeoutError