clusterdock package

clusterdock.models module

This module contains the main abstractions used by clusterdock topologies to bring up clusters.

class clusterdock.models.Cluster(*nodes)[source]

The central abstraction for interacting with Docker container clusters. No Docker behavior is actually invoked until the start method is called.

Parameters:*nodes – One or more clusterdock.models.Node instances.
execute(command, **kwargs)[source]
Execute a command on every clusterdock.models.Node within the
clusterdock.models.Cluster.
Parameters:
Returns:

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

start(network, pull_images=False, update_etc_hosts=True)[source]

Start the cluster.

Parameters:
  • network (str) – Name of the Docker network to use for the cluster.
  • pull_images (bool, optional) – Pull every Docker image needed by every clusterdock.models.Node instance, even if it exists locally. Default: False
  • update_etc_hosts (bool) – Update the /etc/hosts file on the host with the hostname and IP address of the container. Default: True
class clusterdock.models.Node(hostname, group, image, ports=None, volumes=None, devices=None, environment=None, **create_container_kwargs)[source]

Class representing a single cluster host.

Parameters:
  • hostname (str) – Hostname of the node.
  • group (str) – clusterdock.models.NodeGroup to which the node should belong.
  • image (str) – Docker image with which to start the container.
  • ports (list, optional) – A list of container ports to expose to the host. Elements of the list could be integers (in which case a random port on the host will be chosen by the Docker daemon) or dictionaries (with the key being the host port and the value being the container port). Default: None
  • volumes (list, optional) – A list of volumes to create for the node. Elements of the list could be dictionaries of bind volumes (i.e. key: the absolute path on the host, value: the absolute path in the container) or strings representing the names of Docker images from which to get volumes. As an example, [{'/var/www': '/var/www'}, 'my_super_secret_image'] would create a bind mount of /var/www on the host and use any volumes from my_super_secret_image. Default: None
  • devices (list, optional) – Devices on the host to expose to the node. Default: None
  • **create_container_kwargs – Any other keyword arguments to pass directly to docker.api.container.create_container().
DEFAULT_CREATE_CONTAINER_KWARGS = {'detach': True, 'volumes': []}
DEFAULT_CREATE_HOST_CONFIG_KWARGS = {'cap_add': ['ALL'], 'security_opt': ['seccomp=unconfined']}
commit(repository, tag=None, push=False, **kwargs)[source]

Commit the Node’s Docker container to a Docker image.

Parameters:
  • repository (str) – The Docker repository to commit the image to.
  • tag (str, optional) – Docker image tag. Default: None
  • push (bool, optional) – Push the image to Docker repository. Default: False
  • **kwargs – Additional keyword arguments to pass to docker.models.Containers.Container.commit()
execute(command, user='root', quiet=False, detach=False)[source]

Execute a command on the node.

Parameters:
  • command (str) – Command to execute.
  • user (str, optional) – User with which to execute the command. Default: root
  • quiet (bool, optional) – Run the command without showing any output. Default: False
  • detach (bool, optional) – Run the command in detached mode. Default: False
Returns:

A collections.namedtuple instance with exit_code and output attributes.

get_file(path)[source]

Get file from the node.

Parameters:path (str) – Absolute path to file.
Returns:A str containing the contents of the file.
put_file(path, contents)[source]

Put file on the node.

Parameters:
  • path (str) – Absolute path to file.
  • contents (str) – The contents of the file.
start(network, cluster_name=None, pull_images=False)[source]

Start the node.

Parameters:
  • network (str) – Docker network to which to attach the container.
  • cluster_name (str, optional) – Cluster name to use for the Node. Default: None
  • pull_images (bool, optional) – Pull every Docker image needed by this node instance, even if it exists locally. Default: False
stop(remove=True)[source]

Stop the node and optionally removing the Docker container.

Parameters:remove (bool, optional) – Remove underlying Docker container. Default: True
class clusterdock.models.NodeGroup(name, *nodes)[source]

Abstraction representing a collection of Nodes that it could be useful to interact with enmasse. For example, a typical HDFS cluster could be seen as consisting of a 1 node group consisting of hosts with NameNodes and an n-1 node group of hosts with DataNodes.

Parameters:
  • name (str) – The name by which to refer to the group.
  • *nodes – One or more clusterdock.models.Node instances.
execute(command, **kwargs)[source]
Execute a command on every clusterdock.models.Node within the
clusterdock.models.NodeGroup.
Parameters:
Returns:

A collections.OrderedDict of str instances (the FQDN of the node)

mapping to the collections.namedtuple instances returned by clusterdock.models.Node.execute().

clusterdock.utils module

Various utilities to be used by other modules.

clusterdock.utils.DEFAULT_TIMEOUT = 60
clusterdock.utils.DEFAULT_TIME_BETWEEN_CHECKS = 1
clusterdock.utils.generate_cluster_name()[source]

Generate a random cluster name.

clusterdock.utils.get_clusterdock_label(cluster_name=None)[source]

Generate a clusterdock meta data label in json format. Meta data such as: clusterdock package name, version, location of clusterdock install, etc.

Args:
cluster_name (str, optional): Cluster name to attach to meta data label.
Default: None
Returns:
(json): clusterdock meta data label
clusterdock.utils.get_container(hostname)[source]

Get running Docker container for a given hostname.

clusterdock.utils.get_containers(clusterdock=False)[source]

Get Docker containers.

Parameters:clusterdock (bool, optional) – clusterdock containers only. Default: False
Returns:List of containers.
Return type:(list)
clusterdock.utils.join_url_parts(*parts)[source]

Join a URL from a list of parts. See http://stackoverflow.com/questions/24814657 for examples of why urllib.parse.urljoin is insufficient for what we want to do.

clusterdock.utils.max_len_list_dict_item(list_dict, attr)[source]

Returns max length of a given attribute from a list of dict items.

clusterdock.utils.nested_get(dict_, keys)[source]

Utility function that returns the value of a sequence of nested keys.

Example

>>> details = {'name': {'first': {'english': 'Dima'}}}
>>> nested_get(details, ['name', 'first', 'english'])
'Dima'
Parameters:
  • dict (dict) – Dictionary to access.
  • keys (list) – A list of keys to access in a nested manner.
Returns:

The value.

clusterdock.utils.print_topology_meta(topology_name, quiet=False)[source]

Given a toplogy name, relative to current directory, print its meta info.

clusterdock.utils.version_str(version)[source]

Convert a version tuple or string to a string. Will return major.minor.release kind of format.

clusterdock.utils.version_tuple(version)[source]

Convert a version string or tuple to a tuple. Will return (major, minor, release) kind of format.

clusterdock.utils.wait_for_condition(condition, condition_args=None, condition_kwargs=None, time_between_checks=1, timeout=60, time_to_success=0, success=None, failure=None)[source]

Wait until a condition is satisfied (or timeout).

Parameters:
  • condition – Callable to evaluate.
  • condition_args (optional) – A list of args to pass to the condition. Default: None
  • condition_kwargs (optional) – A dictionary of kwargs to pass to the condition. Default: None
  • time_between_checks (int, optional) – Seconds between condition checks. Default: DEFAULT_TIME_BETWEEN_CHECKS
  • timeout (int, optional) – Seconds to wait before timing out. Default: DEFAULT_TIMEOUT
  • time_to_success (int, optional) – Seconds for the condition to hold true before it is considered satisfied. Default: 0
  • success (optional) – Callable to invoke when condition succeeds. A time variable will be passed as an argument, so can be used. Default: None
  • failure (optional) – Callable to invoke when timeout occurs. timeout will be passed as an argument. Default: None
Raises:

TimeoutError