flambe.runnable

Package Contents

class flambe.runnable.Runnable(user_provider: Callable[[], str] = None, **kwargs)[source]

Bases: flambe.compile.MappedRegistrable

Base class for all runnables.

A runnable contains a single run method that needs to be implemented. It must contain all the logic for the runnable.

Each runnable has also access to the secrets the user provides.

Examples of Runnables: Experiment, Cluster

config

The secrets that the user provides. For example, ‘config[“AWS”][“ACCESS_KEY”]’

Type:configparser.ConfigParser
extensions

The extensions used for this runnable.

Type:Dict[str, str]
content

This attribute will hold the YAML representation of the Runnable.

Type:Optional[str]
user_provider

The logic for specifying the user triggering this Runnable. If not passed, by default it will pick the computer’s user.

Type:Callable[[], str]
inject_content(self, content: str)

Inject the original YAML string that was used to generate this Runnable instance.

Parameters:content (str) – The YAML, as a string
inject_secrets(self, secrets: str)

Inject the secrets once the Runnable was created.

Parameters:secrets (str) – The filepath to the secrets
inject_extensions(self, extensions: Dict[str, str])

Inject extensions to the Runnable

Parameters:extensions (Dict[str, str]) – The extensions
run(self, **kwargs)

Run the runnable.

Each implementation will implement its own logic, with the parameters it needs.

parse(self)

Parse the runnable to determine if it’s able to run. :raises: ParsingExperimentError – In case a parsing error is found.

class flambe.runnable.ClusterRunnable(user_provider: Callable[[], str] = None, env: Optional[RemoteEnvironment] = None, **kwargs)[source]

Bases: flambe.runnable.Runnable

Base class for all runnables that are able to run on cluster.

This type of Runnables must include logic in the ‘run’ method to deal with the fact that they could be running in a distributed cluster of machines.

To provide useful information about the cluster, a RemoteEnvironment object will be injected when running remotely.

config

The secrets that the user provides. For example, ‘config[“AWS”][“ACCESS_KEY”]’

Type:configparser.ConfigParser
env

The remote environment has information about the cluster where this ClusterRunnable will be running. IMPORTANT: this object will be available only when the ClusterRunnable is running remotely.

Type:RemoteEnvironment
user_provider

The logic for specifying the user triggering this Runnable. If not passed, by default it will pick the computer’s user.

Type:Callable[[], str]
setup(self, cluster: Cluster, extensions: Dict[str, str], force: bool, **kwargs)

Setup the cluster.

Parameters:
  • cluster (Cluster) – The cluster where this Runnable will be running
  • extensions (Dict[str, str]) – The ClusterRunnable extensions
  • force (bool) – The force value provided to Flambe
setup_inject_env(self, cluster: Cluster, extensions: Dict[str, str], force: bool, **kwargs)

Call setup and inject the RemoteEnvironment

Parameters:
  • cluster (Cluster) – The cluster where this Runnable will be running
  • extensions (Dict[str, str]) – The ClusterRunnable extensions
  • force (bool) – The force value provided to Flambe
set_serializable_attr(self, attr, value)

Set an attribute while keep supporting serializaton.

class flambe.runnable.SafeExecutionContext(yaml_file: str)[source]

Context manager handling the experiment’s creation and execution.

Parameters:yaml_file (str) – The experiment filename
__enter__(self)

A SafeExecutionContext should be used as a context manager to handle all possible errors in a clear way.

Examples

>>> with SafeExecutionContext(...) as ex:
>>>     ...
__exit__(self, exc_type: Optional[Type[BaseException]], exc_value: Optional[BaseException], tb: Optional[TracebackType])

Exit method for the context manager.

This method will catch any exception, and return True. This means that all exceptions produced in a SafeExecutionContext (used with the context manager) will not continue to raise.

preprocess(self, secrets: Optional[str] = None, download_ext: bool = True, install_ext: bool = False, import_ext: bool = True, check_tags: bool = True, **kwargs)

Preprocess the runnable file.

Looks for syntax errors, import errors, etc. Also injects the secrets into the runnables.

If this method runs and ends without exceptions, then the experiment is ok to be run. If this method raises an Error and the SafeExecutionContext is used as context manager, then the __exit__ method will be executed.

Parameters:
  • secrets (Optional[str]) – Optional path to the secrets file
  • install_ext (bool) – Whether to install the extensions or not. This process also downloads the remote extensions. Defaults to False
  • install_ext – Whether to import the extensions or not. Defaults to True.
  • check_tags (bool) – Whether to check that all tags are valid. Defaults to True.
Returns:

A tuple containing the compiled Runnable and a dict containing the extensions the Runnable uses.

Return type:

Tuple[Runnable, Dict[str, str]]

Raises:

Exception – Depending on the error.

first_parse(self)

Check if valid YAML file and also load config

In this first parse the runnable does not get compiled because it could be a custom Runnable, so it needs the extensions to be imported first.

check_tags(self, content: str)

Check that all the tags are valid.

Parameters:content (str) – The content of the YAML file
Raises:TagError
compile_runnable(self, content: str)

Compiles and returns the Runnable.

IMPORTANT: This method should run after all extensions were registered.

Parameters:content (str) – The runnable, as a YAML string
Returns:The compiled experiment.
Return type:Runnable
class flambe.runnable.RemoteEnvironment(key: str, orchestrator_ip: str, factories_ips: List[str], user: str, local_user: str, public_orchestrator_ip: Optional[str] = None, public_factories_ips: Optional[List[str]] = None, **kwargs)[source]

Bases: flambe.compile.Registrable

This objects contains information about the cluster

This object will be available on the remote execution of the ClusterRunnable (as an attribute).

IMPORTANT: this object needs to be serializable, hence it Needs to be created using ‘compile’ method.

key

The key that communicates the cluster

Type:str
orchestrator_ip

The orchestrator visible IP for the factories (usually the private IP)

Type:str
factories_ips

The list of factories IPs visible for other factories and orchestrator (usually private IPs)

Type:List[str]
user

The username of all machines. This implementations assumes same username for all machines

Type:str
local_user

The username of the local process that launched the cluster

Type:str
public_orchestrator_ip

The public orchestrator IP, if available.

Type:Optional[str]
public_factories_ips

The public factories IPs, if available.

Type:Optional[List[str]]
classmethod to_yaml(cls, representer: Any, node: Any, tag: str)

Use representer to create yaml representation of node

classmethod from_yaml(cls, constructor: Any, node: Any, factory_name: str)

Use constructor to create an instance of cls