Runnable(user_provider: Callable[, str] = None, **kwargs)¶
Base class for all runnables.
A runnable contains a single run method that needs to be implemented. It must contain all the logic for the runnable.
Each runnable has also access to the secrets the user provides.
Examples of Runnables: Experiment, Cluster
The secrets that the user provides. For example, ‘config[“AWS”][“ACCESS_KEY”]’
The extensions used for this runnable.
Type: Dict[str, str]
This attribute will hold the YAML representation of the Runnable.
The logic for specifying the user triggering this Runnable. If not passed, by default it will pick the computer’s user.
Type: Callable[, str]
inject_content(self, content: str)¶
Inject the original YAML string that was used to generate this Runnable instance.
Parameters: content (str) – The YAML, as a string
inject_secrets(self, secrets: str)¶
Inject the secrets once the Runnable was created.
Parameters: secrets (str) – The filepath to the secrets
inject_extensions(self, extensions: Dict[str, str])¶
Inject extensions to the Runnable
Parameters: extensions (Dict[str, str]) – The extensions
Run the runnable.
Each implementation will implement its own logic, with the parameters it needs.
Parse the runnable to determine if it’s able to run. :raises:
ParsingExperimentError– In case a parsing error is found.
Context manager handling the experiment’s creation and execution.
Parameters: yaml_file (str) – The experiment filename
A SafeExecutionContext should be used as a context manager to handle all possible errors in a clear way.
>>> with SafeExecutionContext(...) as ex: >>> ...
__exit__(self, exc_type: Optional[Type[BaseException]], exc_value: Optional[BaseException], tb: Optional[TracebackType])¶
Exit method for the context manager.
This method will catch any exception, and return True. This means that all exceptions produced in a SafeExecutionContext (used with the context manager) will not continue to raise.
preprocess(self, secrets: Optional[str] = None, download_ext: bool = True, install_ext: bool = False, import_ext: bool = True, check_tags: bool = True, **kwargs)¶
Preprocess the runnable file.
Looks for syntax errors, import errors, etc. Also injects the secrets into the runnables.
If this method runs and ends without exceptions, then the experiment is ok to be run. If this method raises an Error and the SafeExecutionContext is used as context manager, then the __exit__ method will be executed.
- secrets (Optional[str]) – Optional path to the secrets file
- install_ext (bool) – Whether to install the extensions or not. This process also downloads the remote extensions. Defaults to False
- install_ext – Whether to import the extensions or not. Defaults to True.
- check_tags (bool) – Whether to check that all tags are valid. Defaults to True.
A tuple containing the compiled Runnable and a dict containing the extensions the Runnable uses.
Tuple[Runnable, Dict[str, str]]
Exception– Depending on the error.
Check if valid YAML file and also load config
In this first parse the runnable does not get compiled because it could be a custom Runnable, so it needs the extensions to be imported first.
Check that all the tags are valid.
Parameters: content (str) – The content of the YAML file Raises:
RemoteEnvironment(key: str, orchestrator_ip: str, factories_ips: List[str], user: str, local_user: str, public_orchestrator_ip: Optional[str] = None, public_factories_ips: Optional[List[str]] = None, **kwargs)¶
This objects contains information about the cluster
This object will be available on the remote execution of the ClusterRunnable (as an attribute).
IMPORTANT: this object needs to be serializable, hence it Needs to be created using ‘compile’ method.
The key that communicates the cluster
The orchestrator visible IP for the factories (usually the private IP)
The list of factories IPs visible for other factories and orchestrator (usually private IPs)
The username of all machines. This implementations assumes same username for all machines
The username of the local process that launched the cluster
The public orchestrator IP, if available.
The public factories IPs, if available.
to_yaml(cls, representer: Any, node: Any, tag: str)¶
Use representer to create yaml representation of node
from_yaml(cls, constructor: Any, node: Any, factory_name: str)¶
Use constructor to create an instance of cls