Runnables

Runnables are top level objects that flambé is able to run. Experiment is an example of a Runnable.

Implementing Runnables is as easy as implementing the Runnable interface. Esentially, it requires a single method run().

Hint

A Runnable object can be executed by flambé:

flambe runnable.yaml

For example, let’s imagine we want to implement an S3Pusher that takes an Experiment output folder and uploads the content to a specific S3 bucket:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
from flambe.runnable import Runnable

class S3Pusher(Runnable):

    def __init__(self, experiment_path: str, bucket_name: str) -> None:
        super().__init__()
        self.experiment_path = experiment_path
        self.bucket_name = bucket_name

    def run(self, **kwargs) -> None:
        """Upload a local folder to a S3 bucket"""

        # Code to upload to S3 bucket
        S3_client = boto3.client("s3")
        for root, dirs, files in os.walk(self.experiment_path):
            for f in files:
                s3C.upload_file(os.path.join(root, f), self.bucketname, f)

This class definition can now be included in an extension (read more about extensions in Extensions) and used as a top level object in a YAML file.

ext: /path/to/S3Pusher/extension:
---

!ext.S3Pusher
    experiment_path: /path/to/my/experiment
    bucket_name: my-bucket

Then, simply execute:

flambe s3pusher.yaml

Providing secrets

All Runnables have access to secret information that the users can share via an ini file.

By executing the Runnable with a --secrets parameter, then the Runnable can access the secrets through the config attribute:

Let’s say that our S3Pusher needs to access the AWS_SECRET_TOKEN. Then:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from flambe.runnable import Runnable

class S3Pusher(Runnable):

    def __init__(self, experiment_path: str, bucket_name: str) -> None:
        # Same as before
        ...

    def run(self, **kwargs) -> None:
        """Upload a local folder to a S3 bucket"""

        # Code to upload to S3 bucket
        S3_client = boto3.client("s3", token=self.config['AWS']['AWS_SECRET_TOKEN'])
        for root, dirs, files in os.walk(self.experiment_path):
            for file in files:
                s3C.upload_file(os.path.join(root, file), self.bucketname, file)

Then if secrets.ini contains:

[AWS]
AWS_SECRET_TOKEN = ABCDEFGHI123456789

We can execute:

flambe s3pusher.yaml --secrets secret.ini

Automatic extensions installation

Important

To understand this section you should be familiar with extensions. For information about extensions, go to Extensions.

When executing a Runnable, it’s possible that extensions are being involved. For example:

ext: /path/to/extension
other_ext: http://github.com/user/some_extension
---

!ext.CustomRunnable
    ...
    param: !other_ext.CustomComponent

Flambé provides a -i / --install-extensions flag to automatically “pip” installs the extensions:

flambe custom_runnable.yaml -i

By default, this is not activated and the user needs to install the extensions beforehand.

Warning

Installing extensions automatically could possibly update libraries in the your environment because of a version reequirement. Flambé will output all libraries that are being updated.

Other flags

When executing flambé CLI passing a YAML file, this additional flags can be provided (among others):

--verbose / -v
All logs will be displayed in the console.
--force
Runnables may chose to accept this flag in its run() method to provide some overriding policies.