Implementing Runnables is as easy as implementing the Runnable interface.
Esentially, it requires a single method
Runnable object can be executed by flambé:
For example, let’s imagine we want to implement an S3Pusher that takes
Experiment output folder and uploads the content to a specific S3 bucket:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
from flambe.runnable import Runnable class S3Pusher(Runnable): def __init__(self, experiment_path: str, bucket_name: str) -> None: super().__init__() self.experiment_path = experiment_path self.bucket_name = bucket_name def run(self, **kwargs) -> None: """Upload a local folder to a S3 bucket""" # Code to upload to S3 bucket S3_client = boto3.client("s3") for root, dirs, files in os.walk(self.experiment_path): for f in files: s3C.upload_file(os.path.join(root, f), self.bucketname, f)
This class definition can now be included in an extension (read more about extensions in Extensions) and used as a top level object in a YAML file.
ext: /path/to/S3Pusher/extension: --- !ext.S3Pusher experiment_path: /path/to/my/experiment bucket_name: my-bucket
Then, simply execute:
Runnables have access to secret information that the users can share via
an ini file.
By executing the
Runnable with a
--secrets parameter, then the
access the secrets through the
Let’s say that our
S3Pusher needs to access the
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
from flambe.runnable import Runnable class S3Pusher(Runnable): def __init__(self, experiment_path: str, bucket_name: str) -> None: # Same as before ... def run(self, **kwargs) -> None: """Upload a local folder to a S3 bucket""" # Code to upload to S3 bucket S3_client = boto3.client("s3", token=self.config['AWS']['AWS_SECRET_TOKEN']) for root, dirs, files in os.walk(self.experiment_path): for file in files: s3C.upload_file(os.path.join(root, file), self.bucketname, file)
[AWS] AWS_SECRET_TOKEN = ABCDEFGHI123456789
We can execute:
flambe s3pusher.yaml --secrets secret.ini
Automatic extensions installation¶
To understand this section you should be familiar with extensions. For information about extensions, go to Extensions.
When executing a
Runnable, it’s possible that extensions are being involved. For example:
ext: /path/to/extension other_ext: http://github.com/user/some_extension --- !ext.CustomRunnable ... param: !other_ext.CustomComponent
Flambé provides a
-i / --install-extensions flag to automatically “pip” installs the extensions:
flambe custom_runnable.yaml -i
By default, this is not activated and the user needs to install the extensions beforehand.
Installing extensions automatically could possibly update libraries in the your environment because of a version reequirement. Flambé will output all libraries that are being updated.