flambe.experiment.experiment

Module Contents

flambe.experiment.experiment.logger[source]
flambe.experiment.experiment.OptionalSearchAlgorithms[source]
flambe.experiment.experiment.OptionalTrialSchedulers[source]
class flambe.experiment.experiment.Experiment(name: str, pipeline: Dict[str, Schema], resume: Optional[Union[str, Sequence[str]]] = None, devices: Dict[str, int] = None, save_path: Optional[str] = None, resources: Optional[Dict[str, Union[str, ClusterResource]]] = None, search: OptionalSearchAlgorithms = None, schedulers: OptionalTrialSchedulers = None, reduce: Optional[Dict[str, int]] = None, env: RemoteEnvironment = None, max_failures: int = 1, stop_on_failure: bool = True, merge_plot: bool = True, user_provider: Callable[[], str] = None)[source]

Bases: flambe.runnable.ClusterRunnable

A Experiment object.

The Experiment object is the top level module in the Flambé workflow. The object is responsible for starting workers, assiging the orchestrator machine, as well as converting the input blocks into Ray Tune Experiment objects.

Parameters:
  • name (str) – A name for the experiment
  • pipeline (OrderedDict[str, Schema[Component]]) – Ordered mapping from block id to a schema of the block
  • force (bool) – When running a local experiment this flag will make flambe override existing results from previous experiments. When running remote experiments this flag will reuse an existing cluster (in case of any) that is running an experiment with the same name in the same cloud service. The use of this flag is discouraged as you may lose useful data.
  • resume (Union[str, List[str]]) – If a string is given, resume all blocks up until the given block_id. If a list is given, resume all blocks in that list.
  • save_path (Optional[str]) – A directory where to save the experiment.
  • devices (Dict[str, int]) – Tune’s resources per trial. For example: {“cpu”: 12, “gpu”: 2}.
  • resources (Optional[Dict[str, Dict[str, Any]]]) – Variables to use in the pipeline section with !@ notation. This section is splitted into 2 sections: local and remote.
  • search (Mapping[str, SearchAlgorithm], optional) – Map from block id to hyperparameter search space generator. May have Schemas of SearchAlgorithm as well.
  • schedulers (Mapping[str, TrialScheduler], optional) – Map from block id to search scheduler. May have Schemas of TrialScheduler as well.
  • reduce (Mapping[str, int], optional) – Map from block to number of trials to reduce to.
  • env (RemoteEnvironment) – Contains remote information about the cluster. This object will be received in case this Experiment is running remotely.
  • max_failures (int) – Number of times to retry running the pipeline if it hits some type of failure, defaults to one.
  • merge_plot (bool) – Display all tensorboard logs in the same plot (per block type). Defaults to True.
  • user_provider (Callable[[], str]) – The logic for specifying the user triggering this Runnable. If not passed, by default it will pick the computer’s user.
process_resources(self, resources: Dict[str, Union[str, ClusterResource]], folder: str)[source]

Download resources that are not tagged with ‘!cluster’ into a given directory.

Parameters:
  • resources (Dict[str, Union[str, ClusterResource]]) – The resources dict
  • folder (str) – The directory where the remote resources will be downloaded.
Returns:

The resources dict where the remote urls that don’t contain ‘!cluster’ point now to the local path where the resource was downloaded.

Return type:

Dict[str, Union[str, ClusterResource]]

run(self, force: bool = False, verbose: bool = False, debug: bool = False, **kwargs)[source]

Run an Experiment

teardown(self)[source]
setup(self, cluster: Cluster, extensions: Dict[str, str], force: bool, **kwargs)[source]

Prepare the cluster for the Experiment remote execution.

This involves:

  1. [Optional] Kill previous flambe execution
  2. [Optional] Remove existing results
  3. Create supporting dirs (exp/synced_results, exp/resources)
  4. Install extensions in all factories
  5. Launch ray cluster
  6. Send resources
  7. Launch Tensorboard + Report site
Parameters:
  • cluster (Cluster) – The cluster where this Runnable will be running
  • extensions (Dict[str, str]) – The ClusterRunnable extensions
  • force (bool) – The force value provided to Flambe
parse(self)[source]

Parse the experiment.

Parse the Experiment in search of errors that won’t allow the experiment to run successfully. If it finds any error, then it raises an ParsingExperimentError.

Raises:ParsingExperimentError – In case a parsing error is found.
get_user(self)[source]

Get the user that triggered this experiment.

Returns:The user as a string.
Return type:str
_dump_experiment_file(self)[source]

Dump the experiment YAML representation to the output folder.