flambe.experiment.experiment

Module Contents

flambe.experiment.experiment.logger[source]
flambe.experiment.experiment.OptionalSearchAlgorithms[source]
flambe.experiment.experiment.OptionalTrialSchedulers[source]
class flambe.experiment.experiment.Experiment(name: str, pipeline: Dict[str, Schema], resume: Optional[Union[str, Sequence[str]]] = None, debug: bool = False, devices: Dict[str, int] = None, save_path: Optional[str] = None, resources: Optional[Dict[str, Dict[str, Any]]] = None, search: OptionalSearchAlgorithms = None, schedulers: OptionalTrialSchedulers = None, reduce: Optional[Dict[str, int]] = None, env: RemoteEnvironment = None, max_failures: int = 2, merge_plot: bool = True)[source]

Bases: flambe.runnable.ClusterRunnable

A Experiment object.

The Experiment object is the top level module in the Flambé workflow. The object is responsible for starting workers, assiging the orchestrator machine, as well as converting the input blocks into Ray Tune Experiment objects.

Parameters:
  • name (str) – A name for the experiment
  • pipeline (OrderedDict[str, Schema[Component]]) – Ordered mapping from block id to a schema of the block
  • force (bool) – When running a local experiment this flag will make flambe override existing results from previous experiments. When running remote experiments this flag will reuse an existing cluster (in case of any) that is running an experiment with the same name in the same cloud service. The use of this flag is discouraged as you may lose useful data.
  • resume (Union[str, List[str]]) – If a string is given, resume all blocks up until the given block_id. If a list is given, resume all blocks in that list.
  • debug (bool) – If debug is True, then a debugger will be available at the beginning of each Component block of the pipeline. Defaults to False. ATTENTION: debug only works when running locally.
  • save_path (Optional[str]) – A directory where to save the experiment.
  • devices (Dict[str, int]) – Tune’s resources per trial. For example: {“cpu”: 12, “gpu”: 2}.
  • resources (Optional[Dict[str, Dict[str, Any]]]) – Variables to use in the pipeline section with !@ notation. This section is splitted into 2 sections: local and remote.
  • search (Mapping[str, SearchAlgorithm], optional) – Map from block id to hyperparameter search space generator. May have Schemas of SearchAlgorithm as well.
  • schedulers (Mapping[str, TrialScheduler], optional) – Map from block id to search scheduler. May have Schemas of TrialScheduler as well.
  • reduce (Mapping[str, int], optional) – Map from block to number of trials to reduce to.
  • env (RemoteEnvironment) – Contains remote information about the cluster. This object will be received in case this Experiment is running remotely.
  • max_failures (int) – number of times to retry running the pipeline if it hits some type of failure, defaults to retrying twice
  • merge_plot (bool) – Display all tensorboard logs in the same plot (per block type). Defaults to True.
run(self, force: bool = False, verbose: bool = False, **kwargs)[source]

Run an Experiment

setup(self, cluster: Cluster, extensions: Dict[str, str], force: bool, **kwargs)[source]

Prepare the cluster for the Experiment remote execution.

This involves:

  1. [Optional] Kill previous flambe execution
  2. [Optional] Remove existing results
  3. Create supporting dirs (exp/synced_results, exp/resources)
  4. Install extensions in all factories
  5. Launch ray cluster
  6. Send resources
  7. Launch Tensorboard + Report site
Parameters:
  • cluster (Cluster) – The cluster where this Runnable will be running
  • extensions (Dict[str, str]) – The ClusterRunnable extensions
  • force (bool) – The force value provided to Flambe
parse(self)[source]

Parse the experiment.

Parse the Experiment in search of errors that won’t allow the experiment to run successfully. If it finds any error, then it raises an ParsingExperimentError.

Raises:ParsingExperimentError – In case a parsing error is found.