flambe.compile.serialization

Module Contents

flambe.compile.serialization.logger[source]
exception flambe.compile.serialization.LoadError[source]

Bases: Exception

Error thrown because of fatal error when loading

class flambe.compile.serialization.SaveTreeNode[source]

Bases: typing.NamedTuple

Tree representation corresponding to the directory save format

state :Dict[str, Any][source]
version :str[source]
class_name :str[source]
source_code :str[source]
config :str[source]
object_stash :Dict[str, Any][source]
children :Dict[str, Any][source]
class flambe.compile.serialization.State[source]

Bases: collections.OrderedDict

A state object for Flambe.

_metadata :Dict[str, Any][source]
flambe.compile.serialization._convert_to_tree(metadata: Dict[str, Any]) → SaveTreeNode[source]
flambe.compile.serialization._fetch_tree_item(save_tree: SaveTreeNode, key: Sequence[str]) → Tuple[str, SaveTreeNode][source]
flambe.compile.serialization._update_save_tree(save_tree: SaveTreeNode, key: Sequence[str], value: Any) → None[source]
flambe.compile.serialization._update_save_tree_metadata(save_tree: SaveTreeNode, key: Sequence[str], value: Any) → None[source]
flambe.compile.serialization._traverse_all_nodes(save_tree: SaveTreeNode, path: Optional[List[str]] = None) → Iterable[Tuple[List[str], SaveTreeNode]][source]
flambe.compile.serialization._extract_prefix(root, directory)[source]
flambe.compile.serialization._prefix_keys(state, prefix)[source]
flambe.compile.serialization.traverse(nested: Mapping[str, Any], path: Optional[List[str]] = None) → Iterable[Any][source]

Iterate over a nested mapping returning the path and key, value.

Parameters:
  • nested (Mapping[str, Any]) – Mapping where some values are also mappings that should be traversed
  • path (List[str]) – List of keys that were used to reach the current mapping
Returns:

Iterable of path, key, value triples

Return type:

Iterable[Any]

Resolve links in schemas at block_id.

Parameters:schema (Dict[str, Schema[Any]]) – Map from block_id to Schema object
flambe.compile.serialization.save_state_to_file(state: State, path: str, compress: bool = False, pickle_only: bool = False, overwrite: bool = False, pickle_module=dill, pickle_protocol=DEFAULT_PROTOCOL) → None[source]

Save state to given path

By default the state will be saved in directory structure that mirrors the object hierarchy, so that you can later inspect the save file and load individual components more easily. If you would like to compress this directory structure using tar + gz, set compress to True. You can also use pickle to write a single output file, more similar to how PyTorch’s save function operates.

Parameters:
  • state (State) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • path (str) – Location to save the file / save directory to; This should be a new non-existent path; if the path is an existing directory and it contains files an exception will be raised. This is because the path includes the final name of the save file (if using pickle or compress) or the final name of the save directory.
  • compress (bool) – Whether to compress the save file / directory via tar + gz
  • pickle_only (bool) – Use given pickle_module instead of the hiearchical save format (the default is False).
  • overwrite (bool) – If true, overwrites the contents of the given path to create a directory at that location
  • pickle_module (type) – Pickle module that has load and dump methods; dump should accpet a pickle_protocol parameter (the default is dill).
  • pickle_protocol (type) – Pickle protocol to use; see pickle for more details (the default is 2).
Raises:

ValueError – If the given path exists, is a directory, and already contains some files.

flambe.compile.serialization.save(obj: Any, path: str, compress: bool = False, pickle_only: bool = False, overwrite: bool = False, pickle_module=dill, pickle_protocol=DEFAULT_PROTOCOL) → None[source]

Save Component object to given path

See save_state_to_file for a more detailed explanation

Parameters:
  • obj (Component) – The component to save.
  • path (str) – Location to save the file / save directory to
  • compress (bool) – Whether to compress the save file / directory via tar + gz
  • pickle_only (bool) – Use given pickle_module instead of the hiearchical save format (the default is False).
  • overwrite (bool) – If true, overwrites the contents of the given path to create a directory at that location
  • pickle_module (type) – Pickle module that has load and dump methods; dump should accept a pickle_protocol parameter (the default is dill).
  • pickle_protocol (type) – Pickle protocol to use; see pickle for more details (the default is 2).
flambe.compile.serialization.load_state_from_file(path: str, map_location=None, pickle_module=dill, **pickle_load_args) → State[source]

Load state from the given path

Loads a flambe save directory, pickled save object, or a compressed version of one of these two formats (using tar + gz). Will automatically infer the type of save format and if the directory structure is used, the serialization protocol version as well.

Parameters:
  • path (str) – Path to the save file or directory
  • map_location (type) – Location (device) where items will be moved. ONLY used when the directory save format is used. See torch.load documentation for more details (the default is None).
  • pickle_module (type) – Pickle module that has load and dump methods; dump should accept a pickle_protocol parameter (the default is dill).
  • **pickle_load_args (type) – Additional args that pickle_module should use to load; see torch.load documentation for more details
Returns:

state_dict that can be loaded into a compatible Component

Return type:

State

flambe.compile.serialization.load(path: str, map_location=None, auto_install=False, pickle_module=dill, **pickle_load_args)[source]

Load object with state from the given path

Loads a flambe object by using the saved config files, and then loads the saved state into said object. See load_state_from_file for details regarding how the state is loaded from the save file or directory.

Parameters:
  • path (str) – Path to the save file or directory
  • map_location (type) – Location (device) where items will be moved. ONLY used when the directory save format is used. See torch.load documentation for more details (the default is None).
  • auto_install (bool) – If True, automatically installs extensions as needed.
  • pickle_module (type) – Pickle module that has load and dump methods; dump should accept a pickle_protocol parameter (the default is dill).
  • **pickle_load_args (type) – Additional args that pickle_module should use to load; see torch.load documentation for more details
Returns:

object with both the architecture (config) and state that was saved to path

Return type:

Component

Raises:

LoadError – If a Component object is not loadable from the given path because extensions are not installed, or the config is empty, nonexistent, or otherwise invalid.