flambe.compile.component

Module Contents

flambe.compile.component._EMPTY[source]
flambe.compile.component.A[source]
flambe.compile.component.C[source]
flambe.compile.component.YAML_TYPES[source]
flambe.compile.component.logger[source]
exception flambe.compile.component.CompilationError[source]

Bases: Exception

exception flambe.compile.component.LoadError[source]

Bases: Exception

class flambe.compile.component.Schema(component_subclass: Type[C], _flambe_custom_factory_name: Optional[str] = None, **keywords: Any)[source]

Bases: MutableMapping[str, Any]

Holds and recursively initializes Component’s with kwargs

Holds a Component subclass and keyword arguments to that class’s compile method. When an instance is called it will perform the recursive compilation process, turning the nested structure of Schema’s into initialized Component objects

Implements MutableMapping methods to facilitate inspection and updates to the keyword args. Implements dot-notation access to the keyword args as well.

Parameters:
  • component_subclass (Type[Component]) – Subclass of Component that will be compiled
  • **keywords (Any) – kwargs passed into the Schema’s compile method

Examples

Create a Schema from a Component subclass

>>> class Test(Component):
...     def __init__(self, x=2):
...         self.x = x
...
>>> tp = Schema(Test)
>>> t1 = tp()
>>> t2 = tp()
>>> assert t1 is t2  # the same Schema always gives you same obj
>>> tp = Schema(Test) # create a new Schema
>>> tp['x'] = 3
>>> t3 = tp()
>>> assert t1.x == 3  # dot notation works as well
component_subclass

Subclass of Schema that will be compiled

Type:Type[Component]
keywords

kwargs passed into the Schema’s compile method

Type:Dict[str, Any]
__call__(self, stash: Optional[Dict[str, Any]] = None, **keywords: Any)[source]
add_extensions_metadata(self, extensions: Dict[str, str])[source]

Add extensions used when loading this schema and children

Uses component_subclass.__module__ to filter for only the single relevant extension for this object; extensions relevant for children are saved only on those children schemas directly. Use aggregate_extensions_metadata to generate a dictionary of all extensions used in the object hierarchy.

aggregate_extensions_metadata(self)[source]

Aggregate extensions used in object hierarchy

contains(self, schema: Schema, original_link: Link)[source]
__setitem__(self, key: str, value: Any)[source]
__getitem__(self, key: str)[source]
__delitem__(self, key: str)[source]
__iter__(self)[source]
__len__(self)[source]
__getattr__(self, item: str)[source]
__setattr__(self, key: str, value: Any)[source]
__repr__(self)[source]

Identical to super (schema), but sorts keywords

classmethod to_yaml(cls, representer: Any, node: Any, tag: str = '')[source]
static serialize(obj: Any)[source]

Return dictionary representation of schema

Includes yaml as a string, and extensions

Parameters:obj (Any) – Should be schema or dict of schemas
Returns:dictionary containing yaml and extensions dictionary
Return type:Dict[str, Any]
static deserialize(data: Dict[str, Any])[source]

Construct Schema from dict returned by Schema.serialize

Parameters:data (Dict[str, Any]) – dictionary returned by Schema.serialize
Returns:Schema or dict of schemas (depending on yaml in data)
Return type:Any
class flambe.compile.component.contextualized_linking(root_obj: Any, prefix: str)[source]

Context manager used to change the representation of links

Links are always defined in relation to some root object and an attribute path, so when representing some piece of a larger object all the links need to be redefined in relation to the target object

__enter__(self)[source]
__exit__(self, exc_type: Any, exc_value: Any, traceback: Any)[source]

Bases: flambe.compile.registrable.Registrable

__call__(self, stash: Dict[str, Any])[source]
classmethod to_yaml(cls, representer: Any, node: Any, tag: str)[source]
classmethod from_yaml(cls, constructor: Any, node: Any, factory_name: str)[source]
exception flambe.compile.component.LinkError[source]

Bases: Exception

exception flambe.compile.component.MalformedLinkError[source]

Bases: flambe.compile.component.LinkError

exception flambe.compile.component.UnpreparedLinkError[source]

Bases: flambe.compile.component.LinkError

Parse link to extract schematic and attribute paths

Links should be of the format obj[key1][key2].attr1.attr2 where obj is the entry point; in a pipeline, obj would be the stage name, in a single-object config obj would be the target keyword at the top level. The following keys surrounded in brackets traverse the nested dictionary structure that appears in the config; this is intentonally analagous to how you would access properties in the dictionary when loaded into python. Then, you can use the dot notation to access the runtime instance attributes of the object at that location.

Parameters:link_str (str) – Link to earlier object in the config of the format obj[key1][key2].attr1.attr2
Returns:Tuple of the schematic and attribute paths respectively
Return type:Tuple[Sequence[str], Sequence[str]]
Raises:MalformedLinkError – If the link is written incorrectly

Examples

Examples should be written in doctest format, and should illustrate how to use the function/class. >>> parse_link_str(‘obj[key1][key2].attr1.attr2’) ([‘obj’, ‘key1’, ‘key2’], [‘attr1’, ‘attr2’])

Create a string representation of the specified link

Performs the reverse operation of parse_link_str()

Parameters:
  • schematic_path (Sequence[str]) – List of entries corresponding to dictionary keys in a nested Schema
  • attr_path (Optional[Sequence[str]]) – List of attributes to access on the target object (the default is None).
Returns:

The string representation of the schematic + attribute paths

Return type:

str

Raises:

MalformedLinkError – If the schematic_path is empty

Examples

Examples should be written in doctest format, and should illustrate how to use the function/class. >>> create_link_str([‘obj’, ‘key1’, ‘key2’], [‘attr1’, ‘attr2’]) ‘obj[key1][key2].attr1.attr2’

Bases: flambe.compile.registrable.Registrable

Delayed access to another object in an object hiearchy

Currently only supported in the context of Experiment but this may be updated in a future release

A Link delays the access of some property, or the calling of some method, until the Link is called. Links can be passed directly into a Component subclass compile, Component’s method called compile will automatically record the links and call them to access their values before running __new__ and __init__. The recorded links will show up in the config if yaml.dump() is called on your object hierarchy. This typically happens when logging individual configs during a grid search, and when serializing between multiple processes.

For example, if the schematic path is [‘model’, ‘encoder’] and the attribute path is [‘rnn’, ‘hidden_size’] then before the link can be compiled, the target attribute should be set to point to the model schema (this is handled automatically by Experiment) then, during compilation the child schema ‘encoder’ will be accessed, and finally the attribute encoder.rnn.hidden_size will be returned

Parameters:
  • schematic_path (Sequence[str]) – Path to the relevant schema denoted by dictionary-like bracket access e.g. [‘model’, ‘encoder’]
  • attr_path (Sequence[str]) – Path to the relevant attribute on the given schema (after it’s been compiled) using standard attribute dot notation e.g. [‘rnn’, ‘hidden_size’]
  • target (Optional[Schema]) – The root object corresponding to the first element in the schematic path; needs to be passed in here or set later before link can be resolved
  • local (bool) – if true, changes tune convert behavior to insert a dummy link; used for links to global variables (“resources” in config) (defaults to True)
root_schema :str[source]
__repr__(self)[source]
__call__(self)[source]
classmethod to_yaml(cls, representer: Any, node: Any, tag: str)[source]

Build contextualized link based on the root node

If the link refers to something inside of the current object hierarchy (as determined by the schema of _link_root_obj) then it will be represented as a link; if the link refers to something out-of-scope, i.e. not inside the current object hiearchy, then replace the link with the resolved value. If the value cannot be represented, pickle it and include a reference to its id in the object stash that will be saved alongside the config

classmethod from_yaml(cls, constructor: Any, node: Any, factory_name: str)[source]
convert(self)[source]

Bases: flambe.compile.component.Link

Calls the link attribute instead of just accessing it

__call__(self)[source]
flambe.compile.component.K[source]
flambe.compile.component.fill_defaults(kwargs: Dict[str, Any], function: Callable[..., Any]) → Dict[str, Any][source]

Use function signature to add missing kwargs to a dictionary

flambe.compile.component.merge_kwargs(kwargs: Dict[str, Any], compiled_kwargs: Dict[str, Any]) → Dict[str, Any][source]

Replace non links in kwargs with corresponding compiled values

For every key in kwargs if the value is NOT a link and IS a Schema, replace with the corresponding value in compiled_kwargs

Parameters:
  • kwargs (Dict[str, Any]) – Original kwargs containing Links and Schemas
  • compiled_kwargs (Dict[str, Any]) – Processes kwargs containing no links and no Schemas
Returns:

kwargs with links, but with Schemas replaced by compiled objects

Return type:

Dict[str, Any]

class flambe.compile.component.Component(**kwargs)[source]

Bases: flambe.compile.registrable.Registrable

Class which can be serialized to yaml and implements compile

IMPORTANT: ALWAYS inherit from Component BEFORE torch.nn.Module

Automatically registers subclasses via Registrable and facilitates immediate usage in YAML with tags. When loaded, subclasses’ initialization is delayed; kwargs are wrapped in a custom schema called Schema that can be easily initialized later.

_flambe_version = 0.0.0[source]
_config_str[source]

Represent object’s architecture as a YAML string

Includes the extensions relevant to the object as well; NOTE: currently this section may include a superset of the extensions actually needed, but this will be changed in a future release.

run(self)[source]

Run a single computational step.

When used in an experiment, this computational step should be on the order of tens of seconds to about 10 minutes of work on your intended hardware; checkpoints will be performed in between calls to run, and resources or search algorithms will be updated. If you want to run everything all at once, make sure a single call to run does all the work and return False.

Returns:True if should continue running later i.e. more work to do
Return type:bool
metric(self)[source]

Override this method to enable scheduling and searching.

Returns:The metric to compare different variants of your Component
Return type:float
register_attrs(self, *names: str)[source]

Set attributes that should be included in state_dict

Equivalent to overriding obj._state and obj._load_state to save and load these attributes. Recommended usage: call inside __init__ at the end: self.register_attrs(attr1, attr2, …) Should ONLY be called on existing attributes.

Parameters:*names (str) – The names of the attributes to register
Raises:AttributeError – If self does not have existing attribute with that name
static _state_dict_hook(self, state_dict: State, prefix: str, local_metadata: Dict[str, Any])[source]

Add metadata and recurse on Component children

This hook is used to integrate with the PyTorch state_dict mechanism; as either nn.Module.state_dict or Component.get_state recurse, this hook is responsible for adding Flambe specific metadata and recursing further on any Component children of self that are not also nn.Modules, as PyTorch will handle recursing to the latter.

Flambe specific metadata includes the class version specified in the Component._flambe_version class property, the name of the class, the source code, and the fact that this class is a Component and should correspond to a directory in our hiearchical save format

Finally, this hook calls a helper _state that users can implement to add custom state to a given class

Parameters:
  • state_dict (State) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • prefix (str) – The current prefix for new compound keys that reflects the location of this instance in the object hierarchy being represented
  • local_metadata (Dict[str, Any]) – A subset of the metadata relevant just to this object and its children
Returns:

The modified state_dict

Return type:

type

Raises:

ExceptionName – Why the exception is raised.

_add_registered_attrs(self, state_dict: State, prefix: str)[source]
_state(self, state_dict: State, prefix: str, local_metadata: Dict[str, Any])[source]

Add custom state to state_dict

Parameters:
  • state_dict (State) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • prefix (str) – The current prefix for new compound keys that reflects the location of this instance in the object hierarchy being represented
  • local_metadata (Dict[str, Any]) – A subset of the metadata relevant just to this object and its children
Returns:

The modified state_dict

Return type:

State

get_state(self, destination: Optional[State] = None, prefix: str = '', keep_vars: bool = False)[source]

Extract PyTorch compatible state_dict

Adds Flambe specific properties to the state_dict, including special metadata (the class version, source code, and class name). By default, only includes state that PyTorch nn.Module includes (Parameters, Buffers, child Modules). Custom state can be added via the _state helper method which subclasses should override.

The metadata _flambe_directories indicates which objects are Components and should be a subdirectory in our hierarchical save format. This object will recurse on Component and nn.Module children, but NOT torch.optim.Optimizer subclasses, torch.optim.lr_scheduler._LRScheduler subclasses, or any other arbitrary python objects.

Parameters:
  • destination (Optional[State]) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • prefix (str) – The current prefix for new compound keys that reflects the location of this instance in the object hierarchy being represented
  • keep_vars (bool) – Whether or not to keep Variables (only used by PyTorch) (the default is False).
Returns:

The state_dict object

Return type:

State

Raises:

ExceptionName – Why the exception is raised.

_load_state_dict_hook(self, state_dict: State, prefix: str, local_metadata: Dict[str, Any], strict: bool, missing_keys: List[Any], unexpected_keys: List[Any], error_msgs: List[Any])[source]

Load flambe-specific state

Parameters:
  • state_dict (State) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • prefix (str) – The current prefix for new compound keys that reflects the location of this instance in the object hierarchy being represented
  • local_metadata (Dict[str, Any]) – A subset of the metadata relevant just to this object and its children
  • strict (bool) – Whether missing or unexpected keys should be allowed; should always be False in Flambe
  • missing_keys (List[Any]) – Missing keys so far
  • unexpected_keys (List[Any]) – Unexpected keys so far
  • error_msgs (List[Any]) – Any error messages so far
Raises:

LoadError – If the state for some object does not have a matching major version number

_load_registered_attrs(self, state_dict: State, prefix: str)[source]
_load_state(self, state_dict: State, prefix: str, local_metadata: Dict[str, Any], strict: bool, missing_keys: List[Any], unexpected_keys: List[Any], error_msgs: List[Any])[source]

Load custom state (that was included via _state)

Subclasses should override this function to add custom state that isn’t normally included by PyTorch nn.Module

Parameters:
  • state_dict (State) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • prefix (str) – The current prefix for new compound keys that reflects the location of this instance in the object hierarchy being represented
  • local_metadata (Dict[str, Any]) – A subset of the metadata relevant just to this object and its children
  • strict (bool) – Whether missing or unexpected keys should be allowed; should always be False in Flambe
  • missing_keys (List[Any]) – Missing keys so far
  • unexpected_keys (List[Any]) – Unexpected keys so far
  • error_msgs (List[Any]) – Any error messages so far
load_state(self, state_dict: State, strict: bool = False)[source]

Load state_dict into self

Loads state produced by get_state into the current object, recursing on child Component and nn.Module objects

Parameters:
  • state_dict (State) – The state_dict as defined by PyTorch; a flat dictionary with compound keys separated by ‘.’
  • strict (bool) – Whether missing or unexpected keys should be allowed; should ALWAYS be False in Flambe (the default is False).
Raises:

LoadError – If the state for some object does not have a matching major version number

classmethod load_from_path(cls, path: str, map_location: Union[torch.device, str] = None, use_saved_config_defaults: bool = True, **kwargs: Any)[source]
save(self, path: str, **kwargs: Any)[source]
classmethod to_yaml(cls, representer: Any, node: Any, tag: str)[source]
classmethod from_yaml(cls: Type[C], constructor: Any, node: Any, factory_name: str)[source]
classmethod precompile(cls: Type[C], **kwargs: Any)[source]

Change kwargs before compilation occurs.

This hook is called after links have been activated, but before calling the recursive initialization process on all other objects in kwargs. This is useful in a number of cases, for example, in Trainer, we compile several objects ahead of time and move them to the GPU before compiling the optimizer, because it needs to be initialized with the model parameters after they have been moved to GPU.

Parameters:
  • cls (Type[C]) – Class on which method is called
  • **kwargs (Any) – Current kwargs that will be compiled and used to initialize an instance of cls after this hook is called
aggregate_extensions_metadata(self)[source]

Aggregate extensions used in object hierarchy

TODO: remove or combine with schema implementation in refactor

classmethod compile(cls: Type[C], _flambe_custom_factory_name: Optional[str] = None, _flambe_extensions: Optional[Dict[str, str]] = None, _flambe_stash: Optional[Dict[str, Any]] = None, **kwargs: Any)[source]

Create instance of cls after recursively compiling kwargs

Similar to normal initialization, but recursively initializes any arguments that should be compiled and allows overriding arbitrarily deep kwargs before initializing if needed. Also activates any Link instances passed in as kwargs, and saves the original kwargs for dumping to yaml later.

Parameters:**kwargs (Any) – Keyword args that should be forwarded to the initialization function (a specified factory, or the normal __new__ and __init__ methods)
Returns:An instance of the class cls
Return type:C
flambe.compile.component.dynamic_component(class_: Type[A], tag: str, tag_namespace: Optional[str] = None, parent_component_class: Type[Component] = Component) → Type[Component][source]

Decorate given class, creating a dynamic Component

Creates a dynamic subclass of class_ that inherits from Component so it will be registered with the yaml loader and receive the appropriate functionality (from_yaml, to_yaml and compile). class_ should not implement any of the aforementioned functions.

Parameters:
  • class (Type[A]) – Class to register with yaml and the compilation system
  • tag (str) – Tag that will be used with yaml
  • tag_namespace (str) – Namespace aka the prefix, used. e.g. for !torch.Adam torch is the namespace
Returns:

New subclass of _class and Component

Return type:

Type[Component]