flambe.nlp.language_modeling.fields

Module Contents

class flambe.nlp.language_modeling.fields.LMField(**kwargs)[source]

Bases: flambe.field.TextField

Language Model field.

Generates the original tensor alongside its shifted version.

process(self, example: str)[source]

Process an example and create 2 Tensors.

Parameters:example (str) – The example to process, as a single string
Returns:The processed example, tokenized and numericalized
Return type:Tuple[torch.Tensor, ..]