flambe.nlp.transformers.field

Module Contents

class flambe.nlp.transformers.field.PretrainedTransformerField(alias: str, cache_dir: Optional[str] = None, max_len_truncate: int = 500, add_special_tokens: bool = True, **kwargs)[source]

Bases: flambe.field.Field

Field intergation of the transformers library.

Instantiate this object using any alias available in the transformers library. More information can be found here:

https://huggingface.co/transformers/

padding_idx :int[source]

Get the padding index.

Returns:The padding index in the vocabulary
Return type:int
vocab_size :int[source]

Get the vocabulary length.

Returns:The length of the vocabulary
Return type:int
process(self, example: Union[str, Tuple[Any], List[Any], Dict[Any, Any]])[source]

Process an example, and create a Tensor.

Parameters:example (str) – The example to process, as a single string
Returns:The processed example, tokenized and numericalized
Return type:torch.Tensor