flambe.tokenizer.label

Module Contents

class flambe.tokenizer.label.LabelTokenizer(multilabel_sep: Optional[str] = None)[source]

Bases: flambe.tokenizer.Tokenizer

Base label tokenizer.

This object tokenizes string labels into a list of a single or multiple elements, depending on the provided separator.

tokenize(self, example: str)[source]

Tokenize an input example.

Parameters:example (str) – The input example, as a string
Returns:The output tokens, as a list of strings
Return type:List[str]