Module Contents

class flambe.tokenizer.tokenizer.Tokenizer[source]

Bases: flambe.Component

Base interface to a Tokenizer object.

Tokenizers implement the tokenize method, which takes a string as input and produces a list of strings as output.

tokenize(self, example: str)[source]

Tokenize an input example.

Parameters:example (str) – The input example, as a string
Returns:The output tokens, as a list of strings
Return type:List[str]
__call__(self, example: str)[source]

Make a tokenizer callable.