flambe.tokenizer.char

Module Contents

class flambe.tokenizer.char.CharTokenizer[source]

Bases: flambe.tokenizer.Tokenizer

Implement a character level tokenizer.

tokenize(self, example: str)[source]

Tokenize an input example.

Parameters:example (str) – The input example, as a string
Returns:The output charachter tokens, as a list of strings
Return type:List[str]