FTS5Tokenizer

Tokenizes the text described by pText and nText, and notifies found tokens to the tokenCallback function.

It matches the xTokenize function documented at https://www.sqlite.org/fts5.html#custom_tokenizers

Tokenizes the string argument as a document that would be inserted into an FTS5 table.

For example:

let tokenizer = try db.makeTokenizer(.ascii())
try tokenizer.tokenize(document: "foo bar") // [("foo", flags), ("bar", flags)]

See also tokenize(query:).

Tokenizes the string argument as an FTS5 query.

For example:

let tokenizer = try db.makeTokenizer(.ascii())
try tokenizer.tokenize(query: "foo bar") // [("foo", flags), ("bar", flags)]

See also tokenize(document:).