FTS5Tokenizer
The protocol for FTS5 tokenizers
-
Tokenizes the text described by
pText
andnText
, and notifies found tokens to thetokenCallback
function.It matches the
xTokenize
function documented at https://www.sqlite.org/fts5.html#custom_tokenizers