FTS5Tokenizer
public protocol FTS5Tokenizer : AnyObject
The protocol for FTS5 tokenizers
-
Tokenizes the text described by
pText
andnText
, and notifies found tokens to thetokenCallback
function.It matches the
xTokenize
function documented at https://www.sqlite.org/fts5.html#custom_tokenizersDeclaration
Swift
func tokenize(context: UnsafeMutableRawPointer?, tokenization: FTS5Tokenization, pText: UnsafePointer<Int8>?, nText: Int32, tokenCallback: @escaping FTS5TokenCallback) -> Int32
Parameters
context
An opaque pointer that is the first argument to the
tokenCallback
functiontokenization
The reason why FTS5 is requesting tokenization.
pText
The tokenized text bytes. May or may not be nul-terminated.
nText
The number of bytes in the tokenized text.
tokenCallback
The function to call for each found token. It matches the
xToken
callback at https://www.sqlite.org/fts5.html#custom_tokenizers