FTS5Tokenizer
The protocol for FTS5 tokenizers
-
Tokenizes the text described by
pTextandnText, and notifies found tokens to thetokenCallbackfunction.It matches the
xTokenizefunction documented at https://www.sqlite.org/fts5.html#custom_tokenizers -
tokenize(_:for:)Extension methodTokenizes the string argument into an array of (String, FTS5TokenFlags) pairs.
let tokenizer = try db.makeTokenizer(.ascii()) try tokenizer.tokenize("foo bar", for: .document) // [("foo", flags), ("bar", flags)] -
nonSynonymTokens(in:for:)Extension method
View on GitHub
Install in Dash
FTS5Tokenizer Protocol Reference