FTS5Tokenizer

public protocol FTS5Tokenizer : AnyObject

The protocol for FTS5 tokenizers


                    
                    
                    tokenize(context:tokenization:pText:nText:tokenCallback:)

Tokenizes the text described by pText and nText, and notifies found tokens to the tokenCallback function.

It matches the xTokenize function documented at https://www.sqlite.org/fts5.html#custom_tokenizers

Declaration

Swift

func tokenize(context: UnsafeMutableRawPointer?, tokenization: FTS5Tokenization, pText: UnsafePointer<Int8>?, nText: Int32, tokenCallback: @escaping FTS5TokenCallback) -> Int32

Parameters

`context`	An opaque pointer that is the first argument to the `tokenCallback` function
`tokenization`	The reason why FTS5 is requesting tokenization.
`pText`	The tokenized text bytes. May or may not be nul-terminated.
`nText`	The number of bytes in the tokenized text.
`tokenCallback`	The function to call for each found token. It matches the `xToken` callback at https://www.sqlite.org/fts5.html#custom_tokenizers

Show on GitHub