FTS5CustomTokenizer

The protocol for custom FTS5 tokenizers.

name

The name of the tokenizer; should uniquely identify your custom tokenizer.
init(db:arguments:)
Creates a custom tokenizer.

The arguments parameter is an array of String built from the CREATE VIRTUAL TABLE statement. In the example below, the arguments will be ["arg1", "arg2"].
```
CREATE VIRTUAL TABLE document USING fts5(
    tokenize='custom arg1 arg2'
)
```


                    
                    
                    tokenizerDescriptor(arguments:)

Extension method

Creates an FTS5 tokenizer descriptor.

class MyTokenizer : FTS5CustomTokenizer { ... }

db.create(virtualTable: "book", using: FTS5()) { t in
    let tokenizer = MyTokenizer.tokenizerDescriptor(arguments: ["unicode61", "remove_diacritics", "0"])
    t.tokenizer = tokenizer
}