FTS5CustomTokenizer

The protocol for custom FTS5 tokenizers.

  • The name of the tokenizer; should uniquely identify your custom tokenizer.

  • Creates a custom tokenizer.

    The arguments parameter is an array of String built from the CREATE VIRTUAL TABLE statement. In the example below, the arguments will be ["arg1", "arg2"].

    CREATE VIRTUAL TABLE document USING fts5(
        tokenize='custom arg1 arg2'
    )
    
  • Creates an FTS5 tokenizer descriptor.

    class MyTokenizer : FTS5CustomTokenizer { ... }
    
    db.create(virtualTable: "book", using: FTS5()) { t in
        let tokenizer = MyTokenizer.tokenizerDescriptor(arguments: ["unicode61", "remove_diacritics", "0"])
        t.tokenizer = tokenizer
    }