FTS5TokenizerDescriptor

The tokenizer components

// ["unicode61"]
FTS5TokenizerDescriptor.unicode61().components

// ["unicode61", "remove_diacritics", "0"]
FTS5TokenizerDescriptor.unicode61(removeDiacritics: false)).components

Creates an FTS5 tokenizer descriptor.

db.create(virtualTable: "book", using: FTS5()) { t in
    let tokenizer = FTS5TokenizerDescriptor(components: ["porter", "unicode61", "remove_diacritics", "0"])
    t.tokenizer = tokenizer
}

The ascii tokenizer

db.create(virtualTable: "book", using: FTS5()) { t in
    t.tokenizer = .ascii()
}

See https://www.sqlite.org/fts5.html#ascii_tokenizer

See https://www.sqlite.org/fts5.html#porter_tokenizer

An unicode61 tokenizer

db.create(virtualTable: "book", using: FTS5()) { t in
    t.tokenizer = .unicode61()
}

See https://www.sqlite.org/fts5.html#unicode61_tokenizer