FTS5TokenizerDescriptor

public struct FTS5TokenizerDescriptor

An FTS5 tokenizer, suitable for FTS5 table definitions:

db.create(virtualTable: "book", using: FTS5()) { t in
    t.tokenizer = .unicode61() // FTS5TokenizerDescriptor
}

See https://www.sqlite.org/fts5.html#tokenizers

  • The tokenizer components

    // ["unicode61"]
FTS5TokenizerDescriptor.unicode61().components

// ["unicode61", "remove_diacritics", "0"]
FTS5TokenizerDescriptor.unicode61(removeDiacritics: false)).components

    Declaration

    Swift

    public let components: [String]
    Show on GitHub

  • Creates an FTS5 tokenizer descriptor.

    db.create(virtualTable: "book", using: FTS5()) { t in
    let tokenizer = FTS5TokenizerDescriptor(components: ["porter", "unicode61", "remove_diacritics", "0"])
    t.tokenizer = tokenizer
}

    Precondition

    Components is not empty

    Declaration

    Swift

    public init(components: [String])
    Show on GitHub

  • The “ascii” tokenizer

    db.create(virtualTable: "book", using: FTS5()) { t in
    t.tokenizer = .ascii()
}

    See https://www.sqlite.org/fts5.html#ascii_tokenizer

    Declaration

    Swift

    public static func ascii(separators: Set<Character> = []) -> FTS5TokenizerDescriptor

    Parameters

    separators

    Unless empty (the default), SQLite will consider these characters as token separators.
    Show on GitHub

  • See https://www.sqlite.org/fts5.html#porter_tokenizer

    Declaration

    Swift

    public static func porter(wrapping base: FTS5TokenizerDescriptor? = nil) -> FTS5TokenizerDescriptor
    Show on GitHub

  • An “unicode61” tokenizer

    db.create(virtualTable: "book", using: FTS5()) { t in
    t.tokenizer = .unicode61()
}

    See https://www.sqlite.org/fts5.html#unicode61_tokenizer

    Declaration

    Swift

    public static func unicode61(
    diacritics: FTS5.Diacritics = .removeLegacy,
    separators: Set<Character> = [],
    tokenCharacters: Set<Character> = [])
    -> FTS5TokenizerDescriptor

    Parameters

    diacritics

    By default SQLite will strip diacritics from latin characters.
    separators

    Unless empty (the default), SQLite will consider these characters as token separators.
    tokenCharacters

    Unless empty (the default), SQLite will consider these characters as token characters.
    Show on GitHub