#tokenization

0 blog · 0 news · 4 wiki

§03

Wiki

A tokenisation algorithm that builds a sub-word vocabulary by iteratively merging the most frequent character pairs.

Google's likelihood-driven sub-word algorithm, similar in spirit to BPE and used by BERT.

Google's language-agnostic tokeniser library that treats whitespace as just another character.

The smallest unit a language model processes — a word fragment, character, or symbol.