Labs · Size and tokens
BPE against WordPiece
Two merge rules on one editable word table — the most frequent pair against the highest-scoring one — and how each then splits a word it has never seen.
10 words · 20 merges
low, lower, lowest, new, newer, newest, wide, wider, widest, slow: the -er / -est endings become tokens.
Merge by merge, then a word neither has seen
Reads /api/labs/subword — loading
Computing…