BPE

November 15, 2026 · 16 min read

NLP #3: Byte-Pair Encoding

Train a tiny tokenizer by hand: merge the most frequent pair of tokens again and again, then watch it split words it has never seen.

• NLP
• Machine Learning
• Tokenization
• BPE
• Words and Tokens
• Learning in Public