{"slug":"tokenization","sectionId":"nlp","title":"Tokenization for modern LLMs","summary":"Characters, words, subwords, BPE, WordPiece, Unigram, vocabulary, special tokens, padding, truncation, and masks.","focus":["BPE","WordPiece","special tokens","attention masks"],"featureIdea":"BPE/WordPiece/Unigram tokenizer comparator for Russian and English text plus token cost estimates.","status":"shell","requiresBackend":true,"tags":["tokenization","bpe","llm"],"locale":"en","sectionTitle":"NLP","statusLabel":"shell","backendLabel":"backend needed later","pagePath":"/learn/tokenization","apiPath":"/api/learning/tokenization","selfCheck":["What is the core job of \"Tokenization for modern LLMs\"?","Which common mistake would break a production implementation of this topic?","Which inputs or limits must be validated before the interactive feature ships?","What is the smallest test that proves the future implementation behaves correctly?","When does this module really need backend compute, and when is a UI simulation enough?"],"implementationNotes":["Start with one focused feature, not a full course inside one page.","All public inputs must be typed, bounded, and covered by reject-case tests.","If a model, dataset, or job is added, document source, license, limits, and fallback.","The interaction must explain the topic rather than serve as decoration."]}