Tokenization
Tutorial · Beginner · 21 min
A model never sees your text. It sees tokens: chunks of characters produced by a compression algorithm learned from training data.
TokensModels
Includes Colab notebook
By: Anacodic Team
Start Reading →Beginner · 3 articles · MODULE 4: LLM ENGINEERING
Tutorial · Beginner · 21 min
A model never sees your text. It sees tokens: chunks of characters produced by a compression algorithm learned from training data.
Includes Colab notebook
By: Anacodic Team
Start Reading →Tutorial · Beginner · 22 min
A language model does exactly one thing: given the tokens so far, output a probability for every possible next token.
By: Anacodic Team
Start Reading →Tutorial · Beginner · 21 min
A model hands you a probability for every possible next token; sampling is the separate step that picks one, and it is not part of the model.
Includes Colab notebook
By: Anacodic Team
Start Reading →