The answer
The tokenizer typically applies an already learned vocabulary and merge rules during inference. Learning those rules is a separate preparation step from generating an answer.
How does a tokenizer divide text into smaller units?
Does byte-pair encoding merge words while a model is answering?
More FAQs in How does a tokenizer divide text into smaller units?
Some vocabularies include whitespace as part of a token or encode word boundaries explicitly. A leading space can therefore change the units selected for otherwise identical letters.
Many tokenizer libraries provide offset mappings that connect tokens to original character spans. The precise behavior depends on normalization, special tokens, and the library being used.
Still have questions?
Browse all our FAQs or reach out to our support team
Want to upskill yourself?
Your learning library
Small steps. Big possibilities.
Find your next course. Build something great.