The answer
Yes. Models can be designed around characters or bytes, but those are still computational input units. Many widely used text models instead choose subword vocabularies.
What is tokenization, and why is it necessary for LLMs?
Can a language model operate directly on characters or bytes?
More FAQs in What is tokenization, and why is it necessary for LLMs?
No. Tokenization supplies discrete units and IDs, while embeddings provide learned numerical vectors. These steps serve different purposes in a typical neural language-model pipeline.
No. Space-based splitting misses punctuation, languages without spaces, and subword structure. Tokenizers use model-specific rules that can divide or combine text in less obvious ways.
Still have questions?
Browse all our FAQs or reach out to our support team
Want to upskill yourself?
Your learning library
Small steps. Big possibilities.
Find your next course. Build something great.