The answer
Yes. Separate vocabulary items may acquire related numerical representations during training, even when their integer IDs are far apart in the vocabulary table.
What is the difference between a token ID and an embedding?
Can two token IDs have similar embeddings?
More FAQs in What is the difference between a token ID and an embedding?
The initial lookup vector is typically fixed for a given set of model weights. Contextual representations produced by later layers can differ between occurrences.
Not necessarily. An image encoder can produce vectors from pixels or patches without using text vocabulary IDs. The input representation depends on the architecture.
Still have questions?
Browse all our FAQs or reach out to our support team
Want to upskill yourself?
Your learning library
Small steps. Big possibilities.
Find your next course. Build something great.