What does “tokenization” mean?
-
noun
The act of turning text into tokens
Tokenization breaks text into words and symbols for computer use.
Word forms
tokenizations
Often used as
- the tokenization process is needed
- tokenization helps break words apart
- we run tokenization on text data
- tokenization splits sentences into tokens
Related words and nuances
- segmentation
- processing
- parsing
- encoding
- conversion
- analysis