tokenization

/ˌtoʊ.kən.aɪˈzeɪ.ʃən/

B2 · Upper intermediateTechnologyCommon
Explore 20,000+ words in Discover Filter by topic and level, and save words with one tap. Sign up free

What does “tokenization” mean?

  1. noun

    The act of turning text into tokens

    Tokenization breaks text into words and symbols for computer use.

Word forms

tokenizations

Often used as

  • the tokenization process is needed
  • tokenization helps break words apart
  • we run tokenization on text data
  • tokenization splits sentences into tokens

Related words and nuances

  • segmentation
  • processing
  • parsing
  • encoding
  • conversion
  • analysis

Related words

Explore more words in Discover →