NLTK provides a function called word_tokenize() for splitting strings into tokens (nominally words). It splits tokens based on white space and punctuation. For example, commas and periods are taken as separate tokens. Contractions are split apart (e.g. “What's” becomes “What” “'s“).

What does word_tokenize return?

word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables. Return : Return the list of syllables of words.

What is NLTK word_tokenize?

word_tokenize is a function in Python that splits a given sentence into words using the NLTK library. ... In Python, we can tokenize with the help of the Natural Language Toolkit ( NLTK ) library.