NLTK Tokenize: Create a list of words from a given string
NLTK Tokenize: Exercise-3 with Solution
Write a Python NLTK program to create a list of words from a given string.
Sample Solution:
Python Code-1:
from nltk.tokenize import word_tokenize
text = "Joe waited for the train. The train was late. Mary and Samantha took the bus. I looked for Mary and Samantha at the bus station."
print("\nOriginal string:")
print(text)
print("\nList of words:")
print(word_tokenize(text))
Sample Output:
Original string: Joe waited for the train. The train was late. Mary and Samantha took the bus. I looked for Mary and Samantha at the bus station. List of words: ['Joe', 'waited', 'for', 'the', 'train', '.', 'The', 'train', 'was', 'late', '.', 'Mary', 'and', 'Samantha', 'took', 'the', 'bus', '.', 'I', 'looked', 'for', 'Mary', 'and', 'Samantha', 'at', 'the', 'bus', 'station', '.']
It's equivalent to the following code:
Python Code-2:
from nltk.tokenize import TreebankWordTokenizer
tokenizer = TreebankWordTokenizer()
text = "Joe waited for the train. The train was late. Mary and Samantha took the bus. I looked for Mary and Samantha at the bus station."
print("\nOriginal string:")
print(text)
print("\nList of words:")
print(tokenizer.tokenize(text))
Output:
Original string: Joe waited for the train. The train was late. Mary and Samantha took the bus. I looked for Mary and Samantha at the bus station. List of words: ['Joe', 'waited', 'for', 'the', 'train.', 'The', 'train', 'was', 'late.', 'Mary', 'and', 'Samantha', 'took', 'the', 'bus.', 'I', 'looked', 'for', 'Mary', 'and', 'Samantha', 'at', 'the', 'bus', 'station', '.']
Have another way to solve this solution? Contribute your code (and comments) through Disqus.
Previous: Write a Python NLTK program to tokenize sentences in languages other than English.
Next: Write a Python NLTK program to split all punctuation into separate tokens.
What is the difficulty level of this exercise?
Test your Programming skills with w3resource's quiz.
Python: Tips of the Day
Get the Key Whose Value Is Maximal in a Dictionary:
>>> model_scores = {'model_a': 100, 'model_z': 198, 'model_t': 150} >>> # workaround >>> keys, values = list(model_scores.keys()), list(model_scores.values()) >>> keys[values.index(max(values))] 'model_z' >>> # one-line >>> max(model_scores, key=model_scores.get) 'model_z'
- Exercises: Weekly Top 12 Most Popular Topics
- Pandas DataFrame: Exercises, Practice, Solution
- Conversion Tools
- JavaScript: HTML Form Validation
- SQL Exercises, Practice, Solution - SUBQUERIES
- C Programming Exercises, Practice, Solution : For Loop
- Python Exercises, Practice, Solution
- Python Data Type: List - Exercises, Practice, Solution
- C++ Basic: Exercises, Practice, Solution
- SQL Exercises, Practice, Solution - exercises on Employee Database
- SQL Exercises, Practice, Solution - exercises on Movie Database
- SQL Exercises, Practice, Solution - exercises on Soccer Database
- C Programming Exercises, Practice, Solution : Recursion