w3resource

NLTK Tokenize: Exercises with Solution

Python NLTK Tokenize [9 exercises with solution]

What is Tokenize?

Tokenization is the process of demarcating and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input.

1. Write a Python NLTK program to split the text sentence/paragraph into a list of words.
Click me to see the sample solution

2. Write a Python NLTK program to tokenize sentences in languages other than English.
Click me to see the sample solution

3. Write a Python NLTK program to create a list of words from a given string.
Click me to see the sample solution

4. Write a Python NLTK program to split all punctuation into separate tokens.
Click me to see the sample solution

5. Write a Python NLTK program to tokenize words, sentence wise.
Click me to see the sample solution

6. Write a Python NLTK program to tokenize a twitter text.
Click me to see the sample solution

7. Write a Python NLTK program to remove Twitter username handles from a given twitter text.
Click me to see the sample solution

8. Write a Python NLTK program that will read a given text through each line and look for sentences. Print each sentence and divide two sentences with "==============".
Click me to see the sample solution

9. Write a Python NLTK program to find parenthesized expressions in a given string and divides the string into a sequence of substrings.
Click me to see the sample solution

 

[ Want to contribute to Python - NLTK exercises? Send your code (attached with a .zip file) to us at w3resource[at]yahoo[dot]com. Please avoid copyrighted materials.]



Python: Tips of the Day

Finding the most common elements in an iterable:

Example:

# collections.Counter lets you find the most common
# elements in an iterable:

import collections
c = collections.Counter('helloworld')

print(c)

print (c.most_common(3))

Output:

Counter({'l': 3, 'o': 2, 'h': 1, 'e': 1, 'w': 1, 'r': 1, 'd': 1})
[('l', 3), ('o', 2), ('h', 1)]