Python Web Scraping Exercises

Python Web Scraping: Scrap number of tweets of a given Twitter account

Python Web Scraping: Exercise-20 with Solution

Write a Python program to scrap number of tweets of a given Twitter account.

Sample Solution:-

Python Code:

from bs4 import BeautifulSoup
import requests
handle = input('Input your account name on Twitter: ')
ctr = int(input('Input number of tweets to scrape: '))
res=requests.get('https://twitter.com/'+ handle)
all_tweets = bs.find_all('div',{'class':'tweet'})
if all_tweets:
  for tweet in all_tweets[:ctr]:
    context = tweet.find('div',{'class':'context'}).text.replace("\n"," ").strip()
    content = tweet.find('div',{'class':'content'})
    header = content.find('div',{'class':'stream-item-header'})
    user = header.find('a',{'class':'account-group js-account-group js-action-profile js-user-profile-link js-nav'}).text.replace("\n"," ").strip()
    time = header.find('a',{'class':'tweet-timestamp js-permalink js-nav js-tooltip'}).find('span').text.replace("\n"," ").strip()
    message = content.find('div',{'class':'js-tweet-text-container'}).text.replace("\n"," ").strip()
    footer = content.find('div',{'class':'stream-item-footer'})
    stat = footer.find('div',{'class':'ProfileTweet-actionCountList u-hiddenVisually'}).text.replace("\n"," ").strip()
    if context:
    print("List is empty/account name not found.")


Input your account name on Twitter:  @prasanta
Input number of tweets to scrape:  2
Prasanta Kundu Retweeted
WorkApps‏ @WorkAppsInc 13 Nov 2017
A Dedicated Mobile App for your Sales Teams to keep All Product Knowledge handy. Works even when Offline!! Log on to http://www.KnowledgeApps.ai  to know more!pic.twitter.com/5F7v1CRHma
0 replies     3 retweets     2 likes

Prasanta Kundu Retweeted
MVS Murthy‏ @mittispeaks 8 Nov 2017
This one is for all the #BFSI folks ... #HRManager #HRTech Help your sales team sell better. Complete #Product #Knowledge on a dedicated mobile app which works even in #offline mode.  Know more @ http://www.knowledgeapps.ai pic.twitter.com/dwsOpFgX0A
0 replies     2 retweets     0 likes


Python Web Scraping Flowchart: Scrap number of tweets of a given Twitter account.

Python Code Editor:

Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Write a Python program to count number of tweets by a given Twitter account.
Next: Write a Python program to find the live weather report (temperature, wind speed, description and weather) of a given city.

What is the difficulty level of this exercise?

Python: Tips of the Day

Merges two or more lists into a list of lists, combining elements from each of the input lists based on their positions


def tips_merge(*args, fill_value=None):
  max_length = max([len(lst) for lst in args])
  result = []
  for i in range(max_length):
      args[k][i] if i < len(args[k])
	  else fill_value for k in range(len(args))
  return result
print(tips_merge(['x', 'y'], [1, 2], [True, False])) 
print(tips_merge(['x'], [1, 2], [True, False])) 
print(tips_merge(['x'], [1, 2], [True, False], fill_value = '_')) 


[['x', 1, True], ['y', 2, False]]
[['x', 1, True], [None, 2, False]]
[['x', 1, True], ['_', 2, False]]