Recap of Previous Session
Best Practices for Web Scraping & Some Points to Ponder
Example 1: Searching and Downloading Images for ML Classification:https://google.com
Example 2: Scraping Comments from a YouTube Video for NLP:https://www.youtube.com/watch?v=mHONNcZbwDY
Example 3: Scraping Jobs from a Job Website: https://pk.indeed.com
Example 4: Scraping Tweets of a celebirty: https://twitter.com/login
Example 5: Scraping News Articles for a News Website: https://www.thenews.com.pk/today
Exercise:
tag[attribute='value']
input[id='username']
or input#username
input[class='form-control']
or input.form-control
input[any-attr='attr-value']
input.form-control[attr='value']
parent-locator > child-locator
div > input[attr = 'value']
/html/body/div[1]/div[3]/form/div[1]/div[1]/div[1]/div/div[2]/input
//input[@title='Search']
//input[contains(@id, 'userN')]
or //input[@id, 'userNname']
//tagname[starts-with(@attribute, 'initial partial value of attribute')]
//input[text() = 'text of the element')]
//tagname[@name='value' and @id='value']
//tagname[@name='value' or @id='value']
ancestor
: Locates ancestors of current node, which includes the parents upto the root node.descendant
: Locates descendants of current node, which includes the children upto the leaf node.child
: Locates the children of current node.parent
: Locates parent of the current node.
These days most of the web apps are using AJAX techniques. When a page is loaded by the browser, the elements within that page may load at different time intervals. This makes locating elements difficult: if an element is not yet present in the DOM, a locate function will raise an ElementNotVisibleException.
One can use time.sleep(10)
to make our script wait for exact 10 seconds before proceeding. One should avoid using these static wait statements rather should use the dynamic waits provided by Selenium Webdriver.
Many times the web elements are not interactable, not clickable, or not visible, and thats where you have to put the wait so that the page gets loaded and your script can find that particular web element and proceed further.
Implicit wait:
driver.implicitly_wait(30)
will wait for a maximum of 30 seconds before throwing a timeout exception. If all the web elements are available before 30 seconds, control will move to the next LOC.Explicit wait:
element = WebDriverWait(driver, 30).until(EC.presence_of_element_located(By.XPATH, 'xpath'))
will wait for a maximum of 30 seconds before throwing a timeout exception. If the specific web element becomes visible within 30 seconds, control will move to the next LOC.Fluent Wait is quite similar to explicit wait, where you can specify the polling frequency. See Selenium documentations for details: https://www.selenium.dev/documentation/webdriver/waits/
robots.txt
, is a standard used by websites to communicate with web crawlers/bots, informing them about which areas of the website can be scanned or scraped.robots.txt
file is mostly placed in a website's top level directory and is publically available. A sub-domain on a root domain can also have separate robots.txt
files.robots.txt
file provides instructions for bots, however, it can't actually enforce the instructions.
time.sleep(2)
in between requests.
You can get your User-Agent by typing
what is my user agent
in Google’s search bar.
You can get your IP by typing
what is my ip
in Google’s search bar.
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
#Create an instance of webdriver and load the google webpage
s = Service('/Users/arif/Documents/chromedriver')
myoptions = Options()
myoptions.headless = False # default settings
driver = Chrome(service=s, options=myoptions)
driver.maximize_window()
driver.get('https://google.com')
# locate the search textbox, enter the search string and press enter key
driver.implicitly_wait(30)
#tbox = driver.find_element(By.XPATH, '/html/body/div[1]/div[3]/form/div[1]/div[1]/div[1]/div/div[2]/input')
#tbox = driver.find_element(By.XPATH, "//input[@title='Search']")
tbox = driver.find_element(By.CSS_SELECTOR, "input[title='Search']")
tbox.send_keys("Cat")
# Instead of locating and clicking the search button, you can simplay press enter
time.sleep(2)
tbox.send_keys(Keys.ENTER)
# Locate the image tab and click it to visit the images tab
driver.implicitly_wait(30)
#menu_img_link = driver.find_element(By.XPATH, '/html/body/div[7]/div/div[4]/div/div[1]/div/div[1]/div/div[2]/a')
menu_img_link = driver.find_element(By.XPATH, '//*[@id="hdtb-msb"]/div[1]/div/div[2]/a')
menu_img_link.click()
driver.execute_script(JS)
method is used to synchronously execute JavaScript in the current window/frame.driver.execute_script('alert("Hello JavaScript")')
window.scrollTo()
method is used to perform scrolling operation. The pixels to be scrolled horizontally along the x-axis and pixels to be scrolled vertically along the y-axis are passed as parameters to the method.# Self-Scroll the entire page till you reach the bottom
last_height =driver.execute_script('return document.body.scrollHeight')
while True:
driver.execute_script('window.scrollTo(0,document.body.scrollHeight)')
time.sleep(4)
new_height =driver.execute_script('return document.body.scrollHeight')
if (new_height == last_height):
break
last_height = new_height
print("Done... Reached the bottom of the page")
Done... Reached the bottom of the page
screenshot()
method¶Two ways to take a screenshot:
driver.save_screenshot(filename)
Saves a screenshot of the current window to a PNG image file and returns a bool value
element.screenshot(filename)
saves a screenshot of the current element to a PNG image file and Returns a bool value
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# downlaod and save 40 images of cats
for i in range(1,41):
try:
WebDriverWait(driver, 5).until
(EC.presence_of_element_located((By.XPATH, '//*[@id="islrg"]/div[1]/div['+str(i)+']/a[1]/div[1]/img')))
cat_img = driver.find_element(By.XPATH, '//*[@id="islrg"]/div[1]/div['+str(i)+']/a[1]/div[1]/img')
cat_img.screenshot('/Users/arif/Downloads/cat_images/cat_img'+str(i)+'.png')
except:
continue
print("Done... Check the folder for images of cats")
Done... Check the folder for images of cats
driver.quit()
!ls /Users/arif/Downloads/cat_images
cat_img1.png cat_img17.png cat_img24.png cat_img32.png cat_img4.png cat_img10.png cat_img18.png cat_img26.png cat_img33.png cat_img40.png cat_img11.png cat_img19.png cat_img27.png cat_img34.png cat_img5.png cat_img12.png cat_img2.png cat_img28.png cat_img35.png cat_img6.png cat_img13.png cat_img20.png cat_img29.png cat_img36.png cat_img7.png cat_img14.png cat_img21.png cat_img3.png cat_img37.png cat_img8.png cat_img15.png cat_img22.png cat_img30.png cat_img38.png cat_img9.png cat_img16.png cat_img23.png cat_img31.png cat_img39.png
from matplotlib import image
from matplotlib import pyplot as plt
img1 = image.imread("/Users/arif/Downloads/cat_images/cat_img20.png")
plt.imshow(img1);
driver.quit()
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time
#Create an instance of webdriver and load/run the youtube video page
s = Service('/Users/arif/Documents/chromedriver')
myoptions = Options()
myoptions.headless = False # default settings
driver = Chrome(service=s, options=myoptions)
driver.maximize_window()
time.sleep(1)
driver.get('https://www.youtube.com/watch?v=mHONNcZbwDY&t=80s')
driver.implicitly_wait(30)
play = driver.find_element(By.XPATH, '//*[@id="movie_player"]/div[5]/button')
play.click()
# Perform three scrolls to get around 60 comments
for scroll in range(1, 4):
body = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.TAG_NAME, "body")))
body.send_keys(Keys.END)
time.sleep(12)
# Scrape the comments
comments = []
comments_list = driver.find_elements(By.CSS_SELECTOR,"#content-text" )
for comment in comments_list:
text = comment.text.strip()
comments.append(text)
# Scrape the authors who made the comments
authors = []
authors_list = driver.find_elements(By.ID,"author-text")
for author in authors_list:
text = author.text.strip()
authors.append(text)
# Save the comments in csv file
data = {'Authors':authors, 'Comments':comments}
df = pd.DataFrame(data, columns=['Authors', 'Comments'])
df.to_csv('hello.csv', index=False)
Authors | Comments | |
---|---|---|
0 | NaN | You're an absolute legend Lionel |
1 | Logene Temonio | It's 2022 and I'm listening to this Masterpiece ♡ |
2 | Joelfantastic ASMR | Going on 40 and this song is still one of my t... |
3 | NaN | ️ |
4 | Sofia | 2022 and i'm still addicted to this LEGENDARY ... |
5 | Daiane Santos | Sou fã do Lionel \nEle tem uma voz linda\nViva... |
6 | NaN | Um anjo está cantando. Uma pausa para alegrar ... |
7 | Marcia Corbin | Beautiful words...beautiful music...beautiful ... |
8 | Music Mane | This masterpiece gives me teary eyes every tim... |
9 | dlnnyc64 | Still hits close to the heart almost 40 years ... |
10 | Edmar Fernandes Couto | Isso que é música de valor.... muito sensivel ... |
11 | Yancy Johnson | It’s 2022 and THIS SONG STILL GOES STRONG!!! |
12 | Maria Rosa Helena Do Prado E Silva | Que hino!!!!!! Amooooo |
13 | Nina Schimel | Ano de 1984\nSó eu sei o que representa essa c... |
14 | Ceciliasantos | Eu amo este clipe ️ |
15 | Sra. Francisco | Um dos clipes mais belos do mundo! Sinto-me en... |
16 | João Marcos Rodrigues | Essa canção é emocionante, Lionel Richie é uma... |
17 | Billyn Ivy | Don’t worry you’re not the the only one listen... |
18 | Cicada3301 | Masterpiece. simply no words can describe the ... |
19 | Carmen Bazan | One morning in 1984 I was getting ready for wo... |
20 | NaN | I don’t care what I’m doing, whenever this son... |
21 | Elnadrion | Um anjo está cantando. Uma pausa para alegrar ... |
22 | Volnei Carvalho | Como estás músicas são perfeitas lembra da min... |
23 | adeyosola | Lionel Richie is one of the best song writers ... |
24 | Thisura Yapa | That guitar solo is next level. No wonder tha... |
25 | Luis Machado | Linda demais |
26 | DachshundsRule | This song brings me to tears, it was our song.... |
27 | Lourdes. Conceicaodossantos. | Amei essa cena, muito liinda! |
28 | Rita Silva | Lionel Richie você é o cara.charmoso maravilho... |
29 | drharshz | 2022 ..... This is golden! One of the best eve... |
30 | Sherry George | I grew up in the 80's. Some of the best music ... |
31 | Gabriel apaixonado music | Música linda |
32 | Angela Arrington | It's 2022,and I'm listening.never get tired of... |
33 | Marcos Pinto | Tempo excelente que não volta mais maravilhos... |
34 | Андрей Баздырев | As they say, this song is for all time! It is ... |
35 | Denise Hedden | 2022 and I'm still listening to Lionel Richie.... |
36 | Saif Messi | It's simply an eternal song and it will still ... |
37 | joshua jennings | So many memories of this song ..a masterpiece ... |
38 | Joel Jimenez | This song is enchanting and touches the soul. ... |
39 | Vilmara Reghini | Essa música faz parte da minha infância , meu ... |
driver.quit()
import pandas as pd
df = pd.read_csv('hello.csv')
pd.set_option('max_colwidth',150)
df
Authors | Comments | |
---|---|---|
0 | NaN | You're an absolute legend Lionel |
1 | Logene Temonio | It's 2022 and I'm listening to this Masterpiece ♡ |
2 | Joelfantastic ASMR | Going on 40 and this song is still one of my top 10. Still gives me goose bumps! |
3 | NaN | ️ |
4 | Sofia | 2022 and i'm still addicted to this LEGENDARY SONG.️ |
5 | Daiane Santos | Sou fã do Lionel \nEle tem uma voz linda\nViva anos 70,80,90 tempos bons,com músicas boas.\nHoje em dia só porcarias. |
6 | NaN | Um anjo está cantando. Uma pausa para alegrar o coração em tempos difíceis. |
7 | Marcia Corbin | Beautiful words...beautiful music...beautiful emotions...will always be timeless... |
8 | Music Mane | This masterpiece gives me teary eyes every time, a pleasing pain and joy. |
9 | dlnnyc64 | Still hits close to the heart almost 40 years ago. I can remember details of where I was and what my life was back then. I adored the video but ca... |
10 | Edmar Fernandes Couto | Isso que é música de valor.... muito sensivel a pessoa q compôs essa letra |
11 | Yancy Johnson | It’s 2022 and THIS SONG STILL GOES STRONG!!! |
12 | Maria Rosa Helena Do Prado E Silva | Que hino!!!!!! Amooooo |
13 | Nina Schimel | Ano de 1984\nSó eu sei o que representa essa canção pra mim....\nSaudades!! |
14 | Ceciliasantos | Eu amo este clipe ️ |
15 | Sra. Francisco | Um dos clipes mais belos do mundo! Sinto-me encantada, apaixonada! |
16 | João Marcos Rodrigues | Essa canção é emocionante, Lionel Richie é uma lenda |
17 | Billyn Ivy | Don’t worry you’re not the the only one listening to this masterpiece in 2021 |
18 | Cicada3301 | Masterpiece. simply no words can describe the feelings this song evokes . |
19 | Carmen Bazan | One morning in 1984 I was getting ready for work when I first heard this song . Called the radio station to enquire about the title, once I knew i... |
20 | NaN | I don’t care what I’m doing, whenever this song plays on the radio, I stop everything I’m doing and focus on Lionels amazing voice |
21 | Elnadrion | Um anjo está cantando. Uma pausa para alegrar o coração em tempos difíceis. |
22 | Volnei Carvalho | Como estás músicas são perfeitas lembra da minha infância ,pena que o tempo não volta |
23 | adeyosola | Lionel Richie is one of the best song writers of all time: this song is timeless ... it reminds me so much of my childhood \nEndless Love is also ... |
24 | Thisura Yapa | That guitar solo is next level. No wonder that this went to number one on the UK Singles Chart for six weeks (In 1984).\n\nSuch a nice song........ |
25 | Luis Machado | Linda demais |
26 | DachshundsRule | This song brings me to tears, it was our song. He's been gone long years now, and our children have grown up and have families of their own. Our o... |
27 | Lourdes. Conceicaodossantos. | Amei essa cena, muito liinda! |
28 | Rita Silva | Lionel Richie você é o cara.charmoso maravilhoso muito charme cantor otimo voz maravilhosa melodias lindas.gosto muito de ti.rita Silva Souza Salv... |
29 | drharshz | 2022 ..... This is golden! One of the best ever songs |
30 | Sherry George | I grew up in the 80's. Some of the best music came from the 70's and 80's. Lionel Ritchie's music is so beautiful and romantic. |
31 | Gabriel apaixonado music | Música linda |
32 | Angela Arrington | It's 2022,and I'm listening.never get tired of Lionel's voice. |
33 | Marcos Pinto | Tempo excelente que não volta mais maravilhosas canções e obrigado a cantor que Deus abençoe |
34 | Андрей Баздырев | As they say, this song is for all time! It is not often that you will find such a harmony of music, words and video in songs and clips. Thank you ... |
35 | Denise Hedden | 2022 and I'm still listening to Lionel Richie. I love this song it's so precious to me️ |
36 | Saif Messi | It's simply an eternal song and it will still an icon of all romantic songs for along time , greetings to the legend Lionel Richie from Iraq |
37 | joshua jennings | So many memories of this song ..a masterpiece of hard work and dedication |
38 | Joel Jimenez | This song is enchanting and touches the soul. It brings back memories of a lost love. Recorded in 1983 and released on Feb 13, 1984. This song r... |
39 | Vilmara Reghini | Essa música faz parte da minha infância , meu Deus como era tudo tao lindo e perfeito , meus pais os melhores do mundo , o amor que eles tiveram e... |
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time
#Create an instance of webdriver and go the the appropriate job page
s = Service('/Users/arif/Documents/chromedriver')
myoptions = Options()
myoptions.headless = False
driver = Chrome(service=s, options=myoptions)
driver.maximize_window()
driver.get('https://pk.indeed.com')
time.sleep(5)
# Enter your search parameters and click FindJobs button
what_box = driver.find_element(By.XPATH,'//*[@id="text-input-what"]')
where_box = driver.find_element(By.XPATH,'//*[@id="text-input-where"]')
button = driver.find_element(By.XPATH,'//*[@id="jobsearch"]/button')
what_box.send_keys('Full Stack Web Developer')
where_box.send_keys('Lahore')
button.click()
time.sleep(5)
# Function that scrape the three pieces of information of each job and is called on each page
jobtitles = []
companies = []
salaries = []
def jobs():
time.sleep(2)
postings = driver.find_elements(By.CSS_SELECTOR, '.resultContent')
for posting in postings:
try:
job_title = posting.find_element(By.CSS_SELECTOR,'h2 a').text
except:
job_title = 'No Job title'
try:
company = posting.find_element(By.CSS_SELECTOR,'.companyName').text
except:
company = "No company name"
try:
salary = posting.find_element(By.CSS_SELECTOR,'.salary-snippet-container').text
except:
salary = "No Salary"
companies.append(company)
salaries.append(salary)
jobtitles.append(job_title)
# Click the next page button in the pagination bar
while(True):
time.sleep(4)
try:
pop_up = driver.find_element(By.CSS_SELECTOR,'.popover-x-button-close.icl-CloseButton')
driver.find_element(By.CSS_SELECTOR,'.popover-x-button-close.icl-CloseButton').click()
except:
pass
jobs()
try:
driver.find_element(By.CLASS_NAME,'pagination-list')
driver.execute_script("arguments[0].scrollIntoView();", driver.find_element(By.CLASS_NAME,'pagination-list'))
try:
driver.find_element(By.XPATH,'//*[@aria-label="Next"]').click()
except:
break
except:
break
# # Writing in the file
data = {'Company':companies, 'Job Title':jobtitles, 'Salary':salaries}
df = pd.DataFrame(data, columns=['Company', 'Job Title', 'Salary'])
df.to_csv('jobs.csv', index=False)
df = pd.read_csv('jobs.csv')
df
Company | Job Title | Salary | |
---|---|---|---|
0 | Smart Placement (Pvt) Ltd | Junior/Intern Web Developer | No Salary |
1 | Digital Media Line | Web Developer Intern | Rs 5,000 - Rs 10,000 a month |
2 | ThingTrax | Jr. Full Stack Developer | Rs 50,000 a month |
3 | Alliance Solutions | Web Developer | Rs 25,000 a month |
4 | iDig Digital | Full Stack PHP Laravel Developer - Remote Posi... | Rs 60,000 - Rs 126,159 a month |
... | ... | ... | ... |
70 | VeriPark Software Solutions | Software Developer. NET | No Salary |
71 | Cinco Solutions | PHP Developers | Rs 30,000 - Rs 50,000 a month |
72 | FINCA International | Full Stack Web Developer | No Salary |
73 | Nextbridge Pvt Ltd | Senior MERN Developer | No Salary |
74 | ENTERTAINER FZ LLC | PHP Full Stack Web Developer | No Salary |
75 rows × 3 columns
driver.quit()
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time
import os
# Create an instance of webdriver and get the twitter login page
s = Service('/Users/arif/Documents/chromedriver')
myoptions = Options()
driver = Chrome(service=s, options=myoptions)
driver.maximize_window()
driver.get('https://twitter.com/login')
driver.implicitly_wait(30)
# Enter username and password
time.sleep(5)
username = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//input[@name="text"]')))
username.send_keys('username')
username.send_keys(Keys.ENTER)
time.sleep(2)
passwd = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, '//input[@name="password"]')))
passwd.send_keys(os.environ['yourtwitterpassword']) # actual passwd is saved in an environment variable :)
passwd.send_keys(Keys.ENTER)
# Enter Celebrity name (Imran Khan) in Search Textbox
time.sleep(2)
search_input = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH,'//input[@aria-label="Search query"]')))
search_input.send_keys("Imran Khan")
time.sleep(2)
search_input.send_keys(Keys.ENTER)
## Click on People tab for People Profiles using LINK_TEXT Locator
time.sleep(2)
people = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.LINK_TEXT, 'People')))
people.click()
# Click on the twitter link of Imran Khan
time.sleep(2)
click_imran = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.LINK_TEXT, 'Imran Khan')))
click_imran.click()
# Scrape username
user_name = driver.find_element(By.XPATH,'((//*[@data-testid="UserName"])//span)[last()]').text
# Scrape 25 tweets with their dates
articles = []
tweets = []
times=[]
while True:
time.sleep(1)
article = driver.find_elements(By.TAG_NAME,'article')
for a in article:
if a not in articles:
tweet = a.find_element(By.XPATH, './/*[@data-testid="tweetText"]')
articles.append(a)
t = a.find_element(By.XPATH,'.//time')
times.append(t.text)
tweets.append(tweet.text)
if len(tweets) >=25:
break
driver.execute_script("window.scrollBy(0,500);")
# Write scraped data in csv file
data = {'User':user_name, 'Times':times,'Tweets':tweets}
df = pd.DataFrame(data, columns=['User', 'Times','Tweets'])
df.to_csv('tweets.csv', index=False)
df = pd.read_csv('tweets.csv')
df
User | Times | Tweets | |
---|---|---|---|
0 | @ImranKhanPTI | 1h | The nation continues to suffer crushing econom... |
1 | @ImranKhanPTI | 1h | This was despite the sharp slow down after reg... |
2 | @ImranKhanPTI | 3 Jul | امپورٹڈ حکومت اوراسکےپشت پناہوں کیلئےمیرا واضح... |
3 | @ImranKhanPTI | 3 Jul | بدمعاشوں کا یہ ٹولہ جس انداز میں عوام کو مہنگا... |
4 | @ImranKhanPTI | 3 Jul | My clear message to Imported govt & its backer... |
5 | @ImranKhanPTI | 3 Jul | wealth, its only a matter of time before we g... |
6 | @ImranKhanPTI | 3 Jul | نہایت کثیر تعداد میں باہرنکلنے اور مجرموں کی ا... |
7 | @ImranKhanPTI | 1h | The nation continues to suffer crushing econom... |
8 | @ImranKhanPTI | 1h | This was despite the sharp slow down after reg... |
9 | @ImranKhanPTI | Jul 3 | امپورٹڈ حکومت اوراسکےپشت پناہوں کیلئےمیرا واضح... |
10 | @ImranKhanPTI | Jul 3 | بدمعاشوں کا یہ ٹولہ جس انداز میں عوام کو مہنگا... |
11 | @ImranKhanPTI | Jul 3 | My clear message to Imported govt & its backer... |
12 | @ImranKhanPTI | Jul 3 | wealth, its only a matter of time before we g... |
13 | @ImranKhanPTI | Jul 3 | نہایت کثیر تعداد میں باہرنکلنے اور مجرموں کی ا... |
14 | @ImranKhanPTI | Jul 3 | I want to thank the people of Islamabad & Pind... |
15 | @ImranKhanPTI | Jul 2 | میں پاکستان+دنیابھرمیں مقیم پاکستانیوں سےملتمس... |
16 | @ImranKhanPTI | Jul 2 | I request all Pakistanis living in Pakistan an... |
17 | @ImranKhanPTI | Jul 1 | لاہورمیں آج سینئرصحافی ایاز امیر پر تشدد کی شد... |
18 | @ImranKhanPTI | Jul 1 | I condemn in strongest terms the violence agai... |
19 | @ImranKhanPTI | Jul 1 | امپورٹڈحکومت کےسیاسی عدمِ استحکام اور موسمِ گر... |
20 | @ImranKhanPTI | Jul 1 | InshaAllah tomorrow will be our historic Islam... |
21 | @ImranKhanPTI | Jul 1 | روس سےسستا تیل خریدنےکی بجائےتبدیلئ سرکار کی س... |
22 | @ImranKhanPTI | Jul 1 | Instead of buying cheaper oil from Russia Impo... |
23 | @ImranKhanPTI | Jun 27 | Congratulations Ahmad Nawaz on being elected P... |
24 | @ImranKhanPTI | Jun 27 | اپنی ٹائیگر فورس، اپنے نوجوانوں اور اپنی خواتی... |
driver.quit()
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time
# Create an instance of webdriver and load the newspaper
s = Service('/Users/arif/Documents/chromedriver')
myoptions = Options()
driver = Chrome(service=s, options=myoptions)
driver.maximize_window()
driver.get('https://www.thenews.com.pk/today')
time.sleep(2)
# Create a list of all the URLs of your interest
urls = []
try:
s = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT,"Imran")))
search_urls = driver.find_elements(By.PARTIAL_LINK_TEXT,"Imran")
for i in search_urls:
urls.append(i.get_attribute("href"))
except:
print("I did not find it ")
for url in urls:
print(url)
https://www.thenews.com.pk/print/971449-shireen-asks-sc-to-take-notice-of-imran-s-phone-tapping https://www.thenews.com.pk/print/971615-gogi-pinky-economic-corruption-corridor-nawaz-made-cpec-imran-gpec-maryam https://www.thenews.com.pk/print/971591-amendments-to-nab-law-imran-files-appeal-in-sc-against-registrar-s-objections https://www.thenews.com.pk/print/971602-punjab-bypolls-imran-to-start-election-campaign-from-july-7 https://www.thenews.com.pk/print/971539-imran-ismail-made-pti-s-additional-secretary-general
original_window = driver.current_window_handle
news_articles = []
authors = []
headings = []
for url in urls:
driver.switch_to.new_window('tab')
driver.get(url)
try:
heading = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR,".detail-heading h1")))
author = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR,".category-source")))
article = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR,".story-detail")))
headings.append(heading.text)
news_articles.append(article.text)
authors.append(author.text)
except:
pass
driver.switch_to.window(original_window)
for heading in headings:
print(heading)
Shireen Mazari asks Supreme Court to take notice of Imran Khan’s ‘phone tapping’ ‘Gogi-Pinky Economic Corruption Corridor’: Nawaz made CPEC, Imran GPEC: Maryam Amendments to NAB law: Imran files appeal in SC against registrar’s objections Punjab bypolls: Imran to start election campaign from July 7 Imran Ismail made PTI’s additional secretary general
data = {'Headings':headings, 'Authors':authors, 'News Articles':news_articles}
df = pd.DataFrame(data, columns=['Headings', 'Authors', 'News Articles'])
df.to_csv('news.csv', index=False)
df = pd.read_csv('news.csv')
df
Headings | Authors | News Articles | |
---|---|---|---|
0 | Shireen Mazari asks Supreme Court to take noti... | By Mumtaz Alvi | PTI leaders, Shireen Mazari and Fawad Chaudhry... |
1 | ‘Gogi-Pinky Economic Corruption Corridor’: Naw... | By Our Correspondent | PML-N Vice President Maryam Nawaz addressing a... |
2 | Amendments to NAB law: Imran files appeal in S... | By Our Correspondent | ISLAMABAD: Former prime minister and Pakistan ... |
3 | Punjab bypolls: Imran to start election campai... | By Our Correspondent | PTI Chairman Imran Khan speaks during a media ... |
4 | Imran Ismail made PTI’s additional secretary g... | By Our Correspondent | The Pakistan Tehreek-e-Insaf on Monday appoint... |
df['News Articles'][0]
'PTI leaders, Shireen Mazari and Fawad Chaudhry addressing a press conference in islamabad on July 4, 2022. Photo: Screengrab of a Twitter video. \nISLAMABAD: Pakistan Tehreek-e-Insaf (PTI) Senior Vice-President and former human rights minister Dr Shireen Mazari said Monday the intelligence agencies illegally tap phones and the Supreme Court (SC) should take a suo motu notice of tapping then PM Imran Khan\'s phone.\nReferring to media reports, she said that another phone conversation on the secure line between Imran Khan and his principal secretary Azam Khan is also going to be leaked.\nMazari said what is in the audio tape is not the issue, insisting the real issue was phone tapping. She, along with PTI leader Chaudhry Fawad Hussain, told a hurriedly-called news conference here that an audio tape of the former first lady is circulating, which should be brought to light after a forensic test. She also asked how much assistance was given by the US in phone tapping.\nRelated Stories\nPTI warns against leaking Imran Khan’s audio calls\nGovt proposes forensic audit of Bushra Bibi’s audio\nDr Mazari emphasised that the real issue is phone tapping. The government of former premier Benazir Bhutto was ousted in 1997 on the issue of phone tapping and the Supreme Court in its judgment held that phone tapping is illegal under Articles 8 and 14 of the Constitution. The court had said in its judgment that official or personal conversations cannot be recorded.\nCiting an English daily, she pointed out that apart from this decision, there is a decision of Justice Saqib Nisar of 2015 in which one of the most interesting things was admission on part of country\'s prime intelligence agency that it recorded 6,856 phone calls in the month of May alone.\nShe said that intelligence agencies illegally tap phones because sensitive agencies have phone tapping technology. She remarked, “I ask why this series of illegal phone tapping continues despite Supreme Court orders as the secure line of Imran Khan\'s house was tapped. The Supreme Court should take a suo motu notice as to which agencies are there that are, despite court orders, taking illegal steps and violating the apex court order”.\nMazari contended that it has been reported that another audio is going to be leaked which is based on the conversation between Imran Khan and his principal secretary Azam Khan on the secure line. The former minister said if the audio is made public, it will not only violate the Supreme Court order, but also the Official Secret Act, and she will not remain silent on it.\nThe purpose of such acts, she claimed, was to hide the conspiracy and the nation has accepted the American conspiracy, so the neutrals and those who brought them are doing such acts so that the nation\'s attention is diverted from the issue.\nShireen Mazari said that after ‘our successful rallies, such conspirators, their handlers, neutrals and those they have brought, are nervous and trying to divert the attention of the people from this conspiracy and the country\'s complicated affairs in some way.\nThe IMF, she noted, was also asking this government to hold accountability against corruption while they (rulers) are not getting a place to flee. “I appeal to the defence institutions of Pakistan as to why they are pushing Pakistan to such difficult situations from where it becomes very difficult to return. The date of elections should be announced so that the nation elects its representatives and then form policies for the betterment of the country,” she contended.\nShe accused PMLN leader Maryam Nawaz of violating the Official Secret Act during her rallies every day, and wondered if this government and its handler are not violating the Constitution and law by showing official documents to a convicted person. “There are many similar questions that we want to ask neutrals and this government,” she said.\nSpeaking on the occasion, PTI Senior Vice President Chaudhry Fawad Hussain said that the loadshedding crisis in this country is getting serious and now it is being said that loadshedding will take place even during Eid holidays, while the government has admitted that all plants are on imported fuel and they do not have money to buy fuel.\nFawad said that the government has accepted all IMF demands to make petroleum products costlier, tax the public yet the IMF is not ready to pay, the IMF is saying it is hesitant to issue a package on changing anti-corruption laws.\nThe government, he claimed, immediately accepted the tax on the people but stuck to the issue of not enacting anti-corruption laws which gives an idea of the mindset of this government.\nHe alleged that the government changed the laws and gave itself a benefit of Rs1,100 billion as they gave themselves financial benefit under NRO, similarly Asif Zardari has been given NRO in the fake account case. “Therefore, Imran Khan says corruption makes your country poor and corruption worth billions of rupees of Sharif and Zardari family is hollowing out the roots of Pakistan,” he charged.\nFawad also said that phone tapping is a very important matter and phone tapping is being done in Pakistan and there is no monitoring, these calls are edited, not put to forensic test, human rights are being violated, allegations are being levelled against people.\nHe said that only yesterday, Farah Gujjar was accused by the PMLN of acquiring a plot in Faisalabad Industrial Zone at a price less than the market value, ‘while according to the documents we have, Ayaz Sadiq also acquired two plots at the same place, so Ayaz Sadiq also committed corruption’.\nPTI leader noted that Farah Gujjar was reportedly being issued red warrants. He asked how warrants could be issued when no case was registered against her and the purpose of these false allegation campaign issued by the PMLN is only that they know that Imran Khan is the leader who has no lust for money.\nFawad said that ‘once again we repeat that we want good relations with the US, Europe, Russia and Western countries but that does not mean that a country can tell us who will rule our country as we cannot allow this thing. “It does not mean that no one even invited you to the ceremony and still you go there forcibly,” he said.\n“After the accident that happened to Khawaja Asif at the age of 70-72, he suffers from psychological problems and often talks nonsense. If he has any evidence of contact with Donald Lu, he should make it public,” he said. He also talked on how \'rigging plan had been readied by PMLN leadership\' for upcoming by-election.\nMeanwhile, Minister for Information and Broadcasting Marriyum Aurangzeb Monday said PTI leaders Fawad Chaudhry and Shireen Mazari confirmed that Bushra Bibi is the head of PTI\'s social media, generating a false narrative.\nReferring to the news conference held earlier by these two PTI leaders, she said they confirmed Bushra Bibi was the ‘mastermind’ of the campaign of treason certificates and the drive against the institutions. “Fawad Chaudhry and Shireen Mazari have accepted that the audio is of Bushra Bibi and that ‘linking’ political opponents with treason, is her ‘opinion’.\nThe minister contented that the campaign against institutions was being run by Bushra Bibi, adding it was proved that an immoral press conference with dirty-language speaking spokespersons was being done at her behest.\n“Bushra Bibi is running a social media campaign against national institutions, the narratives of treason, external conspiracy while hiding in Bani Gala. Bushra Bibi is in the forefront and behind the drive to link political opponents with treason and bad-mouthing. She is the mastermind of the campaign against institutions,” the minister charged.\nMarriyum alleged that Bushra Bibi fabricated the narrative of external conspiracy and treason to hide her corruption. Bushra Bibi campaign against political opponents, journalists and institutions.\n“PTI admits to making treason cases to hide its corruption. The champions of the false narrative of conspiracy abroad have been caught apologising and cajoling the US. Imran Sahib is delivering "bygones are bygones" apology to the US,” she concluded. '