Libraries¶

In [1]:

import requests, bs4, webbrowser
import pandas as pd
import re

Scraping¶

In [2]:

query = 'what is null pointer exception'
url_query = 'https://stackoverflow.com/search?q='+query

def vote_ans(child1):
    nums = re.findall(r'\d+', child1)
    if (len(nums)) == 2:
        return nums[0], nums[1]
    elif (len(nums)) == 1:
        return nums[0], 0
    else:
        return 0, 0
    
def link(child3):
    return 'https://stackoverflow.com'+re.findall(r'href="(.*?)"', child3)[0]
def title(child3):
    return re.findall(r'<a .*? title="(.*?)">', child3)[0]
def summary(child3):
    raw = re.findall(r'<div class="excerpt">((.|\n)*?)</div>', child3)[0][0]
    raw = raw.strip()
    remove_tags_re = re.compile(r"<.*?>")
    raw = remove_tags_re.sub("", raw)
    raw = raw.replace("&lt;", "<")
    raw = raw.replace("&gt;", ">")
    raw = raw.replace("&amp;", "&")
    return raw

results = pd.DataFrame(columns=['Title', 'Link', 'Summary', 'Vote', 'Answers'])
res = requests.get(url_query)
if res.status_code == requests.codes.ok:
    ressoup = bs4.BeautifulSoup(res.text, 'lxml')
    elems = ressoup.select('.question-summary')
    for i,e in enumerate(elems):
        for j, c in enumerate(e.children):
            if j == 1:
                vote, ans = vote_ans(str(c))
                vote, ans = int(vote), int(ans)
            if j == 3:
                t = title(str(c))
                l = link(str(c))
                s = summary(str(c))
        results = pd.concat([results, pd.DataFrame({
            'Title': [t], 'Link': [l], 'Summary': [s], 'Vote': [vote], 'Answers': [ans]
        })], axis=0)
results= results.reset_index(drop=True)

In [3]:

results

Out[3]:

	Title	Link	Summary	Vote	Answers
0	What is a NullReferenceException, and how do I...	https://stackoverflow.com/questions/4660142/wh...	. The .NET runtime then detects this error and...	2305	0
1	What is a NullPointerException, and how do I f...	https://stackoverflow.com/questions/218384/wha...	, but it does not actually contain a primitive...	3604	0
2	What is a smart pointer and when should I use ...	https://stackoverflow.com/questions/106508/wha...	. Hopefully raises some NULL pointer exception...	1812	0
3	Check whether a string is not null and not empty	https://stackoverflow.com/questions/3598770/ch...	ensuring you will not get a null pointer excep...	861	0
4	what is causing my Null Pointer Exception	https://stackoverflow.com/questions/13782424/w...	I know that there are lots of threads on Null ...	1	4
5	Facing Null Pointer Exception	https://stackoverflow.com/questions/29468824/f...	belongs to no group it hits a null pointer exc...	1	2
6	Null Pointer Exception in program	https://stackoverflow.com/questions/3478736/nu...	The Below code does not fail to compile but ho...	0	4
7	Getting a null pointer exception on this line ...	https://stackoverflow.com/questions/16780588/g...	if (methodCall.getClassName(cg.getConstantPool...	1	2
8	What is the reason of null pointer exception?	https://stackoverflow.com/questions/24647305/w...	am continuously getting null pointer exception...	0	1
9	what is the reason of the null pointer exception	https://stackoverflow.com/questions/19637054/w...	hi all i try to do calculator,i'm write the fo...	0	4
10	What is wrong with this code? (Array of object...	https://stackoverflow.com/questions/17431296/w...	(); //This line gives null pointer exceptio...	1	4
11	Class inside a function causing null pointer e...	https://stackoverflow.com/questions/12599477/c...	put it into the function, it will cause null p...	0	2
12	Java code generating a null pointer exception	https://stackoverflow.com/questions/14821107/j...	I'm getting this Null Pointer Exception:\n\nja...	4	5
13	Null Pointer Exception: addTextChangedListener	https://stackoverflow.com/questions/15916242/n...	="+URLEncoder.encode(destination.getText().toS...	0	3
14	Null pointer Exception - findViewById()	https://stackoverflow.com/questions/19078461/n...	Can anyone help me to find out what can be the...	46	10

Scraping2¶

In [4]:

query = 'what is null pointer exception'
url_query = 'https://stackoverflow.com/search?q='+query

res = requests.get(url_query)
if res.status_code == requests.codes.ok:
    ressoup = bs4.BeautifulSoup(res.text, 'lxml')
    elems = ressoup.select('.question-summary')
    results = ""
    for i,e in enumerate(elems):
        l = 'https://stackoverflow.com'+re.findall(r'href="(.*?)"', str(list(e.children)[3]))[0]
        t = re.findall(r'<a .*? title="(.*?)">', str(list(e.children)[3]))[0]
        results += str(i)+'\n'+t+'\n'+l+'\n\n'
    print(results)
else:
    print('Something went wrong')

0
What is a NullReferenceException, and how do I fix it?
https://stackoverflow.com/questions/4660142/what-is-a-nullreferenceexception-and-how-do-i-fix-it/4660186?r=SearchResults#4660186

1
What is a NullPointerException, and how do I fix it?
https://stackoverflow.com/questions/218384/what-is-a-nullpointerexception-and-how-do-i-fix-it/218510?r=SearchResults#218510

2
What is a smart pointer and when should I use one?
https://stackoverflow.com/questions/106508/what-is-a-smart-pointer-and-when-should-i-use-one/106614?r=SearchResults#106614

3
Check whether a string is not null and not empty
https://stackoverflow.com/questions/3598770/check-whether-a-string-is-not-null-and-not-empty/3598792?r=SearchResults#3598792

4
what is causing my Null Pointer Exception
https://stackoverflow.com/questions/13782424/what-is-causing-my-null-pointer-exception?r=SearchResults

5
Facing Null Pointer Exception
https://stackoverflow.com/questions/29468824/facing-null-pointer-exception?r=SearchResults

6
Null Pointer Exception in program
https://stackoverflow.com/questions/3478736/null-pointer-exception-in-program?r=SearchResults

7
Getting a null pointer exception on this line of code. What is a null pointer? And how do i fix
https://stackoverflow.com/questions/16780588/getting-a-null-pointer-exception-on-this-line-of-code-what-is-a-null-pointer-a?r=SearchResults

8
What is the reason of null pointer exception?
https://stackoverflow.com/questions/24647305/what-is-the-reason-of-null-pointer-exception?r=SearchResults

9
what is the reason of the null pointer exception
https://stackoverflow.com/questions/19637054/what-is-the-reason-of-the-null-pointer-exception?r=SearchResults

10
What is wrong with this code? (Array of objects, null pointer exception.)
https://stackoverflow.com/questions/17431296/what-is-wrong-with-this-code-array-of-objects-null-pointer-exception?r=SearchResults

11
Class inside a function causing null pointer exception
https://stackoverflow.com/questions/12599477/class-inside-a-function-causing-null-pointer-exception?r=SearchResults

12
Java code generating a null pointer exception
https://stackoverflow.com/questions/14821107/java-code-generating-a-null-pointer-exception?r=SearchResults

13
Null Pointer Exception: addTextChangedListener
https://stackoverflow.com/questions/15916242/null-pointer-exception-addtextchangedlistener?r=SearchResults

14
Null pointer Exception - findViewById()
https://stackoverflow.com/questions/19078461/null-pointer-exception-findviewbyid?r=SearchResults

In [ ]: