Predictive Hacks

There are a lot of paid services that are providing google results and it’s for a reason. In the right hands, google results can be gold. In this post, we will show you how you can get the results in a few lines of code for free.

#importing the libraries we will need
import pandas as pd
import numpy as np
import urllib
import requests
import re
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup

The key here is to build the google URL using our keyword and the number of results. To do this we have to encode the keyword into HTML using urllib and add the id to the URL. Let’s say our keyword is “elbow method python”.

keyword= "elbow method python"
html_keyword= urllib.parse.quote_plus(keyword)
print(html_keyword)
'elbow+method+python'

Now let’s build the google URL

number_of_result=20
print(google_url)
'https://www.google.com/search?q=elbow+method+python&num=20'

We need now to hit the URL and get the results. Beautiful Soup will help us with that.

headers = {"User-Agent": "Mozilla/5.0"}
soup = BeautifulSoup(response.text, "html.parser")

The only thing we need now is regular expressions to extract the information we want.

result = soup.find_all('div', attrs = {'class': 'ZINbbc'})
results=[re.search('\/url\?q\=(.*)\&sa',str(i.find('a', href = True)['href'])) for i in result if "url" in str(i)]

#this is because in rare cases we can't get the urls
links=[i.group(1) for i in results if i != None]

links
['https://predictivehacks.com/k-means-elbow-method-code-for-python/',
'https://www.scikit-yb.org/en/latest/api/cluster/elbow.html',
'https://pythonprogramminglanguage.com/kmeans-elbow-method/',
'https://www.geeksforgeeks.org/elbow-method-for-optimal-value-of-k-in-kmeans/',
'https://blog.cambridgespark.com/how-to-determine-the-optimal-number-of-clusters-for-k-means-clustering-14f27070048f',
'https://stackoverflow.com/questions/41540751/sklearn-kmeans-equivalent-of-elbow-method',
'https://campus.datacamp.com/courses/cluster-analysis-in-python/k-means-clustering-3%3Fex%3D6',
'https://github.com/topics/elbow-method',
'https://github.com/topics/elbow-method%3Fl%3Dpython',
'https://towardsdatascience.com/clustering-metrics-better-than-the-elbow-method-6926e1f723a6',
'https://vitalflux.com/k-means-elbow-point-method-sse-inertia-plot-python/',
'https://www.kdnuggets.com/2019/10/clustering-metrics-better-elbow-method.html',
'https://realpython.com/k-means-clustering-python/',
'https://pyclustering.github.io/docs/0.8.2/html/d3/d70/classpyclustering_1_1cluster_1_1elbow_1_1elbow.html',
'https://jtemporal.com/kmeans-and-elbow-method/']

And this is how you can scrape Google results using python. If you want to go even further you can use a VPN so you can have google results from different Countries and Cities.

Let’s sum it up in a single function.

def google_results(keyword, n_results):
query = keyword
query = urllib.parse.quote_plus(query) # Format into URL encoding
number_result = n_results
soup = BeautifulSoup(response.text, "html.parser")
result = soup.find_all('div', attrs = {'class': 'ZINbbc'})
results=[re.search('\/url\?q\=(.*)\&sa',str(i.find('a', href = True)['href'])) for i in result if "url" in str(i)]
links=[i.group(1) for i in results if i != None]
return (links)
google_results('machine learning in python', 10)
['https://www.coursera.org/learn/machine-learning-with-python',
'https://www.w3schools.com/python/python_ml_getting_started.asp',
'https://machinelearningmastery.com/machine-learning-in-python-step-by-step/',
'https://www.tutorialspoint.com/machine_learning_with_python/index.htm',
'https://towardsai.net/p/machine-learning/machine-learning-algorithms-for-beginners-with-python-code-examples-ml-19c6afd60daa',
'https://www.edx.org/course/machine-learning-with-python-a-practical-introduct',
'https://scikit-learn.org/',
'https://www.geeksforgeeks.org/introduction-machine-learning-using-python/']

1. I got “‘NoneType’ object is not subscriptable” if I use more than 1 word in the google_results function.. :C

• Hi John! Thanks for your feedback. It seems that Google made some changes. It is fixed now and the code is updated!
We only need to add this if "URL" in str(i) at the end of the list comprehension in the results list.

2. Hey!

i have used this a lot, but know it doesn’t seem like working.

I use the exactly same code as you, but it doesn’t return any results now.

are you having the same issues, when you run it?

3. Hey Again

I have just tried to find the solution.

I think the reason why i ain’t recieving any results, is due to a cookie box, do you have any work arounds?

This is the code i used to see the pop up box

import urllib

import requests
from fake_useragent import UserAgent
from bs4 import BeautifulSoup

query = “microsoft”
query = urllib.parse.quote_plus(query) # Format into URL encoding
number_result = 5

ua = UserAgent()

soup = BeautifulSoup(response.text, “html.parser”)
print(soup)

4. Same issue here. It used to work but now the function returns an empty list. Any thoughts?

• Hi Guido, I just updated the code and now it’s working.

5. Thanks for your feedback, I just updated the code.

6. Hello and thanks for the code!

While it does the work perfectly to extract the links, I am trying to use the soup variable to extract other information like for example the wiki info box and for that I get no results. Is there a different approach when scraping other info?

Get updates and learn from the best

Miscellaneous

How to Redirect and Save Errors in Unix

In Unix, there are three types of redirection such as: Standard Input (stdin) that is denoted by 0. Usually, it’s

Python

Content-Based Recommender Systems with TensorFlow Recommenders

In this post, we will consider as a reference point the “Building deep retrieval models” tutorial from TensorFlow and we