Learn practical skills, build real-world projects, and advance your career
Updated 3 years ago
source : https://realpython.com/python-web-scraping-practical-introduction/
One useful package for web scraping that you can find in Python’s standard library is urllib, which contains tools for working with URLs. In particular, the urllib.request module contains a function called urlopen() that can be used to open a URL within a program
from urllib.request import urlopenurl = "http://olympus.realpython.org/profiles/aphrodite"
page = urlopen(url) #To open the web page, pass url to urlopen():
#urlopen() returns an HTTPResponse object:
print(page)
html_bytes = page.read()
html = html_bytes.decode("utf-8")
print(html)<http.client.HTTPResponse object at 0x0000020B67A90A58>
<html>
<head>
<title>Profile: Aphrodite</title>
</head>
<body bgcolor="yellow">
<center>
<br><br>
<img src="/static/aphrodite.gif" />
<h2>Name: Aphrodite</h2>
<br><br>
Favorite animal: Dove
<br><br>
Favorite color: Red
<br><br>
Hometown: Mount Olympus
</center>
</body>
</html>
# or in a more simplistic manner
import requests
f = requests.get(url)
print(f.text)<html>
<head>
<title>Profile: Aphrodite</title>
</head>
<body bgcolor="yellow">
<center>
<br><br>
<img src="/static/aphrodite.gif" />
<h2>Name: Aphrodite</h2>
<br><br>
Favorite animal: Dove
<br><br>
Favorite color: Red
<br><br>
Hometown: Mount Olympus
</center>
</body>
</html>
url = "http://olympus.realpython.org/profiles/dionysus"
page = urlopen(url)
html = page.read().decode("utf-8")
print(html)<html>
<head>
<TITLE >Profile: Dionysus</title / >
</head>
<body bgcolor="yellow">
<center>
<br><br>
<img src="/static/dionysus.jpg" />
<h2>Name: Dionysus</h2>
<img src="/static/grapes.png"><br><br>
Hometown: Mount Olympus
<br><br>
Favorite animal: Leopard <br>
<br>
Favorite Color: Wine
</center>
</body>
</html>