![An image of blue sky with clouds.](https://datawookie.dev/blog/2024/07/bypassing-cloudflare-with-cloudscraper/clouds_hu1644d171fb7ca0d26e83518ff236f0e9_697704_0x0_q75_h2__2.webp)
Cloudflare is a service that aims improve the performance and security of websites. It operates as a content delivery network (CDN) to ensure faster load times and consequently better user experience. However, it also protects against online threats by filtering “malicious” traffic.
Web scraping requests are often deemed to be malicious (certainly by Cloudflare!) and thus blocked. There are various approaches to circumventing this, most of which involve running a live browser instance. For some applications though, this is a bit hammer for a small nail. The cloudscraper
package provides a lightweight option for dealing with Cloudflare and has an API similar to the requests
package.
Sites using Cloudflare
Take a look at the list of sites using Cloudflare. We’ll pick the first item on the list, OpenAI, as a test target.
![OpenAI home page.](openai-screenshot.webp)
Setup
Install the cloudscraper
and requests
Python packages.
beautifulsoup4==4.12.3
cloudscraper==1.2.71
requests==2.32.3
Also throw in beautifulsoup4
so that we can parse a response… when we get one!
Using Requests
Let’s try retrieving the content of the OpenAI homepage using a GET request via requests
.
import requests
response = requests.get("https://openai.com/")
print(response.status_code)
Not surprisingly, Cloudflare intervenes and we get a 403 “Forbidden” response.
Using Cloudscraper
Maybe we’ll have more success using the cloudscraper
package?
import cloudscraper
from bs4 import BeautifulSoup
scraper = cloudscraper.CloudScraper()
response = scraper.get("http://openai.com")
print(response.status_code)
soup = BeautifulSoup(response.text, "html.parser")
banner = soup.select_one("h2")
print(banner.text)
And the result?
200
ChatGPT on your desktop
Success!
There are lots of options for tweaking the Cloudscraper configuration. For example, you can specify the browser type (Chrome or Firefox), platform (Linux, Windows, Darwin, Android or IOS) and whether or not it’s on desktop or mobile. You can also choose from a selection of JavaScript engines.
import cloudscraper
URL = "http://openai.com"
scraper = cloudscraper.CloudScraper(
# Browser specifications.
browser={
"browser": "firefox",
"platform": "linux",
"desktop": True,
"mobile": False
},
# JavaScript Engine
interpreter="nodejs",
# debug=True
)
response = scraper.get(URL)
print(response.status_code)
Various third party Captcha solvers, like 2captcha, anticaptcha and CapSolver, are also supported.