Blog Posts by Andrew B. Collier / @datawookie


What is a Proxy?

A reception desk at an old world hotel.

A proxy is a server or software that acts as an intermediary between a client (often a web browser) and one or more servers, typically on the internet. Proxies are used for a variety of purposes, including improving security, enhancing privacy, managing network traffic, and bypassing restrictions.

Read More →

Migrating from GitLab Pages to Vercel

A bird with a white body and dark cap against a sky with scattered clouds.

I recently migrated this blog from GitLab Pages to Vercel. There were two main reasons for the move:

  1. The blog was taking too long to build on GitLab Pages, which hindered efficient updates and added unnecessary delays to my workflow. Admittedly, this was partially my own doing since my build process was far too complicated.
  2. I want to have greater control over redirects (specifically the ability to redirect URLs that didn’t end in a slash to ones that did, which was apparently important for SEO purposes).
Read More →

Avoiding Duplication

An image of a computer on a wooden desk. On either side of the computer is a table lamp. The computer screen has an image of two cows.

Avoiding data duplication is a persistent challenge with acquiring data from websites or APIs. You can try to brute force it: pull the data again and then compare it locally to establish whether it’s fresh or stale. But there are other approaches that, if supported, can make this a lot simpler.

Read More →

Bypassing Cloudflare with Cloudscraper

An image of blue sky with clouds.

Cloudflare is a service that aims improve the performance and security of websites. It operates as a content delivery network (CDN) to ensure faster load times and consequently better user experience. However, it also protects against online threats by filtering “malicious” traffic.

Web scraping requests are often deemed to be malicious (certainly by Cloudflare!) and thus blocked. There are various approaches to circumventing this, most of which involve running a live browser instance. For some applications though, this is a bit hammer for a small nail. The cloudscraper package provides a lightweight option for dealing with Cloudflare and has an API similar to the requests package.

Read More →

Unpacking cURL Commands

An abstract pattern, reminiscent of Paisley.

cURL is the ultimate Swiss Army Knife for interacting with network protocols. But to be honest, I really only scratch the surface of what’s possible. Usually my workflow is something like this:

  1. Copy a cURL command from my browser’s Developer Tools.
  2. Test out the cURL command in a terminal.
  3. Convert the cURL command into a programming language (normally Python or R).
  4. Prosper.

I’m going to take a look at my favourite online tool for converting a cURL command to code and then see what other tools there, focusing on Python and R as target languages.

Read More →

Updates to the Big Book of R

Title image for the "**Big Book of R**".

The Big Book of R provides a comprehensive and ever-growing overview of a broad selection of R programming books. It was created and is maintained by Oscar Baruffa. The collection began with approximately 100 books and, with the help of contributions from the R community, has subsequently expanded to over 400. The books are grouped into topics such as geospatial, machine learning, statistics, text analysis, and many more. The Big Book of R is an excellent resource for anyone learning R programming, whether they are a beginner or advanced user.

Read More →