Andrew B. Collier / @datawookie

The distances of Hasler kayak races for various divisions are nominally 4, 8 and 12 miles. However, the actual distances vary to some degree from one race venue to another. This makes it difficult to compare race times across different races. Using data from Paddle UK I attempt to estimate the actual distances.

Sometimes you’ll want to initiate a Selenium or Playwright session with an existing set of cookies. My approach to this is to retrieve those cookies using a browser and save them to a file so that I can easily load them into my script.

Sometimes a site will work fine with Selenium or Playwright until you try headless mode. Then it might fling up some anti-both mechanism. Or just stop responding altogether. Fortunately there are some simple things that you can do to work around this.

These are the approaches that I usually take.

Notes to self on using pytest.

I previously looked at the NetNut proxies. This post reviews the Webshare proxy service.

In the previous post we considered a few approaches to testing a Selenium web scraper. Now we’ll do the same for web scrapers using Playwright.

In previous posts we considered a few approaches for testing scrapers targeting static sites. Sometimes you won’t be able to get away with these static tools and you’ll be forced to use browser automation. In this post I’ll look at some options for testing a Selenium web scraper.

A common web crawler requirement is to iterate over a paginated list of links, following each link to retrieve detailed data. For example:

• iterating over a list of profiles on LinkedIn and then retrieving employment history from each profile;
• traversing a list of products on Amazon and then retrieving reviews for each product; or
• navigating the list of items on BBC News and retrieving the full articles.

What if your text data is contaminated with Unicode characters and HTML entities? Ideally you want your persisted data to be pristine. Metaphorically it should be prêt à manger (ready to eat). In principle I also want my text to be as simple as possible: ASCII characters, nothing else. This is sometimes achievable without the loss of too much information.

JSON-LD (JavaScript Object Notation for Linked Data) is a lightweight, flexible and standardised format intended to provide context and meaning to the data on a webpage. It’s easy and convenient for both humans and machines to read and write.

The previous post in this series considered the mocking capabilities in the unittest package. Now we’ll look at what it offers for patching.

Previous posts in this series used the responses and vcr packages to mock HTTP responses. Now we’re going to look at the capabilities for mocking in the unittest package, which is part of the Python Standard Library. Relative to responses and vcr this functionality is rather low-level. There’s more work required, but as a result there’s potential for greater control.

In the previous post I used the responses package to mock HTTP responses, producing tests that were quick and stable. Now I’ll look at an alternative approach to mocking using VCR.py.

As mentioned in the introduction to web scraper testing, unit tests should be self-contained and not involve direct access to the target website. The responses package allows you to easily mock the responses returned by a website, so it’s well suited to the job. The package is stable and well documented.

Site evolution. DOM drift. Selector decay. XPath extinction. Scraper rot. CAPTCHA catastrophe. Anti-bot apocalypse.

Inevitably even a carefully crafted web scraper will fail because the target site has changed in some way. Regular systematic testing is vital to ensure that you don’t lose valuable data.

The Zyte API implements session management, which makes it possible to emulate a browser session when interacting with a site via the API.

In a previous post I looked at various ways to use the Zyte API to retrieve web content. Now I’m going to delve into options for managing cookies via the Zyte API.

Zyte is a data extraction platform, useful for web scraping and data processing at scale. It’s intended to simplify data collection and, based on my experience certainly does!

Quick notes on the process of installing the CPLEX optimiser.

Quick notes on the process of installing the MOSEK optimiser.