Blog Posts by Andrew B. Collier / @datawookie

If you use Selenium for browser automation then at some stage you are likely to need to download a file by clicking a button or link on a website. Sometimes this just works. Other times it doesn’t.

I needed an offline copy of an economic calendar with all of the major international economic events. After grubbing around the internet I found the Economic Calendar on Myfxbook which had everything that I needed.

A few months ago I listened to an episode on the Founder’s Journal podcast that reviewed an essay, The Opportunity Cost of Everything, by Jack Raines. If you haven’t read it, then I suggest you invest 10 minutes in doing so. It will be time well spent.

Cloudflare is a service that aims improve the performance and security of websites. It operates as a content delivery network (CDN) to ensure faster load times and consequently better user experience. However, it also protects against online threats by filtering “malicious” traffic.

Web scraping requests are often deemed to be malicious (certainly by Cloudflare!) and thus blocked. There are various approaches to circumventing this, most of which involve running a live browser instance. For some applications though, this is a bit hammer for a small nail. The cloudscraper package provides a lightweight option for dealing with Cloudflare and has an API similar to the requests package.

cURL is the ultimate Swiss Army Knife for interacting with network protocols. But to be honest, I really only scratch the surface of what’s possible. Usually my workflow is something like this:

1. Copy a cURL command from my browser’s Developer Tools.
2. Test out the cURL command in a terminal.
3. Convert the cURL command into a programming language (normally Python or R).
4. Prosper.

I’m going to take a look at my favourite online tool for converting a cURL command to code and then see what other tools there, focusing on Python and R as target languages.

The Big Book of R provides a comprehensive and ever-growing overview of a broad selection of R programming books. It was created and is maintained by Oscar Baruffa. The collection began with approximately 100 books and, with the help of contributions from the R community, has subsequently expanded to over 400. The books are grouped into topics such as geospatial, machine learning, statistics, text analysis, and many more. The Big Book of R is an excellent resource for anyone learning R programming, whether they are a beginner or advanced user.

The ability to specify a message ID in emails sent from the {emayili} package makes it possible to create email threads.

A new minor version of the openai-python package was released late on Friday 7 June 2024, only a couple of days after the last minor release. This release adds a chunking_strategy argument to the methods for adding files to vector stores.

This question on Stack Overflow was a fun challenge: extract the markers off an embedded Google Map.

The R version of my Desert Island Docker talk. Similar idea to Desert Island Docker: Python Edition.

Over the years that I’ve been dabbling in public speaking I’ve generally developed a talk, presented it once and then moved on. However, I’ve noticed other speakers who give the same (or similar) talk at different events, where the talk evolves and improves over time.

From time to time I want to extract the table of contents from a PDF. Here’s how I do that using simple shell tools.

The {emayili} package has adapters which make it simple to send email via a variety of services. For example, it caters specifically for ZeptoMail, MailerSend, Mailfence and Sendinblue. The latest version of {emayili}, 0.8.0 published on 23 April 2024, adds an an adapter for Mailtrap.

The key to successful backtesting is to ensure that you only use the data that were available at the time of the prediction. No “future” data can be included in the model training set, otherwise the model will suffer from look-ahead bias (having unrealistic access to future data).

Are you trying to build and push a Docker image from GitHub Actions?

The Two-Fund Separation Theorem introduced by James Tobin, a Nobel Prize-winning economist, is a fundamental concept in investment theory. It addresses how investors can optimally allocate their assets. In an efficient market an optimal portfolio is a combination of a risk-free asset and a market portfolio.

Sprinkling status messages across you code using print() statements can be a good temporary fix for tracking down issues in your code.

But it can get messy and there’s no way for you to selectively disable them. Sure you can redirect output to a file or /dev/null but this is an all-or-nothing solution. How about disabling some messages and retaining others?

This is where the logging module comes into its own.