Andrew B. Collier / @datawookie

A few months ago I listened to an episode on the Founder’s Journal podcast that reviewed an essay, The Opportunity Cost of Everything, by Jack Raines. If you haven’t read it, then I suggest you invest 10 minutes in doing so. It will be time well spent.

Cloudflare is a service that aims improve the performance and security of websites. It operates as a content delivery network (CDN) to ensure faster load times and consequently better user experience. However, it also protects against online threats by filtering “malicious” traffic.

Web scraping requests are often deemed to be malicious (certainly by Cloudflare!) and thus blocked. There are various approaches to circumventing this, most of which involve running a live browser instance. For some applications though, this is a bit hammer for a small nail. The cloudscraper package provides a lightweight option for dealing with Cloudflare and has an API similar to the requests package.

cURL is the ultimate Swiss Army Knife for interacting with network protocols. But to be honest, I really only scratch the surface of what’s possible. Usually my workflow is something like this:

1. Copy a cURL command from my browser’s Developer Tools.
2. Test out the cURL command in a terminal.
3. Convert the cURL command into a programming language (normally Python or R).
4. Prosper.

I’m going to take a look at my favourite online tool for converting a cURL command to code and then see what other tools there, focusing on Python and R as target languages.

The Big Book of R provides a comprehensive and ever-growing overview of a broad selection of R programming books. It was created and is maintained by Oscar Baruffa. The collection began with approximately 100 books and, with the help of contributions from the R community, has subsequently expanded to over 400. The books are grouped into topics such as geospatial, machine learning, statistics, text analysis, and many more. The Big Book of R is an excellent resource for anyone learning R programming, whether they are a beginner or advanced user.

The ability to specify a message ID in emails sent from the {emayili} package makes it possible to create email threads.

A new minor version of the openai-python package was released late on Friday 7 June 2024, only a couple of days after the last minor release. This release adds a chunking_strategy argument to the methods for adding files to vector stores.

Quick notes on installing Docker on various platforms:

• Linux
• macOS and
• Windows.

This question on Stack Overflow was a fun challenge: extract the markers off an embedded Google Map.

The R version of my Desert Island Docker talk. Similar idea to Desert Island Docker: Python Edition.

Over the years that I’ve been dabbling in public speaking I’ve generally developed a talk, presented it once and then moved on. However, I’ve noticed other speakers who give the same (or similar) talk at different events, where the talk evolves and improves over time.

From time to time I want to extract the table of contents from a PDF. Here’s how I do that using simple shell tools.

The {emayili} package has adapters which make it simple to send email via a variety of services. For example, it caters specifically for ZeptoMail, MailerSend, Mailfence and Sendinblue. The latest version of {emayili}, 0.8.0 published on 23 April 2024, adds an adapter for Mailtrap.

The key to successful backtesting is to ensure that you only use the data that were available at the time of the prediction. No “future” data can be included in the model training set, otherwise the model will suffer from look-ahead bias (having unrealistic access to future data).

Are you trying to build and push a Docker image from GitHub Actions?

The Two-Fund Separation Theorem introduced by James Tobin, a Nobel Prize-winning economist, is a fundamental concept in investment theory. It addresses how investors can optimally allocate their assets. In an efficient market an optimal portfolio is a combination of a risk-free asset and a market portfolio.

Sprinkling status messages across you code using print() statements can be a good temporary fix for tracking down issues in your code.

But it can get messy and there’s no way for you to selectively disable them. Sure you can redirect output to a file or /dev/null but this is an all-or-nothing solution. How about disabling some messages and retaining others?

This is where the logging module comes into its own.

Setting the values of one or more parameters for a GARCH model or applying constraints to the range of permissible values can be useful.

The two quantities we have been modelling (the time-dependent average and standard deviation of the returns) represent respectively the (potential) risk and reward associated with an asset. The relationship between these two quantities is implicit in the GARCH model. However, sometimes the return depends directly on the risk. A variant of the GARCH model can take this explicit relationship into account.

Most often when you are creating a new Docker image it will be based on one of the standard Docker base images like ubuntu, alpine, python or nginx. But sometimes you might want to truly roll your own image. Starting with literally nothing. From scratch. Tabula rasa.

Is this a “good” model? How to validate a model and determine whether it’s a good representation of the training data and likely to produce robust and reliable predictions.

In general a parsimonious model is a good model. A model with too many parameters is likely to overfit the data. How do we determine when a model is “complex enough” but not “too complex”?

The models we have been looking at do not differentiate between positive and negative residuals: both errors are treated the same. However, this does not align with reality, where the volatility resulting from a large negative return is higher than that for the corresponding positive return.

In the previous post we assumed that returns had a normal distribution. This assumption implied that the distribution was symmetric and a positive return was as likely as the corresponding negative return. In reality this assumption is just not true and returns are asymmetrically distributed.

A GARCH (Generalised Autoregressive Conditional Heteroskedasticity) model is a statistical tool used to forecast volatility by analysing patterns in past price movements and volatility.

In the previous post we loaded stock data into R and then calculated return volatility, both for the entire time series and shorter intervals. We saw that volatility is not constant but can change appreciably with time. One way to get a clear view of changes in volatility is by calculating them using a moving or (“rolling”) window.

I’m going to be writing a series of posts which will look at some applications of R (and perhaps Python) to financial modelling. We’ll start here by pulling some stock data into R, calculating the daily returns and then looking at correlations and simple volatility estimates.

The current version of PyInstaller (6.5.0) doesn’t play nicely with the boto3 package. Here’s how to fix it.

I’ve hit my head against this issue from time to time, so it seems like I need to document the solution somewhere for each reference.

A collection of books read by my father, Frank Collier, for Tape Aids for the Blind. Dad was always an enthusiastic and patient reader. One of my earliest memories is of him reading to my sister and me in bed each morning. In retirement he devoted many hours to reading and editing books for Tape Aids for the Blind.

In the previous post Traefik was compared to NGINX. Now let’s take a look at a few simple Traefik setups. We’ll focus on specifying the host and port.

I’ve come across Traefik in a number of questions on Stack Overflow recently. I regularly use NGINX as a reverse proxy and sometimes find it to be a little obscure. Having an alternative would be helpful.

I have seen a few questions on Stack Overflow relating to building a simple standalone Next.js app in a Docker image. Here’s one way to do it.

There are many tools for generating CSS selectors and XPath expressions. However, short of using them in your code, how can you quick test them? In this post I’ll show how you can use your browser’s Developer Tools to establish that your CSS or XPath is doing what you intend.

The parse() function from the html-react-parser package converts HTML strings into React elements. It allows you to take HTML and render it as if it were JSX. This can be particularly useful when you’re working with content that comes as HTML from external sources (such as a CMS) and you want to include that content in your React components. It can also be used to filter and modify the React elements.

In a previous post we looked at how to import content from Medium. Another potential source of content is an RSS feed. In this post we’ll see how RSS content can be imported into a Gatsby side.

In a previous post we looked at how to use AsciiDoc Markdown to author content for a Gatsby site. Another approach to handling Markdown content is MDX, which is “Markdown for the component era”. In this post we’ll see how to integrate MDX into a Gatsby site.

Suppose you want to redirect paths beginning with @ to a specific user page. For example, the @datawookie path would take you to the user page for handle datawookie.

There are probably a few ways to do this, but one approach would be to use dynamic routing.

🚀 TL;DR Show me the code. Look at the 27-dynamic-users branch. This site is deployed here.

First let’s set up the user page at src/pages/user.jsx.

Is my code secure? This is something that we should all be thinking (if not worrying) about. A thorough security audit would be the ideal, but what if you don’t have the skills or resources for that? Well, there are some tools that will at least get you part way there.

You might need to run a Selenium crawler in a GitLab CI pipeline. Here’s how to get that set up.

In a previous post we used WordPress as a CMS for a Gatsby site. We can do something similar with Medium.

Gatsby and Tailwind are a formidable combination for putting together a robust and attractive site. Throw Docker into the mix and you also have robust and reliable deployments. Here’s how to set that up for a minimal site.

Want to use Tailwind CSS with your Next.js site? Here’s how to get that set up. Also how to wrap the whole project in a Docker image.

Class name mangling (or hashing) is becoming increasingly prevalent. There’s no need to let it slow you down though. This is how you can deal with it.

In the interests of full disclosure, I know very little (very little indeed!) about .NET. But I do enjoy figuring things out. In this post I’ve documented what I learned when trying to connect a simple .NET application to MySQL using Docker Compose.

We’re going to try to do this using Docker as far as possible, which will allow me to avoid having to set up .NET on my local machine.

Not everybody is comfortable crafting web pages directly in JavaScript, HTML or even Markdown. Often content writers are more productive in an environment like WordPress. What if you want to develop your site using Gatsby but allow content writers to still craft their content in WordPress? No problem! You can use WordPress simply as a Content Management System (CMS), then pull the content through into your Gatsby site.

In this post we’ll look at how to set up a Headless WordPress CMS as a source of content for Gatsby.

Populating the <head> tag might not be the most scintillating component of building a web site, but it’s an important one to get right. Gatsby’s Head API provides a flexible mechanism for doing this and supersedes the functionality from React Helmet.

It’d be cool to be able to voice chat to other players on the Minecraft server. There are a few ways to implement this, one of which involves setting up and connecting a Discord server.

The original Java Edition of the Minecraft Server that we installed previously implements all of the basic server functionality required for multiplayer Minecraft. But perhaps this is not enough. What if you want to customise the server by installing plugins? In that case you need to install a more sophisticated server forked off the original. The PaperMC Minecraft Server provides a lot of bells and whistles not present in the original.

A quick review of the year.

• I published 55 posts (including this one).
• I spent a lot of time working with GatsbyJS for one of my clients. At first I was quite out of my depth, but I slowly figured out more or less how it works. I documented some of my learning in a series of posts.
• My most popular post is still about Shared Memory & Docker. The runner up looks at how to Install GitLab Runner with Docker.
• I spent some time compiling data on kayak specifications in the hope of producing a definitive table. It’s a work in progress but it’s already getting quite a lot of interest.

Now onto a few interesting articles from this week, mostly announcements of new versions.

• Firefox 121.0
• OpenSSH 9.6
• Julia 1.10
• Keras 3.0.2 and
• Love Songs.

When I containerised Selenium crawlers in the past I normally used a remote driver connection from the crawler to Selenium, running a separate Docker image with Selenium and accessing it via port 4444. This has proven to be a robust design. However, it does mean two containers rather than just one, leading to a higher maintenance burden and elevated resource requirements.

What about simply embedding Chrome and ChromeDriver directly into the crawler image? It requires a bit more work, but it’s worth it. The critical point is ensuring compatible versions of Chrome and ChromeDriver.