Blog Posts by Andrew B. Collier / @datawookie

I’m not a web developer. However, I do regularly use HTML and CSS to layout pages which are then transformed into PDF documents. One of the requirements that I encounter fairly often is vertically aligning images and text. Fairly often, but not often enough to remember the solution. So I inevitably have to rediscover the solution (thanks StackOverflow!). Jotting this down for posterity.

In the previous post we looked at how to apply a linter and styler to a Python Project. Now we’re going to do the same for an R project. We’ll use the {precommit} R package to make the setup a breeze.

A linter and a styler can help you to write cleaner and more consistent code. In this post we’ll look at how to set up both for a Python project.

While trolling the internet aimlessly this morning, this TikTok video caught my attention.

In a previous post I took a look at some granular weather data that I acquired via the Weather API. One interesting application of these data is calculating the Fire Danger Index (FDI), which measures the degree of fire danger using information on dryness, wind speed, temperature and humidity.

A quick note on how to use the Shiny Server Docker image, rocker/shiny.

I’m a big believer in starting with the simplest possible setup, getting that to work and then adding complexity in layers. We’ll start with a simple Shiny application in app.R.

I’ve just added some more data to the {blimey} package:

• postal codes and
• postbox locations.

I’ve just added data on schools in England to the {blimey} package. The raw data were obtained from gov.uk.

Getting R set up on Linux can be somewhat frustrating. Many of the fundamental packages (like {devtools} or {remotes}) have implicit system dependencies. So installing these packages can involve numerous iterations back and forth between R and the shell while you figure out what those dependencies are and get them all installed.

I’ve been through this process many times now and finally just created a quick script that will get most of it done quickly and easily.

I’m building a model which requires historical weather data from a selection of locations in South Africa. In this post I demonstrate the process of acquiring the data and doing some simple processing.

I have a challenge: extracting data from an enormous JSON file. The structure of the file is not ideal: it’s a mapping at the top level, which means that for most standard approaches the entire document needs to be loaded before it can be processed. It would have been so much easier if the top level structure was an array. But, alas. It’s almost as if the purveyors of the data have made it intentionally inaccessible.

I occasionally write scripts where I need to persist some information between runs. These scripts are often wrapped in a Docker image and deployed on Amazon ECS. This means that there is no persistent storage. I could use a database, but this would be overkill for the volume of data involved. This post describes a simple approach to storing these data on S3 using a pickle file.

Introducing the nascent R package {blimey} (repository). At this stage it contains only the following data:

• railways — latitude and longitude segments along railway lines (wide format);
• railways_pivot — latitude and longitude segments along railway lines (long format); and
• railway_stations — codes, names and locations of railway stations.

In a previous post I documented my local setup for accessing the Interactive Brokers API via their Gateway application. I’m now in a position where I need to deploy my code onto a VM. My local setup will no longer suffice.

A brief tutorial on setting up the Interactive Brokers Gateway on Linux.

In the previous post I ran through the process of setting up a Mailfence account for encrypting emails using asymmetric encryption. In this post I show how Mailfence can be used with the {emayili} package for sending encrypted email from R.

If you’re thinking about using encrypted email, then Mailfence appears to be a pretty good option for getting started in a relatively painless way.

In a previous post I described a recipe for setting up GitLab Runner using a Docker container. With that setup it was possible to register multiple runners on a single container. However, each runner needed to be registered manually. This setup makes complete sense if the container will be around for a while. But what if you’re spinning up a GitLab Runner container for only a short duration? In this case it might be preferable to have the container pre-configured (or at least easily configured) to provide a runner to a specific project or group. Setting that up is the goal of this post.

Materialised views are a great alternative to views if the underlying query takes a long time to run. However, the principle problem with materialised views is that their content gets stale… and if the database is active, then it gets out of date rather quickly. Sure you can manually refresh a materialised view, but who has the discipline or time to do that? Better to automate the process. Then you can safely forget about it, secure in the knowledge that the data in the materialised view will remain current.

I’ve just started collaborating on a new project, Votela, with Luke. We’re going to be using Firestore for stashing our data. I’ve never worked with Firestore before, so one of my first tasks was just figuring out how to get connected and how to shift some data to and from the database.