Blog Posts by Andrew B. Collier / @datawookie

Using Shiny Server in Docker

A quick note on how to use the Shiny Server Docker image, rocker/shiny.

I’m a big believer in starting with the simplest possible setup, getting that to work and then adding complexity in layers. We’ll start with a simple Shiny application in app.R.

Read More →

Linux Packages for R

Getting R set up on Linux can be somewhat frustrating. Many of the fundamental packages (like {devtools} or {remotes}) have implicit system dependencies. So installing these packages can involve numerous iterations back and forth between R and the shell while you figure out what those dependencies are and get them all installed.

I’ve been through this process many times now and finally just created a quick script that will get most of it done quickly and easily.

Read More →

Historical Weather Data

I’m building a model which requires historical weather data from a selection of locations in South Africa. In this post I demonstrate the process of acquiring the data and doing some simple processing.

Read More →

Unravelling Transparency in Coverage Data

I have a challenge: extracting data from an enormous JSON file. The structure of the file is not ideal: it’s a mapping at the top level, which means that for most standard approaches the entire document needs to be loaded before it can be processed. It would have been so much easier if the top level structure was an array. But, alas. It’s almost as if the purveyors of the data have made it intentionally inaccessible.

Read More →

Persisting Data with Pickle & S3

I occasionally write scripts where I need to persist some information between runs. These scripts are often wrapped in a Docker image and deployed on Amazon ECS. This means that there is no persistent storage. I could use a database, but this would be overkill for the volume of data involved. This post describes a simple approach to storing these data on S3 using a pickle file.

Read More →

Great Britain Railway Network

Introducing the nascent R package {blimey} (repository). At this stage it contains only the following data:

  • railways — latitude and longitude segments along railway lines (wide format);
  • railways_pivot — latitude and longitude segments along railway lines (long format); and
  • railway_stations — codes, names and locations of railway stations.
Read More →

Pre-Registered GitLab Runner in a Container

In a previous post I described a recipe for setting up GitLab Runner using a Docker container. With that setup it was possible to register multiple runners on a single container. However, each runner needed to be registered manually. This setup makes complete sense if the container will be around for a while. But what if you’re spinning up a GitLab Runner container for only a short duration? In this case it might be preferable to have the container pre-configured (or at least easily configured) to provide a runner to a specific project or group. Setting that up is the goal of this post.

Read More →

Scheduling Refresh of a Materialised View

Materialised views are a great alternative to views if the underlying query takes a long time to run. However, the principle problem with materialised views is that their content gets stale… and if the database is active, then it gets out of date rather quickly. Sure you can manually refresh a materialised view, but who has the discipline or time to do that? Better to automate the process. Then you can safely forget about it, secure in the knowledge that the data in the materialised view will remain current.

Read More →

Firing Up Firestore

I’ve just started collaborating on a new project, Votela, with Luke. We’re going to be using Firestore for stashing our data. I’ve never worked with Firestore before, so one of my first tasks was just figuring out how to get connected and how to shift some data to and from the database.

Read More →

Making Sense of Drug Prices

Drug pricing is complicated. In this post I take a look at reconciling ASP and WAC prices, focusing on normalising the WAC price per billing unit to achieve a price which is comparable to ASP. This post includes a number of case studies with the objective of laying out and testing a methodology for understanding and dealing with these data. There’s a fair amount of repetition, but I wanted to test the approach across a number of drugs.

Read More →

{emayili} Updated Gmail Authentication

A recent announcement from Google stated that from 30 May 2022 they will no longer support login via username and password (this is the “less secure” option). The change will have an impact for people using the {emayili} package to send email from R, but will also affect many others who use this form of authentication to access their emails via desktop email clients. In this short post I detail how to work around this by using an application password.

Read More →

Creating Git Commits in CI

I use Continuous Integration (CI) extensively across almost all of my remote Git repositories. These are the typical jobs which it’s used for:

  • running tests
  • building documentation and
  • acquiring data.

This post addresses the last item, acquiring data.

Read More →

Adding Timestamp Columns

If you have database tables in which you are frequently adding or updating data, then it can be useful to have columns which indicates precisely when a specific record was created and updated.

Read More →

Pushing Docker Images to AWS ECR

I’ve been using the image registry on GitLab for quite a while now and loved the convenience of having my images living in the same place as my code. However, recently GitLab introduced a soft limit on transfers and that’s cramping my style. I’m moving a lot of my images onto Amazon Elastic Container Registry (ECR). In this post I look at how to get this set up.

Read More →