Andrew B. Collier / @datawookie

I recently put together a short training course on Spark. One of the initial components of the course involved deploying a Spark cluster on AWS. I wanted to have Jupyter Notebook and RStudio servers available on the master node too and the easiest way to make that happen was to install Docker and then run appropriate images.

There’s already a jupyter/pyspark-notebook image which includes Spark and Jupyter. It’s a simple matter to extend the rocker/verse image (which already includes RStudio server, the Tidyverse, devtools and some publishing utilities) to include the sparklyr package.

Suppose you want to create a replica of your MySQL database. The replica should:

• start with a complete snapshot of the current (initial) state of the master database and
• be updated with any changes to the master database.

This post will outline how MySQL server replication can be done using binary logs.

I’ve worked with both ExpressVPN and NordVPN. Both are great services but, from my perspective, have one major shortcoming: they’re currently blocked by Amazon Web Services (AWS). When using either of them you are simply not able to access any of the AWS services.

The most common scenario in which I’d be using a VPN is if I’m on a restrictive network where I’m only able to access web sites. Typically just ports 80, 8080 and 443 are open. Forget about SSH (port 22), SMTP (ports 25, 465 and 587) or NTP (port 123). I want to be able to connect by SSH to my AWS servers, send mail over SMTP and synchronise my clock. The latter items are normally possible over commercial VPN providers (like ExpressVPN and NordVPN) but not being able to connect to AWS is a deal breaker.

Yesterday I tried to start RStudio and something weird happened: the window launched but it was blank and unresponsive.

There are many factors which can determine the success of a conference: the location, the venue, the catering, the speakers, the social programme, the contents of the swag bag… However, in my opinion, one of the most important components of an enjoyable conference is a collection of competent chairpersons, for they will ensure that all aspects of the sessions (the very core of a conference!) run smoothly.

I’m busy preparing a poster about Stan.jl for JuliaCon 2018. Getting set up is pretty simple, although there are some minor details that I thought I’d document.

Today I finally got around to updating my R to 3.5 (or, more specifically, 3.5.1). The complete instructions for doing the update on Ubuntu are available here. I’ve paraphrased them below.

My Top 20 highlights from eRum (2018) in Budapest.

I’ve been testing out some ideas around the Travelling Salesman Problem using TSP and ggmap.

For classification problems the positive class (which is what you’re normally trying to predict) is often sparsely represented in the data. Unless you do something to address this imbalance then your classifier is likely to be rather underwhelming.

Achieving a reasonable balance in the proportions of the target classes is seldom emphasised. Perhaps it’s not very sexy. But it can have a massive effect on a model.

Join Andrew Collier and Hanjo Odendaal for a workshop on using R for Web Scraping.

Who should attend?

This workshop is aimed at beginner and intermediate R users who want to learn more about using R for data acquisition and management, with a specific focus on web scraping.

What will you learn?

You will learn:

• data manipulation with dplyr, tidyr and purrr;
• tools for accessing the DOM;
• scraping static sites with rvest;
• scraping dynamic sites with RSelenium; and
• setting up an automated scraper in the cloud.

See programme below for further details.

It seems a little counter-intuitive, but a 5 minute lightning talk is far more difficult to prepare (and present!) than a standard 20 minute or longer talk. The principle challenge is fitting everything that you want to say into the allotted time, while still maintaining an engaging narrative.

At the recent satRday conference in Cape Town (17 March 2018) we had a number of great lightning talks. A few of the speakers gave us their tips on creating a brilliant lightning talk.

It took me a little while to figure out the correct sequence for restoring a Django backup. If you have borked your database, this is how to put it back together.

Setting up the requisites to access a SQL Server database from Ubuntu.

Installing the {rJava} package on Ubuntu is not quite as simple as most other R packages. Some quick notes on how to do it.

A quick tutorial on how to create a Linux VM on Azure.

What tools do you need to start working with Ethereum smart contracts?

The Solidity Online Compiler provides a quick way to experiment with smart contracts without installing any software on your machine. Another promising online alternative is Cosmo.

However at some stage you’ll probably want to put together a local Ethereum development environment. Here are some suggestions for how to do that on an Ubuntu machine.

Since I’m just feeling my way into this new domain, I’m not sure to what degree all of these are necessary. I do know for sure, that Truffle and testrpc are crucial.

Once you’ve installed Geth you’re ready to run your own Ethereum node.