One day your Gatsby site is building fine and the next it’s breaking with a JavaScript heap out of memory error. What’s gone wrong and how can you fix it?
Problem
My Gatsby site builds via a GitHub Actions pipeline. This is what the workflow looks like.
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Node.js
uses: actions/setup-node@v3
with:
node-version: '18.x'
cache: 'npm'
- name: Install dependencies
run: |
# Dependencies not in package.json.
npm -g install gatsby-cli
npm -g install asciidoctor
# Dependencies in package.json.
yarn install --frozen-lockfile
- name: Compile TypeScript and run Jest tests
run: yarn tsc && yarn test
- name: Gatsby Build
run: yarn build
Everything runs fine until the last step. And TBH that step runs almost to completion. Then it seems to pause for a while… before breaking with a JavaScript heap out of memory error. The stack trace makes frequent mention of V8 and hash tables. But otherwise it’s pretty impenetrable. I’d guess that during the pause something is happening that is leading to the error.
So what’s going wrong?
JavaScript uses a heap to manage memory allocation. The JavaScript heap out of memory error means that I’m trying to use more memory than is either (i) allocated for the heap or (ii) available on the machine.
There are a couple of things that are worth understanding:
- What is V8?
- What is the heap and why is it causing a problem?
V8: Giving JavaScript its Zoom!
The Gatsby site is written in Typescript. The yarn build command at the end of the pipeline is transpiling Typescript into JavaScript. There are numerous TypeScript files, so many JavaScript files are being produced. These are then bundled and minified to reduce the number and size of the files that need to be processed further. V8 does that processing. V8 is an engine that compiles JavaScript into executable instructions. V8 runs in two contexts: inside a browser (a dynamic website) or outside a browser. For the moment we are concerned with the latter because we’re building the Gatsby site not viewing it in a browser.
A Heap of Trouble
There are two common approaches to memory allocation: the stack and the heap.
The stack stores data using last-in-first-out (LIFO) order. It is typically used for quick access to temporary data. For example, when a function is called, its data is pushed onto the stack, and when the function exits, the data is cleared from the stack. The stack has a finite size and exceeding it will result in a stack overflow error.
The heap is more flexible than the stack. Memory on the heap can be allocated and freed in any order. The size of the heap adjusts dynamically at runtime according to requirements of the program. However, if the size of the heap exceeds that allocated to the program or is bigger than what’s available on the system, then the heap is exhausted, resulting in an out of memory error.
In this case the problem arises on a GitHub Actions runner, which typically has 7 GB of RAM. However, not all of that memory is available for the heap. The default heap size for Node is 2 GB on 64-bit machines. This limit is set by Node, not GitHub Actions.
So what’s probably happening here is that the Typescript is being converted into JavaScript, then V8 is attempting to compile that JavaScript and in the process exhausting its allocated heap space. Seems like the problem might be solved by allocating more heap space.
Solution
The heap space allocated to Node can be adjusted via the --max-old-space-size command-line parameter. For example, to compile script.js with 4 GB of heap space you would use
node --max-old-space-size=4096 script.js
Another option is to use the NODE_OPTIONS environment variable.
export NODE_OPTIONS="--max_old_space_size=4096"
node script.js
This is the best approach if you are not running Node directly, for example if Node is being invoked via a shell script. In our case we’re running Node via Yarn. We could pass the command-line parameter through to Node like this:
yarn build -- --max-old-space-size=4096
Or we could use an environment variable. I prefer the latter approach. So this is what the modified GitHub Actions workflow looks like:
jobs:
build-and-test:
runs-on: ubuntu-latest
env:
NODE_OPTIONS: "--max_old_space_size=4096"
steps:
- uses: actions/checkout@v3
- name: Set up Node.js
uses: actions/setup-node@v3
with:
node-version: '18.x'
cache: 'npm'
- name: Install dependencies
run: |
# Dependencies not in package.json.
npm -g install gatsby-cli
npm -g install asciidoctor
# Dependencies in package.json.
yarn install --frozen-lockfile
- name: Compile TypeScript and run Jest tests
run: yarn tsc && yarn test
- name: Gatsby Build
run: yarn build
The NODE_OPTIONS environment variable is set via the env section. Problem solved. 🚀
If you are building your site on Gatsby Cloud then you can apply this via the Site Settings.
📌 This environment variable is a little ambiguous. To be clear, the name of the environment variable is NODE_OPTIONS and the value is --max_old_space_size=8192.
Afterthoughts
I build this site locally using a Docker image. To simulate limited memory availability I used the --memory command-line parameter to docker run.
I’m also building the site using Gatsby Cloud. You can set environment variables on Gatsby Cloud and NODE_OPTIONS is one of the supported environment variables.