Flexible Environment Variables for a Docker Image

I’ve been following an excellent tutorialfor deploying a Docker image on an EC2 instance via GitLab CI/CD. It covers every step in the process in great detail. If you follow the steps then you’ll definitely end up with a working pipeline.

However, I still wasn’t quite sure how to handle the environment variables and credentials that I wanted to bake into the image, and which varied between my local development environment and the final deployed image.

This is how I got it to work with my existing GitLab Runner setup. I’m sure that there are more elegant ways to do this, but this is what worked for me.

Environment Variables

I have a collection of environment variables which specify the attributes of the required database connection. I want to bake these into the Docker image. However, there are some constraints:

1. I don’t want to include .env in the Git repository. Because this is just a bad idea.
2. The environment variables have different values in development and production.
3. I don’t want a .env file lying around on the production server.

An acceptable solution would need to satisfy all of these requirements.

Local: A .env File

In my development environment the variables are specified in a .env file which looks like this:

DB_HOST=
DB_PORT=
DB_USER=
DB_PASSWORD=
DB_DATABASE=
DB_SCHEMA=

Server: GitLab CI/CD Variables

Since I wasn’t going to store these variables in the repository I needed some way to convey them to the deployed image. No problem: use GitLab CI/CD variables. There are two approaches to this:

1. create variables in .gitlab-ci.yml or
2. create variables in the GitLab UI.

I rejected the first of these for the reasons outlined above (don’t want to store variables in the repository itself).

Then I created the following variables via the GitLab CI/CD UI:

• ENV_FILE — (Type: File) The definition of the required environment variables using the same format as the .env file.
  💡 This variable holds the contents of a file. It mocks the .env file that I have in my development environment.
• SERVER_IP — (Type: Variable) The IP address for the server on which the Docker image is to be deployed.
• SERVER_USER — (Type: Variable) The user on the server which will be used for the deployment.
• SSH_PRIVATE_KEY — (Type: File) The SSH private key for the previously mentioned user.

🚨 If any of these variables are protected then the branch you are building and deploying from also needs to be protected.

This is what the definition of ENV_FILE looks like (without the sensitive bits!):

Dockerfile

The Dockerfile needs to pick up the variables and stash them in the resulting image. And it needs to handle both of the scenarios above.

FROM python:3.8.6-slim

# Rest of setup goes here...

COPY .env .env

This is actually rather simple: it just copies the local .env file onto the image. For the development environment this is easy (because there actually is a .env file!). We need to do just a little more work in the CI/CD environment.

GitLab CI/CD Setup

In GitLab CI/CD the build process is specified by the .gitlab-ci.yml file. This is what a very stripped down .gitlab-ci.yml might look like:

variables:
  TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
  TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA

build:
  image: docker:stable
  services:
    - docker:dind
  script:
    # Where is the environment file?
    - echo $ENV_FILE
    # File is outside of build context, so make local copy.
    - cp $ENV_FILE .env
    - docker build -t $TAG_COMMIT -t $TAG_LATEST .
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - docker push $TAG_COMMIT
    - docker push $TAG_LATEST

Let’s rummage through some of the details.

Variables

Create TAG_LATEST and TAG_COMMIT global variables which will be used to tag the resulting Docker image.

These tags are rather long and involved, but they will allow you to precisely locate the images in the GitLab image registry.

Image & Services

The build job will run in a docker image using the dind (Docker-in-Docker) service. To be clear, this means that Docker will be launching a container running Docker, hence “Docker in Docker”.

🚨 The GitLab runner will need to be configured to allow for Docker-in-Docker.

Build Script

Take a look at the contents of the ENV_FILE variable.

echo $ENV_FILE

It’s an absolute path pointing to a file containing the environment variables. This file is created dynamically by GitLab CI/CD using the value we assigned to the ENV_FILE variable above.

The location of this file is not in the working folder, so we need to make a local copy in order for it to be included in the Docker build context.

cp $ENV_FILE .env

Once this is done, the rest is really just mechanics:

• build the image, assigning both of the tags
• login to the GitLab image registry and
• push the tagged images.

Done!

The resulting image can then be pulled from the GitLab image registry and run in the production environment, where it will use the environment variables specified in the GitLab CI/CD UI.