Python Packages from GitHub

I’ve hit my head against this issue from time to time, so it seems like I need to document the solution somewhere for each reference.

The Problem

I have a requirements.txt file which lists one or more packages that should be installed from GitHub repositories. The entries in requirements.txt all start with git+https://github.com/.

When I do pip3 install -r requirements.txt I get prompted for username and password. If I need to just provide these once then it’s not a big deal to give them manually. But if this needs to be done routinely then we need to automate.

The Solution

Run this:

git config --global \
    url."https://${GITHUB_TOKEN}@github.com/".insteadOf https://github.com/

🚨 This seems to require --global. I tried with --local and it didn’t work. Presumably --system would also work but doesn’t seem to be practical.

This tells Git that when it encounters a repository specified by an HTTPS URL it should insert a GitHub token into the URL to provide credentials.

For this to work you’ll need to:

1. create a GitHub token with permission to read the repositories; and
2. export the token in the GITHUB_TOKEN environment variable.

export GITHUB_TOKEN=ghp_jF76DdIvyXeYoz06gk7Z3WdpqnsWPe99Kvxk

With that in place you should be able to pip3 install -r requirements.txt from GitHub unattended.

Docker

This also works nicely in a Docker image.

FROM python:3.10.6

COPY ./requirements.txt ./

ARG GITHUB_TOKEN
RUN git config --global \
    url."https://${GITHUB_TOKEN}@github.com/".insteadOf https://github.com/

RUN pip install -r requirements.txt

Now build the image.

docker build --build-arg GITHUB_TOKEN=${GITHUB_TOKEN} .

GitHub Actions

If you want to do this in GitHub Actions then you’ll need to store the token as a secret. For the purpose of the code below the secret is called TOKEN. You can then integrate it into your workflow like this:

jobs:
  build-ubuntu:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          git config --global url."https://${{ secrets.TOKEN }}@github.com/".insteadOf "https://github.com/"
          python -m pip install --upgrade pip
          pip install -r requirements.txt

That should work with ubuntu-latest, windows-latest and macos-latest.

🚨 If this doesn’t work on a Windows worker then you can try replacing ${{ secrets.TOKEN }} with git:${{ secrets.TOKEN }}. The specified username (in this case git) is completely arbitrary.