I’ve hit my head against this issue from time to time, so it seems like I need to document the solution somewhere for each reference.
The Problem
I have a requirements.txt
file which lists one or more packages that should be installed from GitHub repositories. The entries in requirements.txt
all start with git+https://github.com/
.
When I do pip3 install -r requirements.txt
I get prompted for username and password. If I need to just provide these once then it’s not a big deal to give them manually. But if this needs to be done routinely then we need to automate.
The Solution
Run this:
git config --global \
url."https://${GITHUB_TOKEN}@github.com/".insteadOf https://github.com/
🚨 This seems to require --global
. I tried with --local
and it didn’t work. Presumably --system
would also work but doesn’t seem to be practical.
This tells Git that whever it encounters a repository specified by an HTTPS URL it should insert a GitHub token into the URL to provide credentials.
For this to work you’ll need to:
- create a GitHub token with permission to read the repositories; and
- export the token in the
GITHUB_TOKEN
environment variable.
export GITHUB_TOKEN=ghp_jF76DdIvyXeYoz06gk7Z3WdpqnsWPe99Kvxk
With that in place you should be able to pip3 install -r requirements.txt
from GitHub unattended.
Docker
This also works nicely in a Docker image.
FROM python:3.10.6
COPY ./requirements.txt ./
ARG GITHUB_TOKEN
RUN git config --global \
url."https://${GITHUB_TOKEN}@github.com/".insteadOf https://github.com/
RUN pip install -r requirements.txt
Now build the image.
docker build --build-arg GITHUB_TOKEN=${GITHUB_TOKEN} .
GitHub Actions
If you want to do this in GitHub Actions then you’ll need to store the token as a secret. For the purpose of the code below the secret is called TOKEN
. You can then integrate it into your workflow like this:
jobs:
build-ubuntu:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: |
git config --global url."https://${{ secrets.TOKEN }}@github.com/".insteadOf "https://github.com/"
python -m pip install --upgrade pip
pip install -r requirements.txt
That should work with ubuntu-latest
, windows-latest
and macos-latest
.
🚨 If this doesn’t work on a Windows worker then you can try replacing ${{ secrets.TOKEN }}
with git:${{ secrets.TOKEN }}
. The specified username (in this case git
) is completely arbitrary.