Python | Poetry | Dockerfile and private repositories

When we have an ecosystem like python — poetry — dockerfile and private repositories, there are few areas that will cause issues. Here are my notes while solving such issues.

Private repo credential management

Scenario

• We have a common python library (let’s call it lib-1 )in a private git repository
• We have another repository where we like to add add lib-1 as a dependency.
• In local, getting this working is simple: add the library to poetry like this: `poetry add git+https://gitlab.com/orgname/lib-1.git#v1.0.2`
• But getting this working in continuous integration environment or within a docker container is challenging.

In this article, we will solve this problem.

Approach

• create .git-credentials dynamically within Dockerfile
• add credentials from the `ARGS` passed to Dockerfile. In context of continuous integration, this will help us. While we build in local, we can pass the credentials manually.
• In the same `RUN` command, we do pip install our private repository
• In the same RUN command, we delete the .git-credentials file. This ensures that there is no security loop hole. In contrast, if we set the credentials as environment variable within the container, anyone who has access to image can know the private repository credentials.

Other approaches

I tried the following approaches but either they are not working or I am not happy with using them in an enterprise production environment.

• Pass credentials to poetry using environment variable
• auth.toml based credentials

Pass credentials to poetry using environment variable

Poetry document on `Install dependencies from a private repository` mentions about this approach but they confuse a lot without proper examples. This approach didn’t work for me. There are many tickets raised in this area.

auth.toml based credentials

auth.toml is a file where we can declare credentials. Here is a good documentation on this — How to use Poetry with private repos in Docker (without secret exposure).

I liked passing secrets to docker in a secure way with new `buildkit` based docker build --secret flag. However getting auth.toml work with poetry was difficult. There is no logs or error message as well.

Let’s say we have a repository foo . We need to add this

Install local package

This will be super helpful if we work on a common library and another repository where we use it (depend on the common library). This setup helps to validate full flow, without pushing and pull from upstreams.

# if it is already installed
$ poetry remove my-package

In `pyproject.toml`, update manually like :

[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = true }

Run following to update the lock and install from local directory

$ poetry lock && poetry install