Defining Python dependencies for both Poetry and Pip

There's more than one way to define optional extra dependencies using Poetry, but one way is also compatible with Pip installation.

I use Poetry to manage Python dependencies and packaging, via a pyproject.toml file, for example:

[tool.poetry]
name = "mccc"
version = "0.0.1"
description = "Monte Carlo criticality code"
authors = [ "msleigh", "msleigh@users.noreply.github.com" ]
license = "Apache 2.0"
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
numpy = "^1.21.6"
matplotlib = "^3.5.3"
pandas = "^2.1.3"
click = "^8.1.7"

[tool.poetry.scripts]
mccc = "mccc.monte_carlo:main"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[project]
classifiers = [
    "Development Status :: 1 - Planning",
    "License :: OSI Approved :: Apache Software License",
    "Natural Language :: English",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
]

[project.urls]
"Homepage" = "https://github.com/msleigh/mccc"
"Bug Tracker" = "https://github.com/msleigh/mccc/issues"
"CI" = "https://github.com/msleigh/mccc/actions"
"Changelog" = "https://github.com/msleigh/mccc/releases"

The dependencies are defined in the [tool.poetry.dependencies] block, and can be installed (along with the package itself) with:

poetry install

But despite the explicit reference to Poetry in the heading, the installation can also be done with Pip; e.g. for local installation from a cloned repository:

pip install .

So far so good, but I've excluded here the development dependencies; the dependencies that are listed are only those that are necessary to install and run. The recommended way to add development dependencies is via dependency groups. For example, you could define a tests group:

[tool.poetry.group.tests.dependencies]
pytest = "^6.0.0"

and a docs group:

[tool.poetry.group.docs.dependencies]
mkdocs-material

Two important points though are:

  1. That Poetry still installs these by default along with the main dependencies, if you don't do anything special. It's basically just a labelling mechanism. You can, however, add an optional = True flag for each group, and then installation happens only if you ask for it with (e.g.) poetry install --with docs.
  2. Pip doesn't recognise these dependencies. You can't, for example, run pip install .[docs] to include the optional dependency group docs.

To solve this latter problem, use Poetry's extras functionality instead. First, declare all the desired optional dependencies as part of the main dependency block, but with the optional keyword. For example:

[tool.poetry.dependencies]
python = "^3.9"
numpy = "^1.21.6"
matplotlib = "^3.5.3"
pandas = "^2.1.3"
click = "^8.1.7"

black = {version = "^23.12.0", optional = true}
mkdocs-material = {version = "^9.5.2", optional = true}
pre-commit = {version = "^3.6.0", optional = true}
pytest = {version = "^7.4.3", optional = true}
ruff = {version = "^0.1.8", optional = true}
towncrier = {version = "^23.11.0", optional = true}

Then, we define extra groups thus:

[tool.poetry.extras]
docs = ["mkdocs-material"]
tests = ["pytest"]
dev = ["black", "pre-commit", "ruff", "towncrier"]

This way, we can install groups of extras with Poetry in this way:

poetry install --extras "docs dev"

but also it works with Pip, so that we can do e.g.:

pip install .[tests,dev]

This means that someone (else) who doesn't use Poetry can develop a package set up in this way. It's also handy for GitHub Actions I think, where it's a bit simpler to install the dependencies using Pip than it is to go to the trouble of setting up Poetry. But on my local machine, I can carry on using Poetry.