Introduction

What is pyproject.nix

Pyproject.nix is a collection of Nix utilities to work with Python project metadata in Nix. It mainly targets PEP-621 compliant pyproject.toml files and data formats, but also implement support for other & legacy formats such as Poetry & requirements.txt.

Pyproject.nix aims to be a swiss army knife of simple customizable utilities that works either together with the nixpkgs Python infrastructure, or our own build infrastructure.

Foreword

This documentation only helps you to get started with pyproject.nix. As it's a toolkit with many use cases not every use case can be documented fully.

First and foremost pyproject.nix is a Python metadata toolkit, and not a 2nix tool.

This documentation is centered around packaging Python applications & managing development environments. For other use cases see the reference documentation.

Concepts

pyproject.nix introduces a few high level abstract concepts. The best way to get started is to understand these concepts and how they fit together.

Project

A project attribute set is a high-level representation of a project that includes:

The parsed pyproject.toml file
Parsed dependencies
Project root directory

It can can be loaded from many different sources:

PEP-621 pyproject.toml
PEP-621 pyproject.toml with PDM extensions
Poetry pyproject.toml
requirements.txt

Validators

Validators work on dependency constraints as defined in a project and offers validation for them. This can be useful to check that a package set is compilant with the specification.

Renderers

A renderer takes a project together with a Python interpreter derivation and renders it into a form understood by various pieces of Python infrastructure.

For example: The buildPythonPackage renderer returns an attribute set that can be passed to either nixpkgs function buildPythonPackage or buildPythonApplication.

There might be information missing from what a renderer returned depending on what can be computed from the project. If any attributes are missing you can manually merge your own attribute set with what the renderer returned.

Tying it together

For a concrete example use see Use cases -> pyproject.toml.

Installation

Classic Nix

Documentation examples in pyproject.nix are using Flakes for the convenience of grouping multiple concepts into a single file.

You can just as easily import pyproject.nix without using Flakes:

let
  pkgs = import <nixpkgs> { };
  inherit (pkgs) lib;

  pyproject-nix = import (builtins.fetchGit {
    url = "https://github.com/pyproject-nix/pyproject.nix.git";
  }) {
    inherit lib;
  };

in ...

Flakes

See use-cases/pyproject.toml.

pyproject.toml

It's possible to develop PEP-621 compliant Python projects without using any Python package manager except Nix.

This example loads pyproject.toml to create an environment using python.withPackages and a consumable package using python.pkgs.buildPythonPackage.

flake.nix

{
  description = "A basic flake using pyproject.toml project metadata";

  inputs.pyproject-nix.url = "github:pyproject-nix/pyproject.nix";
  inputs.pyproject-nix.inputs.nixpkgs.follows = "nixpkgs";

  outputs =
    { nixpkgs, pyproject-nix, ... }:
    let
      # Loads pyproject.toml into a high-level project representation
      # Do you notice how this is not tied to any `system` attribute or package sets?
      # That is because `project` refers to a pure data representation.
      project = pyproject-nix.lib.project.loadPyproject {
        # Read & unmarshal pyproject.toml relative to this project root.
        # projectRoot is also used to set `src` for renderers such as buildPythonPackage.
        projectRoot = ./.;
      };

      # This example is only using x86_64-linux
      pkgs = nixpkgs.legacyPackages.x86_64-linux;

      # We are using the default nixpkgs Python3 interpreter & package set.
      #
      # This means that you are purposefully ignoring:
      # - Version bounds
      # - Dependency sources (meaning local path dependencies won't resolve to the local path)
      #
      # To use packages from local sources see "Overriding Python packages" in the nixpkgs manual:
      # https://nixos.org/manual/nixpkgs/stable/#reference
      #
      # Or use an overlay generator such as uv2nix:
      # https://github.com/pyproject-nix/uv2nix
      python = pkgs.python3;

    in
    {
      # Create a development shell containing dependencies from `pyproject.toml`
      devShells.x86_64-linux.default =
        let
          # Returns a function that can be passed to `python.withPackages`
          arg = project.renderers.withPackages { inherit python; };

          # Returns a wrapped environment (virtualenv like) with all our packages
          pythonEnv = python.withPackages arg;

        in
        # Create a devShell like normal.
        pkgs.mkShell { packages = [ pythonEnv ]; };

      # Build our package using `buildPythonPackage
      packages.x86_64-linux.default =
        let
          # Returns an attribute set that can be passed to `buildPythonPackage`.
          attrs = project.renderers.buildPythonPackage { inherit python; };
        in
        # Pass attributes to buildPythonPackage.
        # Here is a good spot to add on any missing or custom attributes.
        python.pkgs.buildPythonPackage (attrs // { env.CUSTOM_ENVVAR = "hello"; });
    };
}

pyproject.toml

[project]
name = "spam"
version = "2020.0.0"
description = "Lovely Spam! Wonderful Spam!"
readme = "README.rst"
requires-python = ">=3.8"
license = {file = "LICENSE.txt"}
keywords = ["egg", "bacon", "sausage", "tomatoes", "Lobster Thermidor"]
authors = [
  {email = "hi@pradyunsg.me"},
  {name = "Tzu-ping Chung"}
]
maintainers = [
  {name = "Brett Cannon", email = "brett@python.org"}
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python"
]

dependencies = [
  "httpx",
  "gidgethub[httpx]>4.0.0",
  "django>2.1; os_name != 'nt'",
  "django>2.0; os_name == 'nt'"
]

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]

[project.urls]
homepage = "https://example.com"
documentation = "https://readthedocs.org"
repository = "https://github.com"
changelog = "https://github.com/me/spam/blob/master/CHANGELOG.md"

[project.scripts]
spam-cli = "spam:main_cli"

[project.gui-scripts]
spam-gui = "spam:main_gui"

[project.entry-points."spam.magical"]
tomatoes = "spam:main_tomatoes"

requirements.txt

Many projects comes without proper packaging and use requirements.txt files to declare their dependencies.

This example loads requirements.txt to create an environment using python.withPackages with packages from nixpkgs.

flake.nix

{
  description = "Construct development shell from requirements.txt";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  inputs.pyproject-nix.url = "github:pyproject-nix/pyproject.nix";

  outputs =
    { nixpkgs, pyproject-nix, ... }:
    let
      # Load/parse requirements.txt
      project = pyproject-nix.lib.project.loadRequirementsTxt { projectRoot = ./.; };

      pkgs = nixpkgs.legacyPackages.x86_64-linux;
      python = pkgs.python3;

      pythonEnv =
        # Assert that versions from nixpkgs matches what's described in requirements.txt
        # In projects that are overly strict about pinning it might be best to remove this assertion entirely.
        assert project.validators.validateVersionConstraints { inherit python; } == { };
        (
          # Render requirements.txt into a Python withPackages environment
          pkgs.python3.withPackages (project.renderers.withPackages { inherit python; })
        );

    in
    {
      devShells.x86_64-linux.default = pkgs.mkShell { packages = [ pythonEnv ]; };
    };
}

Build

The pyproject.nix build infrastructure is mainly targeted at python2nix authors, and is being used in uv2nix.

Pyproject.nix can be used with nixpkgs buildPythonPackage/packageOverrides/withPackages, but also implements it's own build infrastructure that fixes many structural problems with the nixpkgs implementation.

Problems with nixpkgs Python builders

Nixpkgs Python infrastucture relies on dependency propagation. The propagation mechanism works through making dependencies available to the builder at build-time, and recording their Nix store paths in $out/nix-support/propagated-build-inputs. Setup hooks are then used to add these packages to $PYTHONPATH for discovery by the Python interpreter which adds everything from $PYTHONPATH to sys.path at startup.

Propagation causes several issues downstream.

`$PYTHONPATH` leaking into unrelated builds

Consider the following development shell using nixpkgs Python builders:

let
  pkgs = import <nixpkgs> { };
  pythonEnv = pkgs.python3.withPackages(ps: [ ps.requests ]);
in pkgs.mkShell {
  packages = [
    pkgs.remarshal
    pythonEnv
  ];
}

Any Python package, such as remarshal, will have their dependencies leaking into $PYTHONPATH, making undeclared dependencies available to the Python interpreter. Making matters even worse: Any dependency on $PYTHONPATH takes precedence over virtualenv installed dependencies!

Infinite recursions

Nix dependency graphs are required to be a DAG, but Python dependencies can be cyclic. Dependency propagation is inherently incompatible with cyclic dependencies. In nixpkgs propagation isssues are commonly worked around by patching packages in various ways.

Binary wrapping

Nixpkgs Python builders uses wrappers for Python executables in bin/, these set environment variables NIX_PYTHONPATH & friends These environment variables are picked up by the interpreter using a sitecustomize.py in the system site-packages directory.

Any Python programs executing another child Python interpreter using sys.executable will have it's modules lost on import, as sys.executable isn't pointing to the environment created by withPackages.

Extraneous rebuilds

Because buildPythonPackage uses propagation runtime dependencies of a package are required to be present at build time.

In Python builds runtime dependencies are not actually required to be present. Making runtime dependencies available at build-time results in derivation hashes changing much more frequently than they have to.

Solution presented by pyproject.nix's builders

The solution is to decouple the runtime dependency graph from the build time one, by putting runtime dependencies in passthru:

stdenv.mkDerivation {
  pname = "setuptools-scm";
  version = "8.1.0";
  src = fetchurl {
    url = "https://files.pythonhosted.org/packages/4f/a4/00a9ac1b555294710d4a68d2ce8dfdf39d72aa4d769a7395d05218d88a42/setuptools_scm-8.1.0.tar.gz";
    hash = "";
  };

  passthru = {
    dependencies = {
      packaging = [ ];
      setuptools = [ ];
    };

    optional-dependencies = {
      toml = { toml = [ ]; };
      rich = { rich = [ ]; };
    };
  };

  nativeBuildInputs = [
    pyprojectHook
  ] ++ resolveBuildSystem (
    {
      setuptools = [ ];
    }
  );
}

Resolving

Because runtime dependencies are not propagated every package needs to resolve the runtime dependencies of their build-system's.

Additionally packages can't simply be consumed, but must be aggregated into a virtual environment to be useful:

{ pyproject-nix, pkgs }:

let
  python = pkgs.python312;

  # Inject your own packages on top with overrideScope
  pythonSet = pkgs.callPackage pyproject-nix.build.packages {
    inherit python;
  };

in pythonSet.pythonPkgsHostHost.mkVirtualEnv "test-venv" {
  build = [ ];
}

Cyclic dependencies

Cyclic dependencies are supported thanks to the resolver returning a flat list of required Python packages. For performance reasons two solvers are implemented:

One that does not support cyclic dependencies A much more performant resolver used by resolveBuildSystem and has all known build-systems memoized.
One that does support cyclic dependencies Used to resolve virtual environments

It's possible to override the resolver used entirely, so even though cyclic build-system's are not supported by default, it can be done with overrides.

Less rebuilds

As the runtime dependency graph is decoupled from the build time one derivation hashes change far less frequently.

An additional benefit is improved build scheduling: Because the dependency graph is much flatter than a buildPythonPackage based one derivations can be more efficiently scheduled in parallel.

Use virtualenvs

Instead of binary wrappers & environment variables pyproject.nix's builders use standard Python virtual environments.

Usage

While the nixpkgs Python infrastructure is built mainly for manual packaging, pyproject.nix's builders are mainly targeted at lock file consumers like uv2nix.

This example shows the essence of implementing a lock file converter in pure Nix using pyproject.nix. A real world implementation is more complex. To see a lock file converter built according to pyproject.nix best practices see uv2nix.

example.nix

{
  python,
  callPackage,
  pyproject-nix,
  lib,
}:
let
  # Pyproject.nix packages quite a few, but not all build-system dependencies.
  #
  # We only package PyPI packages because lock file generators often miss this metadata, so it's required to help kickstart a Python set.
  # This set is incomplete, and much smaller in both package count and scope than nixpkgs is.
  baseSet = callPackage pyproject-nix.build.packages {
    inherit python;
  };

  # A hypothetical over-simplified "lock file" format to demonstrate what typical usage would look like.
  # This format is absent of important data such as PEP-508 markers and more.
  #
  # Also note that all sources are the same. In a real-world file these would of course all be different.
  lock =
    let
      src = {
        type = "pypi";
        url = "https://files.pythonhosted.org/packages/63/70/2bf7780ad2d390a8d301ad0b550f1581eadbd9a20f896afe06353c2a2913/requests-2.32.3.tar.gz";
        hash = "sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760";
      };
    in
    {
      package-a = {
        name = "a";
        version = "1.0.0";
        dependencies = { };
        optional-dependencies = { };
        inherit src;
        build-system = {
          flit-core = [ ];
        };
      };

      package-b = {
        name = "b";
        version = "1.0.0";
        # Depend on a with no optionals
        dependencies = {
          a = [ ];
        };
        inherit src;
        build-system = {
          flit-core = [ ];
        };
      };

      package-c = {
        name = "c";
        version = "1.0.0";
        dependencies = {
          a = [ ];
        };
        # Has an optional dependency on b when cool_feature is activated
        optional-dependencies = {
          cool_feature = {
            b = [ ];
          };
        };
        inherit src;
        build-system = {
          flit-core = [ ];
        };
      };

      package-d = {
        name = "d";
        version = "1.0.0";
        dependencies = {
          c = [ "cool_feature" ];
        };
        # A local package dependend on by it's path
        src = {
          type = "path";
          path = ./packages/d;
        };
        build-system = {
          flit-core = [ ];
        };
      };
    };

  # Create a PEP-508 marker environment for marker evaluation
  environ = pyproject-nix.lib.pep508.mkEnviron python;

  # Transform lock into a Pyproject.nix build overlay.
  # This will create packages from the lock.
  overlay =
    pyfinal: _pyprev:
    lib.mapAttrs (
      name: lockpkg:
      # If package is a local package use a project loader from pyproject-nix.lib.project
      if lockpkg.src.type == "path" then
        (
          let
            project = pyproject-nix.project.loadPyprojectDynamic {
              projectRoot = lockpkg.src.path;
            };
          in
          pyfinal.callPackage (
            # Function called with callPackage
            {
              stdenv,
              pyprojectHook,
              resolveBuildSystem,
            }:
            # Call stdenv.mkDerivation with project
            stdenv.mkDerivation (
              # Render stdenv.mkDerivation arguments from project
              pyproject-nix.build.lib.renderers.mkDerivation
                {
                  inherit project environ;
                }
                {
                  inherit pyprojectHook resolveBuildSystem;
                }
            )
          ) { }
        )
      # If a package is a remote (pypi) package there is no ready made renderers to use.
      # You need to apply your own transformations.
      else if lockpkg.src.type == "pypi" then
        pyfinal.callPackage (
          {
            stdenv,
            fetchurl,
            pyprojectHook,
            resolveBuildSystem,
          }:
          stdenv.mkDerivation {
            pname = lockpkg.name;
            inherit (lockpkg) version;
            src = fetchurl lockpkg.src;

            nativeBuildInputs =
              [
                # Add hook responsible for configuring, building & installing.
                pyprojectHook
              ]
              # Build systems needs to be resolved since we don't propagate dependencies.
              # Otherwise dependencies of our build-system will be missing.
              ++ resolveBuildSystem lockpkg.build-system;

            # Dependencies go in passthru to avoid polluting runtime package.
            passthru = {
              inherit (lockpkg) dependencies optional-dependencies;
            };
          }
        ) { }
      else
        throw "Unhandled src type: ${lockpkg.src.type}" null
    ) lock;

  # Override set
  pythonSet = baseSet.overrideScope (
    _final: _prev: {
      # Override build platform dependencies
      #
      # Use this when overriding build-systems that need to run on the build platform.
      pythonPkgsBuildHost = overlay;

      # Override target platform packages.
      #
      # Use this to override packages for the target platform.
      pythonPkgsHostHost = overlay;
    }
  );

in
# Create a virtual environment containing our dependency specification
pythonSet.pythonPkgsHostHost.mkVirtualEnv "example-venv" {
  # Depend on package
  build = [ ];
}

packages

pyproject.nix's base package set only contains the scaffolding for a Python package set, but no actual Python packages.

Creating a base package set

# Returns a scope with base packages.
pkgs.callPackage pyproject-nix.build.packages {
  python = interpreter;
}

Build-system packages

For package managers that lack the ability so solve build-system dependencies pyproject.nix maintains a base package set. This set is much smaller and more narrow in scope than nixpkgs, it's purpose is only to package build-system dependencies, which are missing from Python package manager lock files, so needs to be supplemented from elsewhere.

Overriding scope

See the nixpkgs documentation.

overriding packages

Problem description

Lock file consumers can only work with what it has, and important metadata is notably absent from all current package managers.

Take a uv lock file entry for pyzmq as an example:

[[package]]
name = "pyzmq"
version = "26.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
    { name = "cffi", marker = "implementation_name == 'pypy'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/fd/05/bed626b9f7bb2322cdbbf7b4bd8f54b1b617b0d2ab2d3547d6e39428a48e/pyzmq-26.2.0.tar.gz", hash = "sha256:070672c258581c8e4f640b5159297580a9974b026043bd4ab0470be9ed324f1f", size = 271975 }
wheels = [
    { url = "https://files.pythonhosted.org/packages/28/2f/78a766c8913ad62b28581777ac4ede50c6d9f249d39c2963e279524a1bbe/pyzmq-26.2.0-cp312-cp312-macosx_10_15_universal2.whl", hash = "sha256:ded0fc7d90fe93ae0b18059930086c51e640cdd3baebdc783a695c77f123dcd9", size = 1343105 },
    # More binary wheels removed for brevity
]

And contrast it with a minimal manually package example to build the same package:

{ stdenv, pyprojectHook, fetchurl }:
stdenv.mkDerivation {
  pname = "pyzmq";
  version = "26.2.0";

  src = fetchurl {
    url = "https://files.pythonhosted.org/packages/fd/05/bed626b9f7bb2322cdbbf7b4bd8f54b1b617b0d2ab2d3547d6e39428a48e/pyzmq-26.2.0.tar.gz";
    hash = "sha256:070672c258581c8e4f640b5159297580a9974b026043bd4ab0470be9ed324f1f";
  };

  dontUseCmakeConfigure = true;

  buildInputs = [ zeromq ];

  nativeBuildInputs = [ pyprojectHook ] ++ resolveBuildSystem ({
    cmake = [];
    ninja = [];
    packaging = [];
    pathspec = [];
    scikit-build-core = [];
  } // if python.isPyPy then { cffi = []; } else { cython = []; });

  passthru.dependencies = lib.optionalAttrs python.isPyPy { cffi = []; };
}

Notably absent from uv.lock are:

Native libraries

When building binary wheels pyproject.nix uses https://nixos.org/manual/nixpkgs/stable/#setup-hook-autopatchelfhook. This patches RPATH's of wheels with native libraries, but those must be present at build time.

PEP-517 build systems

Uv, like most Python package managers, installs binary wheels by default, and it's solver doesn't take into account bootstrapping dependencies. When building from an sdist instead of a wheel build systems will need to be added.

Fixups

Basic usage

Wheels

When overriding a binary wheel, only runtime dependencies needs to be added. The build-system.requires section isn't relevant.

{ pkgs, pyproject-nix }:
let
  pythonSet = pkgs.callPackage pyproject-nix.build.packages {
    inherit python;
  };

  pyprojectOverrides = final: prev: {
    pyzmq = prev.pyzmq.overrideAttrs(old: {
      buildInputs = (old.buildInputs or [ ]) ++ [ pkgs.zeromq ];
  });

in
  pythonSet.overrideScope pyprojectOverrides

Sdist

When building from sources, both runtime dependencies and build-system.requires are important.

{ pkgs, pyproject-nix }:
let
  pythonSet = pkgs.callPackage pyproject-nix.build.packages {
    inherit python;
  };

  pyprojectOverrides = final: prev: {
    pyzmq = prev.pyzmq.overrideAttrs(old: {
      buildInputs = (old.buildInputs or [ ]) ++ [ pkgs.zeromq ];
      dontUseCmakeConfigure = true;
      nativeBuildInputs = (old.nativeBuildInputs or []) ++ final.resolveBuildSystem ({
        cmake = [];
        ninja = [];
        packaging = [];
        pathspec = [];
        scikit-build-core = [];
      } // if python.isPyPy then { cffi = []; } else { cython = []; });
    });
  };

in
  pythonSet.overrideScope pyprojectOverrides

Cross compilation

If cross compiling, build fixups might need to be applied to the build platform as well as the target platform.

When native compiling pythonPkgsBuildHost is aliased to the main set, meaning that overrides automatically apply to both. When cross compiling pythonPkgsBuildHost is a Python set created for the build host.

{ pkgs, pyproject-nix }:
let
  pythonSet = pkgs.callPackage pyproject-nix.build.packages {
    inherit python;
  };

  pyprojectOverrides = final: prev: {
    pyzmq = prev.pyzmq.overrideAttrs(old: {
      buildInputs = (old.buildInputs or [ ]) ++ [ pkgs.zeromq ];
  });

  pyprojectCrossOverrides = lib.composeExtensions (_final: prev: {
    pythonPkgsBuildHost = prev.pythonPkgsBuildHost.overrideScope overlay;
  }) overlay;


in
  pythonSet.overrideScope pyprojectCrossOverrides

When not cross compiling pythonPkgsBuildHost is aliased to the main Python set, so overrides will apply to both automatically.

Dealing with common problems

For utility functions to deal with some common packaging issues see hacks.

hacks

This documentation is a guide, for more details see hacks library reference.

Using prebuilt packages from Nixpkgs

Sometimes making a package building from source can be difficult, wheels are not available, and Nixpkgs may already contain source-built packages. In such cases it can be tempting to reuse build outputs from Nixpkgs, just as you would use a binary wheel from PyPI.

For such cases pyproject.nix provides an adapter:

{ callPackage, pyproject-nix, python3, python3Packages }:
let
  python = python3;

  hacks = callPackage pyproject-nix.build.hacks {};

  overlay = final: prev: {
    # Adapt torch from nixpkgs
    torch = hacks.nixpkgsPrebuilt {
      from = python3Packages.torchWithoutCuda;
      prev = prev.torch;
    };
  };

  pythonSet = (callPackage pyproject-nix.build.packages {
    inherit python;
  }).overrideScope overlay;
in
  pythonSet.mkVirtualenv "torch-venv" {
    torch = [ ];
  }

You may also want to filter out certain dependencies, torch in particular depends on a number of PyPI packages containing binary shared objects that are already linked by torch from nixpkgs.

hacks.nixpkgsPrebuilt {
  from = python3Packages.torchWithoutCuda;
  prev = prev.torch.overrideAttrs(old: {
    passthru = old.passthru // {
      dependencies = lib.filterAttrs (name: _: ! lib.hasPrefix "nvidia" name) old.passthru.dependencies;
    };
  });
};

Building Cargo (Rust) packages from source

Rust has it's own package manager, Cargo, that expects to be able to download dependencies at build-time. One way to deal with that is to use rustPlatform.importCargoLock.

This mechanism uses IFD (import-from-derivation) on non-local packages. For background as to why IFD should be avoided see

fzakaria - Nix parallelism & Import From Derivation.

To adapt the cryptography Python package into creating a Rust vendor directory, and use it for building:

final: prev: {
  cryptography =
    (hacks.importCargoLock {
      prev = prev.cryptography;
      # Cryptography uses a non-standard location for it's Rust packaging
      cargoRoot = "src/rust";
    });
}

In reality, the package still lacks some important metadata, such as native non-Rust dependencies that needs to be supplemented. Depending on which lock file produced this package it may also need build-systems added.

final: prev: {
  cryptography =
    (hacks.importCargoLock {
      prev = prev.cryptography;
      # Cryptography uses a non-standard location for it's Rust packaging
      cargoRoot = "src/rust";
    }).overrideAttrs
      (old: {
        nativeBuildInputs =
          old.nativeBuildInputs
          ++ final.resolveBuildSystem {
            maturin = [ ];
            setuptools = [ ];
            cffi = [ ];
            pycparser = [ ];
          };
        buildInputs = old.buildInputs or [ ] ++ [ pkgs.openssl ];
      });
}

ecosystem

Projects currently using the pyproject.nix builders

Python2nix

Lock file consumers producing package sets

uv2nix

Build-system package sets

build-system-pkgs

Override collections

Override collections to deal with packaging issues

uv2nix_hammer_overrides

FAQ

How does package name mapping from Python to Nixpkgs work?

Package names are normalized according to the PyPA normalization specification. Nixpkgs also uses the same normalization but has some legacy package names that do not follow normalization guidelines.

The other case where the automatic mapping goes wrong is when the Nixpkgs python.pkgs set does not contain a dependency. One such example is ruff, a Python linter written in Rust.

Nixpkgs has ruff on the top-level (pkgs), but not in python3.pkgs. In such cases you can use an overlay to add the package to the Python set:

let
  python = pkgs.python3.override {
    packageOverrides = self: super: {
      ruff = pkgs.ruff;
    };
  };
in ...

How do you treat `dynamic` attributes?

Pyproject.nix makes no attempt at parsing dynamic fields as it does not have the required knowledge to infer these.

When using the withPackages renderer most fields that may be dynamic are not even relevant and won't cause issues. At other times, like when using the buildPythonPackage renderer problems occur as there is no way for the renderer to create the version attribute.

let
  project = pyproject.project.loadPyproject { pyproject = lib.importTOML ./pyproject.toml; };
  python = pkgs.python3;
  attrs = pyproject.renderers.buildPythonPackage { inherit python project; };
in python.pkgs.buildPythonPackage attrs

Will result in an error from buildPythonpackage because version is missing:

error: attribute 'version' missing

at /nix/store/gna8i238i3nnz6cizcayyfyfdzn28la5-nixpkgs/pkgs/development/interpreters/python/mk-python-derivation.nix:31:28:

    30|
    31| { name ? "${attrs.pname}-${attrs.version}"
      |                            ^
    32|

In these cases you can manually add attributes to the attribute set returned by the renderer:

let
  project = pyproject.project.loadPyproject { pyproject = lib.importTOML ./pyproject.toml; };
  python = pkgs.python3;
  attrs = pyproject.renderers.buildPythonPackage { inherit python project; };
in python.pkgs.buildPythonPackage (attrs // {
  version = "1.0";  # Not dynamically inferred
})

Reference documentation

The reference documentation is split up into two main categories:

User facing APIs

Contains high-level representations and has notions of things like a project (a fully parsed pyproject.toml) and further operations done on the project level.

Standards APIs

Contains parsers, evaluators & utility functions for dealing with Python packaging standards defined by the through the PEP process & from PyPA.

project

`lib.project.loadPyproject`

Type: loadPyproject :: AttrSet -> AttrSet

Load dependencies from a PEP-621 pyproject.toml.

structured function argument

: pyproject

: The unmarshaled contents of pyproject.toml

extrasAttrPaths

: Example: extrasAttrPaths = [ "tool.pdm.dev-dependencies" ];

extrasListPaths

: Example: extrasListPaths = { "tool.uv.dependencies.dev-dependencies" = "dev-dependencies"; }

groupsAttrPaths

: Example: extrasAttrPaths = [ "tool.pdm.dev-dependencies" ];

groupsListPaths

: Example: extrasListPaths = { "tool.uv.dependencies.dev-dependencies" = "dev-dependencies"; }

projectRoot

: Path to project root

::: {.example #function-library-example-lib.project.loadPyproject}

`lib.project.loadPyproject` usage example

# loadPyproject { pyproject = lib.importTOML }
{
  dependencies = { }; # Parsed dependency structure in the schema of `lib.pep621.parseDependencies`
  build-systems = [ ];  # Returned by `lib.pep518.parseBuildSystems`
  pyproject = { }; # The unmarshaled contents of pyproject.toml
  projectRoot = null; # Path to project root
  requires-python = null; # requires-python as parsed by pep621.parseRequiresPython
}

:::

`lib.project.loadUVPyproject`

Type: loadUVPyproject :: AttrSet -> AttrSet

Load dependencies from a uv pyproject.toml.

structured function argument

: pyproject

: The unmarshaled contents of pyproject.toml

projectRoot

: Path to project root

::: {.example #function-library-example-lib.project.loadUVPyproject}

`lib.project.loadUVPyproject` usage example

# loadUVPyproject { projectRoot = ./.; }
{
  dependencies = { }; # Parsed dependency structure in the schema of `lib.pep621.parseDependencies`
  build-systems = [ ];  # Returned by `lib.pep518.parseBuildSystems`
  pyproject = { }; # The unmarshaled contents of pyproject.toml
  projectRoot = null; # Path to project root
  requires-python = null; # requires-python as parsed by pep621.parseRequiresPython
}

:::

`lib.project.loadPDMPyproject`

Type: loadPDMPyproject :: AttrSet -> AttrSet

Load dependencies from a PDM pyproject.toml.

structured function argument

: pyproject

: The unmarshaled contents of pyproject.toml

projectRoot

: Path to project root

::: {.example #function-library-example-lib.project.loadPDMPyproject}

`lib.project.loadPDMPyproject` usage example

# loadPyproject { projectRoot = ./.; }
{
  dependencies = { }; # Parsed dependency structure in the schema of `lib.pep621.parseDependencies`
  build-systems = [ ];  # Returned by `lib.pep518.parseBuildSystems`
  pyproject = { }; # The unmarshaled contents of pyproject.toml
  projectRoot = null; # Path to project root
  requires-python = null; # requires-python as parsed by pep621.parseRequiresPython
}

:::

`lib.project.loadPoetryPyproject`

Type: loadPoetryPyproject :: AttrSet -> AttrSet

Load dependencies from a Poetry pyproject.toml.

structured function argument

: pyproject

: The unmarshaled contents of pyproject.toml

projectRoot

: Path to project root

::: {.example #function-library-example-lib.project.loadPoetryPyproject}

`lib.project.loadPoetryPyproject` usage example

# loadPoetryPyproject { projectRoot = ./.; }
{
  dependencies = { }; # Parsed dependency structure in the schema of `lib.pep621.parseDependencies`
  build-systems = [ ];  # Returned by `lib.pep518.parseBuildSystems`
  pyproject = { }; # The unmarshaled contents of pyproject.toml
  projectRoot = null; # Path to project root
  requires-python = null; # requires-python as parsed by pep621.parseRequiresPython
}

:::

`lib.project.loadRequirementsTxt`

Type: loadRequirementsTxt :: AttrSet -> AttrSet

Load dependencies from a requirements.txt.

Note that as requirements.txt is lacking important project metadata this is incompatible with some renderers.

structured function argument

: requirements

: The contents of requirements.txt

projectRoot

: Path to project root

::: {.example #function-library-example-lib.project.loadRequirementsTxt}

`lib.project.loadRequirementsTxt` usage example

# loadRequirementstxt { requirements = builtins.readFile ./requirements.txt; projectRoot = ./.; }
{
  dependencies = { }; # Parsed dependency structure in the schema of `lib.pep621.parseDependencies`
  build-systems = [ ];  # Returned by `lib.pep518.parseBuildSystems`
  pyproject = null; # The unmarshaled contents of pyproject.toml
  projectRoot = null; # Path to project root
  requires-python = null; # requires-python as parsed by pep621.parseRequiresPython
}

:::

`lib.project.loadPyprojectDynamic`

Type: loadPyprojectDynamic :: AttrSet -> AttrSet

Load dependencies from a either a PEP-621 or Poetry pyproject.toml file. This function is intended for 2nix authors that wants to include local pyproject.toml files but don't know up front whether they're from Poetry or PEP-621.

structured function argument

: pyproject

: The unmarshaled contents of pyproject.toml

projectRoot

: Path to project root

::: {.example #function-library-example-lib.project.loadPyprojectDynamic}

`lib.project.loadPyprojectDynamic` usage example

# loadPyprojectDynamic { projectRoot = ./.; }
{
  dependencies = { }; # Parsed dependency structure in the schema of `lib.pep621.parseDependencies`
  build-systems = [ ];  # Returned by `lib.pep518.parseBuildSystems`
  pyproject = { }; # The unmarshaled contents of pyproject.toml
  projectRoot = null; # Path to project root
  requires-python = null; # requires-python as parsed by pep621.parseRequiresPython
}

:::

scripts

`lib.scripts.loadScript`

Load a PEP-723 metadata script from file path or string.

structured function argument

: name

: Function argument

script

: Function argument

::: {.example #function-library-example-lib.scripts.loadScript}

`lib.scripts.loadScript` usage example

# loadScript { script = ./with-inline-metadata.py; }
{
  name = "with-inline-metadata";
  metadata = { ... }; # Contains dependencies and requires-python
  renderWithPackages = { python }: ...; # renderWithPackages with loaded script pre-applied
}

:::

`lib.scripts.renderWithPackages`

Render a loaded PEP-723 script as a string with a shebang line pointing to a wrapped Nix store interpreter.

structured function argument

: script

: Script loaded using loadScript

python

: Nixpkgs Python interpreter

environ

: Nixpkgs Python package set Python extras (optional-dependencies) to enable. PEP-508 environment

::: {.example #function-library-example-lib.scripts.renderWithPackages}

`lib.scripts.renderWithPackages` usage example

# Using renderWithPackages directly
let
  script = loadScript { script = ./with-inline-metadata.py; };
in pkgs.writeScript script.name (renderWithPackages { inherit script; python = pkgs.python3; })

# Using script render function
let
  script = loadScript { script = ./with-inline-metadata.py; };
in pkgs.writeScript script.name (script.render { python = pkgs.python3; })

:::

Build infrastructures

renderers

`lib.renderers.withPackages`

Type: withPackages :: AttrSet -> lambda

Renders a project as an argument that can be passed to withPackages

Evaluates PEP-508 environment markers to select correct dependencies for the platform but does not validate version constraints. For validation see lib.validators.

structured function argument

: project

: Project metadata as returned by lib.project.loadPyproject

python

: Python derivation

pythonPackages

: Nixpkgs Python package set

extras

: Python extras (optionals) to enable

groups

: PEP-735 dependency groups to enable.

extraPackages

: Extra withPackages function

environ

: PEP-508 environment

::: {.example #function-library-example-lib.renderers.withPackages}

`lib.renderers.withPackages` usage example

# withPackages (lib.project.loadPyproject { ... })
  «lambda @ «string»:1:1»

:::

`lib.renderers.buildPythonPackage`

Type: buildPythonPackage :: AttrSet -> AttrSet

Renders a project as an argument that can be passed to buildPythonPackage/buildPythonApplication.

Evaluates PEP-508 environment markers to select correct dependencies for the platform but does not validate version constraints. For validation see lib.validators.

structured function argument

: project

: Project metadata as returned by lib.project.loadPyproject

python

: Python derivation

pythonPackages

: Nixpkgs Python package set

extras

: Python extras (optional-dependencies) to enable.

groups

: PEP-735 dependency groups to enable.

extrasAttrMappings

: Map a Python extras group name to a Nix attribute set like: { dev = "checkInputs"; } This is intended to be used with optionals such as test dependencies that you might want to remap to checkInputs.

format

: Which package format to pass to buildPythonPackage If the format is "wheel" PEP-518 build-systems are excluded from the build.

environ

: PEP-508 environment

::: {.example #function-library-example-lib.renderers.buildPythonPackage}

`lib.renderers.buildPythonPackage` usage example

# buildPythonPackage { project = lib.project.loadPyproject ...; python = pkgs.python3;  }
  { pname = "blinker"; version = "1.3.3.7"; dependencies = [ ]; }

:::

`lib.renderers.mkPythonEditablePackage`

Type: mkPythonEditablePackage :: AttrSet -> AttrSet

Renders a project as an argument that can be passed to mkPythonEditablePackage.

Evaluates PEP-508 environment markers to select correct dependencies for the platform but does not validate version constraints. For validation see lib.validators.

Note for Nix Flake users: Flakes are copied to the store when using pure evaluation, meaning that the project root will point to a store directory. Either set root manually to a string using the returned attribute set, or evaluate using --impure.

::: {.example #function-library-example-lib.renderers.mkPythonEditablePackage}

`lib.renderers.mkPythonEditablePackage` usage example

# mkPythonEditablePackage { project = lib.project.loadPyproject ...; python = pkgs.python3;  }
  { pname = "blinker"; version = "1.3.3.7"; dependencies = [ ]; }

:::

`lib.renderers.meta`

Type: meta :: AttrSet -> AttrSet

Renders a project as a meta attribute

This is used internally in renderers.mkPythonPackages

structured function argument

: project

: Function argument

validators

`lib.validators.validateVersionConstraints`

Type: validateVersionConstraints :: AttrSet -> AttrSet

Validates the Python package set held by Python (python.pkgs) against the parsed project.

Returns an attribute set where the name is the Python package derivation pname and the value is a list of the mismatching conditions.

structured function argument

: project

: Project metadata as returned by lib.project.loadPyproject

python

: Python derivation

extras

: Python extras (optionals) to enable

::: {.example #function-library-example-lib.validators.validateVersionConstraints}

`lib.validators.validateVersionConstraints` usage example

# validateVersionConstraints (lib.project.loadPyproject { ... })
{
  resolvelib = {
    # conditions as returned by `lib.pep440.parseVersionCond`
    conditions = [ { op = ">="; version = { dev = null; epoch = 0; local = null; post = null; pre = null; release = [ 1 0 1 ]; }; } ];
    # Version from Python package set
    version = "0.5.5";
  };
  unearth = {
    conditions = [ { op = ">="; version = { dev = null; epoch = 0; local = null; post = null; pre = null; release = [ 0 10 0 ]; }; } ];
    version = "0.9.1";
  };
}

:::

Reference documentation

The reference documentation is split up into two main categories:

User facing APIs

Contains high-level representations and has notions of things like a project (a fully parsed pyproject.toml) and further operations done on the project level.

Standards APIs

Contains parsers, evaluators & utility functions for dealing with Python packaging standards defined by the through the PEP process & from PyPA.

pep440

`lib.pep440.parseVersion`

Type: parseVersion :: string -> AttrSet

Parse a version according to PEP-440.

version

: Function argument

::: {.example #function-library-example-lib.pep440.parseVersion}

`lib.pep440.parseVersion` usage example

# parseVersion "3.0.0rc1"
{
  dev = null;
  epoch = 0;
  local = null;
  post = null;
  pre = {
    type = "rc";
    value = 1;
  };
  release = [ 3 0 0 ];
}

:::

`lib.pep440.parseVersionCond`

Type: parseVersionCond :: string -> AttrSet

Parse a version conditional.

cond

: Function argument

::: {.example #function-library-example-lib.pep440.parseVersionCond}

`lib.pep440.parseVersionCond` usage example

# parseVersionCond ">=3.0.0rc1"
{
  op = ">=";
  version = {
    dev = null;
    epoch = 0;
    local = null;
    post = null;
    pre = {
      type = "rc";
      value = 1;
    };
    release = [ 3 0 0 ];
  };
}

:::

`lib.pep440.parseVersionConds`

Type: parseVersionConds :: string -> [AttrSet]

Parse a list of version conditionals separated by commas.

conds

: Function argument

::: {.example #function-library-example-lib.pep440.parseVersionConds}

`lib.pep440.parseVersionConds` usage example

# parseVersionConds ">=3.0.0rc1,<=4.0"
[
  {
    op = ">=";
    version = {
      dev = null;
      epoch = 0;
      local = null;
      post = null;
      pre = {
        type = "rc";
        value = 1;
      };
      release = [ 3 0 0 ];
    };
  }
  {
    op = "<=";
    version = {
      dev = null;
      epoch = 0;
      local = null;
      post = null;
      pre = null;
      release = [ 4 0 ];
    };
  }
]

:::

`lib.pep440.compareVersions`

Type: compareVersions :: AttrSet -> AttrSet -> int

Compare two versions as parsed by parseVersion according to PEP-440.

Returns:

-1 for less than
0 for equality
1 for greater than

a

: Function argument

b

: Function argument

::: {.example #function-library-example-lib.pep440.compareVersions}

`lib.pep440.compareVersions` usage example

# compareVersions (parseVersion "3.0.0") (parseVersion "3.0.0")
0

:::

`lib.pep440.comparators`

Type: operators.${operator} :: AttrSet -> AttrSet -> bool

Map comparison operators as strings to a comparator function.

Attributes:

Compatible release clause: ~=
Version matching clause: ==
Version exclusion clause: !=
Inclusive ordered comparison clause: <=, >=
Exclusive ordered comparison clause: <, >
Arbitrary equality clause: ===

::: {.example #function-library-example-lib.pep440.comparators}

`lib.pep440.comparators` usage example

# comparators."==" (parseVersion "3.0.0") (parseVersion "3.0.0")
true

:::

pep508

`lib.pep508.parseMarkers`

Type: parseMarkers :: string -> AttrSet

Parse PEP 508 markers into an AST.

::: {.example #function-library-example-lib.pep508.parseMarkers}

`lib.pep508.parseMarkers` usage example

# parseMarkers "(os_name=='a' or os_name=='b') and os_name=='c'"
{
  lhs = {
    lhs = {
      lhs = {
        type = "variable";
        value = "os_name";
      };
      op = "==";
      rhs = {
        type = "string";
        value = "a";
      };
      type = "compare";
    };
    op = "or";
    rhs = {
      lhs = {
        type = "variable";
        value = "os_name";
      };
      op = "==";
      rhs = {
        type = "string";
        value = "b";
      };
      type = "compare";
    };
    type = "boolOp";
  };
  op = "and";
  rhs = {
    lhs = {
      type = "variable";
      value = "os_name";
    };
    op = "==";
    rhs = {
      type = "string";
      value = "c";
    };
    type = "compare";
  };
  type = "boolOp";
}

:::

`lib.pep508.parseString`

Type: parseString :: string -> AttrSet

Parse a PEP-508 dependency string.

input

: Function argument

::: {.example #function-library-example-lib.pep508.parseString}

`lib.pep508.parseString` usage example

# parseString "cachecontrol[filecache]>=0.13.0"
{
  conditions = [
    {
      op = ">=";
      version = {
        dev = null;
        epoch = 0;
        local = null;
        post = null;
        pre = null;
        release = [ 0 13 0 ];
      };
    }
  ];
  markers = null;
  name = "cachecontrol";
  extras = [ "filecache" ];
  url = null;
}

:::

`lib.pep508.mkEnviron`

Type: mkEnviron :: derivation -> AttrSet

Create an attrset of platform variables. As described in https://peps.python.org/pep-0508/#environment-markers.

python

: Function argument

::: {.example #function-library-example-lib.pep508.mkEnviron}

`lib.pep508.mkEnviron` usage example

# mkEnviron pkgs.python3
{
  implementation_name = {
    type = "string";
    value = "cpython";
  };
  implementation_version = {
    type = "version";
    value = {
      dev = null;
      epoch = 0;
      local = null;
      post = null;
      pre = null;
      release = [ 3 10 12 ];
    };
  };
  os_name = {
    type = "string";
    value = "posix";
  };
  platform_machine = {
    type = "string";
    value = "x86_64";
  };
  platform_python_implementation = {
    type = "string";
    value = "CPython";
  };
  # platform_release maps to platform.release() which returns
  # the running kernel version on Linux.
  # Because this field is not reproducible it's left empty.
  platform_release = {
    type = "string";
    value = "";
  };
  platform_system = {
    type = "string";
    value = "Linux";
  };
  # platform_version maps to platform.version() which also returns
  # the running kernel version on Linux.
  # Because this field is not reproducible it's left empty.
  platform_version = {
    type = "version";
    value = {
      dev = null;
      epoch = 0;
      local = null;
      post = null;
      pre = null;
      release = [ ];
    };
  };
  python_full_version = {
    type = "version";
    value = {
      dev = null;
      epoch = 0;
      local = null;
      post = null;
      pre = null;
      release = [ 3 10 12 ];
    };
  };
  python_version = {
    type = "version";
    value = {
      dev = null;
      epoch = 0;
      local = null;
      post = null;
      pre = null;
      release = [ 3 10 ];
    };
  };
  sys_platform = {
    type = "string";
    value = "linux";
  };
}

:::

`lib.pep508.setEnviron`

Update one or more keys in an environment created by mkEnviron.

environ

: Function argument

updates

: Function argument

::: {.example #function-library-example-lib.pep508.setEnviron}

`lib.pep508.setEnviron` usage example

# setEnviron (mkEnviron pkgs.python3) { platform_release = "5.10.65";  }

:::

`lib.pep508.evalMarkers`

Type: evalMarkers :: AttrSet -> AttrSet -> bool

Evaluate an environment as returned by mkEnviron against markers as returend by parseMarkers.

environ

: Function argument

value

: Function argument

::: {.example #function-library-example-lib.pep508.evalMarkers}

`lib.pep508.evalMarkers` usage example

# evalMarkers (mkEnviron pkgs.python3) (parseMarkers "python_version < \"3.11\"")
true

:::

pep518

`lib.pep518.parseBuildSystems`

Type: readPyproject :: AttrSet -> list

Parse PEP-518 build-system.requires from pyproject.toml.

pyproject

: Function argument

::: {.example #function-library-example-lib.pep518.parseBuildSystems}

`lib.pep518.parseBuildSystems` usage example

# parseBuildSystems (lib.importTOML ./pyproject.toml)
  [ ]  # List of parsed PEP-508 strings as returned by `lib.pep508.parseString`.

:::

pep599

`lib.pep599.manyLinuxTargetMachines`

Map Nixpkgs CPU values to target machines known to be supported for manylinux* wheels (a.k.a. uname -m), in nixpkgs found under the attribute stdenv.targetPlatform.parsed.cpu.name

::: {.example #function-library-example-lib.pep599.manyLinuxTargetMachines}

`lib.pep599.manyLinuxTargetMachines` usage example

# legacyAliases.powerpc64
"ppc64"

:::

pep600

`lib.pep600.legacyAliases`

Type: legacyAliases.${tag} :: AttrSet -> string

Map legacy (pre PEP-600) platform tags to PEP-600 compliant ones.

https://peps.python.org/pep-0600/#legacy-manylinux-tags

::: {.example #function-library-example-lib.pep600.legacyAliases}

`lib.pep600.legacyAliases` usage example

# legacyAliases."manylinux1_x86_64" or "manylinux1_x86_64"
"manylinux_2_5_x86_64"

:::

`lib.pep600.manyLinuxTagCompatible`

Type: manyLinuxTagCompatible :: AttrSet -> derivation -> string -> bool

Check if a manylinux tag is compatible with a given stdenv.

platform

: Platform attrset (lib.systems.elaborate "x86_64-linux")

libc

: Libc derivation

tag

: Platform tag string

::: {.example #function-library-example-lib.pep600.manyLinuxTagCompatible}

`lib.pep600.manyLinuxTagCompatible` usage example

# manyLinuxTagCompatible pkgs.stdenv.targetPlatform pkgs.stdenv.cc.libc "manylinux_2_5_x86_64"
true

:::

pep621

`lib.pep621.parseDependencies`

Type: parseDependencies :: AttrSet -> AttrSet

Parse dependencies from pyproject.toml.

structured function argument

: pyproject

: Function argument

extrasAttrPaths

: Function argument

extrasListPaths

: Function argument

groupsAttrPaths

: Function argument

groupsListPaths

: Function argument

::: {.example #function-library-example-lib.pep621.parseDependencies}

`lib.pep621.parseDependencies` usage example

# parseDependencies {
#
#   pyproject = (lib.importTOML ./pyproject.toml);
#   # Don't just look at `project.optional-dependencies` for groups, also look at these:
#   extrasAttrPaths = [ "tool.pdm.dev-dependencies" ];
# }
{
  dependencies = [ ];  # List of parsed PEP-508 strings (lib.pep508.parseString)
  extras = {
    dev = [ ];  # List of parsed PEP-508 strings (lib.pep508.parseString)
  };
  build-systems = [ ];  # PEP-518 build-systems (List of parsed PEP-508 strings)
}

:::

`lib.pep621.parseRequiresPython`

Type: parseRequiresPython :: AttrSet -> list

Parse project.python-requires from pyproject.toml

pyproject

: Function argument

::: {.example #function-library-example-lib.pep621.parseRequiresPython}

`lib.pep621.parseRequiresPython` usage example

#  parseRequiresPython (lib.importTOML ./pyproject.toml)
[ ]  # List of conditions as returned by `lib.pep440.parseVersionCond`

:::

`lib.pep621.filterDependenciesByEnviron`

Type: filterDependenciesByEnviron :: AttrSet -> AttrSet -> AttrSet

Filter dependencies not relevant for this environment.

environ

: Environ as created by lib.pep508.mkEnviron.

extras

: Extras as a list of strings

dependencies

: Dependencies as parsed by lib.pep621.parseDependencies.

::: {.example #function-library-example-lib.pep621.filterDependenciesByEnviron}

`lib.pep621.filterDependenciesByEnviron` usage example

# filterDependenciesByEnviron (lib.pep508.mkEnviron pkgs.python3) (lib.pep621.parseDependencies (lib.importTOML ./pyproject.toml))
{ }  # Structure omitted in docs

:::

pep656

`lib.pep656.muslLinuxTagCompatible`

Type: muslLinuxTagCompatible :: AttrSet -> derivation -> string -> bool

Check if a musllinux tag is compatible with a given stdenv.

platform

: Platform attrset (lib.systems.elaborate "x86_64-linux")

libc

: Libc derivation

tag

: Platform tag string

::: {.example #function-library-example-lib.pep656.muslLinuxTagCompatible}

`lib.pep656.muslLinuxTagCompatible` usage example

# muslLinuxTagCompatible pkgs.stdenv.targetPlatform pkgs.stdenv.cc.libc "musllinux_1_1_x86_64"
true

:::

pep723

`lib.pep723.parseScript`

Type: parseScript :: string -> AttrSet

Parse the script metadata section from a PEP-723 script.

script

: Function argument

::: {.example #function-library-example-lib.pep723.parseScript}

`lib.pep723.parseScript` usage example

# parseScript (readFile ./script.py)
{
  requires-python = [ ];  # List of parsed version conditions (lib.pep440.parseVersionConds)
  dependencies = [ ];  # List of parsed PEP-508 strings (lib.pep508.parseString)
}

:::

poetry

`lib.poetry.translatePoetryProject`

Type: translatePoetryProject :: AttrSet -> lambda

Translate a Pyproject.toml from Poetry to PEP-621 project metadata. This function transposes a PEP-621 project table on top of an existing Pyproject.toml populated with data from tool.poetry. Notably does not translate dependencies/optional-dependencies.

For parsing dependencies from Poetry see lib.poetry.parseDependencies.

pyproject

: Function argument

::: {.example #function-library-example-lib.poetry.translatePoetryProject}

`lib.poetry.translatePoetryProject` usage example

# translatePoetryProject (lib.importTOML ./pyproject.toml)
{ }  # TOML contents, structure omitted. See PEP-621 for more information on data members.

:::

`lib.poetry.parseDependencies`

Type: parseDependencies :: AttrSet -> AttrSet

Parse dependencies from pyproject.toml (Poetry edition). This function is analogous to lib.pep621.parseDependencies.

pyproject

: Function argument

::: {.example #function-library-example-lib.poetry.parseDependencies}

`lib.poetry.parseDependencies` usage example

# parseDependencies {
#
#   pyproject = (lib.importTOML ./pyproject.toml);
# }
{
  dependencies = [ ];  # List of parsed PEP-508 strings (lib.pep508.parseString)
  extras = {
    dev = [ ];  # List of parsed PEP-508 strings (lib.pep508.parseString)
  };
  build-systems = [ ];  # PEP-518 build-systems (List of parsed PEP-508 strings)
}

:::

`lib.poetry.parseVersionCond`

Type: parseVersionCond :: string -> [ AttrSet ]

Parse a version conditional. Supports additional non-standard operators ^ and ~ used by Poetry.

Because some expressions desugar to multiple expressions parseVersionCond returns a list.

cond

: Function argument

`lib.poetry.parseVersionConds`

Type: parseVersionConds :: string -> [ AttrSet ]

Parse a comma separated list version conditionals. Supports additional non-standard operators ^ and ~ used by Poetry.

s

: Function argument

pypa

`lib.pypa.normalizePackageName`

Type: normalizePackageName :: string -> string

Normalize package name as documented in https://packaging.python.org/en/latest/specifications/name-normalization/#normalization

::: {.example #function-library-example-lib.pypa.normalizePackageName}

`lib.pypa.normalizePackageName` usage example

# readPyproject "Friendly-Bard"
"friendly-bard"

:::

`lib.pypa.parsePythonTag`

Type: parsePythonTag :: string -> AttrSet

Parse Python tags.

As described in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#python-tag.

tag

: Function argument

::: {.example #function-library-example-lib.pypa.parsePythonTag}

`lib.pypa.parsePythonTag` usage example

# parsePythonTag "cp37"
{
  implementation = "cpython";
  version = "37";
}

:::

`lib.pypa.parseABITag`

Type: parseABITag :: string -> AttrSet

Parse ABI tags.

As described in https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#python-tag.

tag

: Function argument

::: {.example #function-library-example-lib.pypa.parseABITag}

`lib.pypa.parseABITag` usage example

# parseABITag "cp37dmu"
{
  rest = "dmu";
  implementation = "cp";
  version = "37";
}

:::

`lib.pypa.isSdistFileName`

Type: isSdistFileName :: string -> bool

Check whether string is a sdist file or not.

name

: The filename string

::: {.example #function-library-example-lib.pypa.isSdistFileName}

`lib.pypa.isSdistFileName` usage example

# isSdistFileName "cryptography-41.0.1.tar.gz"
true

:::

`lib.pypa.matchWheelFileName`

Type: matchWheelFileName :: string -> [ string ]

Regex match a wheel file name, returning a list of match groups. Returns null if no match.

name

: Function argument

`lib.pypa.isWheelFileName`

Type: isWheelFileName :: string -> bool

Check whether string is a wheel file or not.

name

: The filename string

::: {.example #function-library-example-lib.pypa.isWheelFileName}

`lib.pypa.isWheelFileName` usage example

# isWheelFileName "cryptography-41.0.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
true

:::

`lib.pypa.parseWheelFileName`

Type: parseFileName :: string -> AttrSet

Parse PEP-427 wheel file names.

name

: The wheel filename is {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.

::: {.example #function-library-example-lib.pypa.parseWheelFileName}

`lib.pypa.parseWheelFileName` usage example

# parseFileName "cryptography-41.0.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
 {
  abiTag = {  # Parsed by pypa.parseABITag
    implementation = "abi";
    version = "3";
    rest = "";
  };
  buildTag = null;
  distribution = "cryptography";
  languageTags = [  # Parsed by pypa.parsePythonTag
    {
      implementation = "cpython";
      version = "37";
    }
  ];
  platformTags = [ "manylinux_2_17_aarch64" "manylinux2014_aarch64" ];
  version = "41.0.1";
}

:::

`lib.pypa.isABITagCompatible`

Type: isABITagCompatible :: derivation -> string -> bool

Check whether an ABI tag is compatible with this python interpreter.

python

: Python interpreter derivation

abiTag

: ABI tag string

::: {.example #function-library-example-lib.pypa.isABITagCompatible}

`lib.pypa.isABITagCompatible` usage example

# isABITagCompatible pkgs.python3 (pypa.parseABITag "cp37")
true

:::

`lib.pypa.isPlatformTagCompatible`

Type: isPlatformTagCompatible :: AttrSet -> derivation -> string -> bool

Check whether a platform tag is compatible with this python interpreter.

platform

: Platform attrset (lib.systems.elaborate "x86_64-linux")

libc

: Libc derivation

platformTag

: Python tag

::: {.example #function-library-example-lib.pypa.isPlatformTagCompatible}

`lib.pypa.isPlatformTagCompatible` usage example

# isPlatformTagCompatible pkgs.python3 "manylinux2014_x86_64"
true

:::

`lib.pypa.isPythonTagCompatible`

Type: isPythonTagCompatible :: derivation -> AttrSet -> bool

Check whether a Python language tag is compatible with this Python interpreter.

python

: Python interpreter derivation

pythonTag

: Python tag

::: {.example #function-library-example-lib.pypa.isPythonTagCompatible}

`lib.pypa.isPythonTagCompatible` usage example

# isPythonTagCompatible pkgs.python3 (pypa.parsePythonTag "py3")
true

:::

`lib.pypa.isWheelFileCompatible`

Type: isWheelFileCompatible :: derivation -> AttrSet -> bool

Check whether wheel file name is compatible with this python interpreter.

platform

: Platform attrset (lib.systems.elaborate "x86_64-linux")

libc

: Libc derivation

python

: Python interpreter derivation

file

: The parsed wheel filename

::: {.example #function-library-example-lib.pypa.isWheelFileCompatible}

`lib.pypa.isWheelFileCompatible` usage example

# isWheelFileCompatible pkgs.python3 (pypa.parseWheelFileName "Pillow-9.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl")
true

:::

`lib.pypa.selectWheels`

Type: selectWheels :: AttrSet -> derivation -> [ AttrSet ] -> [ AttrSet ]

Select compatible wheels from a list and return them in priority order.

platform

: Platform attrset (lib.systems.elaborate "x86_64-linux")

python

: Python interpreter derivation

files

: List of files as parsed by parseWheelFileName

::: {.example #function-library-example-lib.pypa.selectWheels}

`lib.pypa.selectWheels` usage example

# selectWheels (lib.systems.elaborate "x86_64-linux") [ (pypa.parseWheelFileName "Pillow-9.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl") ]
[ (pypa.parseWheelFileName "Pillow-9.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl") ]

:::

eggs

`lib.eggs.matchEggFileName`

Type: matchEggFileName :: string -> [ string ]

Regex match an egg file name, returning a list of match groups. Returns null if no match.

name

: Function argument

`lib.eggs.isEggFileName`

Type: isEggFileName :: string -> bool

Check whether string is an egg file or not.

name

: The filename string

::: {.example #function-library-example-lib.eggs.isEggFileName}

`lib.eggs.isEggFileName` usage example

# isEggFileName "cryptography-41.0.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
false

:::

`lib.eggs.parseEggFileName`

Type: parsehEggFileName :: string -> AttrSet

Parse an egg file name.

name

: Function argument

::: {.example #function-library-example-lib.eggs.parseEggFileName}

`lib.eggs.parseEggFileName` usage example

# parseEggFileName

:::

`lib.eggs.selectEggs`

Type: selectEggs :: derivation -> [ AttrSet ] -> [ AttrSet ]

Select compatible eggs from a list and return them in priority order.

python

: Python interpreter derivation

files

: List of files parsed by parseEggFileName

pip

`lib.pip.parseRequirementsTxt`

Type: parseRequirementsTxt :: AttrSet -> list

Parse dependencies from requirements.txt

requirements

: The contents of or path to requirements.txt

::: {.example #function-library-example-lib.pip.parseRequirementsTxt}

`lib.pip.parseRequirementsTxt` usage example

# parseRequirements ./requirements.txt
[ { flags = []; requirement = {}; # Returned by pep508.parseString } ]

:::

Build

The pyproject.nix build infrastructure is mainly targeted at python2nix authors, and is being used in uv2nix.

Problems with nixpkgs Python builders

Propagation causes several issues downstream.

`$PYTHONPATH` leaking into unrelated builds

Consider the following development shell using nixpkgs Python builders:

let
  pkgs = import <nixpkgs> { };
  pythonEnv = pkgs.python3.withPackages(ps: [ ps.requests ]);
in pkgs.mkShell {
  packages = [
    pkgs.remarshal
    pythonEnv
  ];
}

Infinite recursions

Binary wrapping

Extraneous rebuilds

Because buildPythonPackage uses propagation runtime dependencies of a package are required to be present at build time.

Solution presented by pyproject.nix's builders

The solution is to decouple the runtime dependency graph from the build time one, by putting runtime dependencies in passthru:

stdenv.mkDerivation {
  pname = "setuptools-scm";
  version = "8.1.0";
  src = fetchurl {
    url = "https://files.pythonhosted.org/packages/4f/a4/00a9ac1b555294710d4a68d2ce8dfdf39d72aa4d769a7395d05218d88a42/setuptools_scm-8.1.0.tar.gz";
    hash = "";
  };

  passthru = {
    dependencies = {
      packaging = [ ];
      setuptools = [ ];
    };

    optional-dependencies = {
      toml = { toml = [ ]; };
      rich = { rich = [ ]; };
    };
  };

  nativeBuildInputs = [
    pyprojectHook
  ] ++ resolveBuildSystem (
    {
      setuptools = [ ];
    }
  );
}

Resolving

Because runtime dependencies are not propagated every package needs to resolve the runtime dependencies of their build-system's.

Additionally packages can't simply be consumed, but must be aggregated into a virtual environment to be useful:

{ pyproject-nix, pkgs }:

let
  python = pkgs.python312;

  # Inject your own packages on top with overrideScope
  pythonSet = pkgs.callPackage pyproject-nix.build.packages {
    inherit python;
  };

in pythonSet.pythonPkgsHostHost.mkVirtualEnv "test-venv" {
  build = [ ];
}

Cyclic dependencies

Cyclic dependencies are supported thanks to the resolver returning a flat list of required Python packages. For performance reasons two solvers are implemented:

One that does not support cyclic dependencies A much more performant resolver used by resolveBuildSystem and has all known build-systems memoized.
One that does support cyclic dependencies Used to resolve virtual environments

It's possible to override the resolver used entirely, so even though cyclic build-system's are not supported by default, it can be done with overrides.

Less rebuilds

As the runtime dependency graph is decoupled from the build time one derivation hashes change far less frequently.

An additional benefit is improved build scheduling: Because the dependency graph is much flatter than a buildPythonPackage based one derivations can be more efficiently scheduled in parallel.

Use virtualenvs

Instead of binary wrappers & environment variables pyproject.nix's builders use standard Python virtual environments.

build.lib

build.lib.renderers

`build.lib.renderers.mkDerivation`

Type: mkDerivation :: AttrSet -> AttrSet

Renders a project as an argument that can be passed to stdenv.mkDerivation.

Evaluates PEP-508 environment markers to select correct dependencies for the platform but does not validate version constraints.

structured function argument

: project

: Loaded pyproject.nix project

environ

: PEP-508 environment

extras

: Extras to enable (markers only, optional-dependencies are not enabled by default)

`build.lib.renderers.mkDerivationEditable`

Type: mkDerivation :: AttrSet -> AttrSet

Renders a project as an argument that can be passed to stdenv.mkDerivation.

Evaluates PEP-508 environment markers to select correct dependencies for the platform but does not validate version constraints.

Note: This API is unstable and subject to change.

structured function argument

: project

: Loaded pyproject.nix project

environ

: PEP-508 environment

extras

: Extras to enable (markers only, optional-dependencies are not enabled by default)

root

: Editable root directory as a string

build.lib.resolvers

`build.lib.resolvers.resolveNonCyclic`

Resolve dependencies using a non-circular supporting approach.

This implementation is faster than the one supporting circular dependencies, and is memoized.

resolveNonCyclic is intended to resolve build-system dependencies.

memoNames

: List of package names to memoize

set

: Package set to resolve packages from

`build.lib.resolvers.resolveCyclic`

Resolve dependencies using a cyclic supporting approach.

resolveCyclic is intended to resolve virtualenv dependencies.

set

: Package set to resolve packages from

spec

: Attribute set of dependencies -> extras { requests = [ "socks" ]; }

build.packages.hooks

`build.packages.hooks.pyprojectConfigureHook`

Undo any $PYTHONPATH changes done by nixpkgs Python infrastructure dependency propagation.

Used internally by pyprojectHook.

`build.packages.hooks.pyprojectBuildHook`

Build a pyproject.toml/setuptools project.

Used internally by pyprojectHook.

`build.packages.hooks.pyprojectWheelDistHook`

Symlink prebuilt wheel sources.

Used internally by pyprojectWheelHook.

`build.packages.hooks.pyprojectInstallHook`

Install built projects from dist/*.whl.

Used internally by pyprojectHook.

`build.packages.hooks.pyprojectOutputSetupHook`

Create pyproject.nix setup hook in package output.

Used internally by pyprojectHook.

`build.packages.hooks.pyprojectCrossShebangHook`

Rewrite shebangs for cross compiled Python programs.

When cross compiling & installing a Python program the shebang gets written for the install-time Pythhon, which for cross compilation is for the build host.

This hook rewrites any shebangs pointing to the build host Python to the target host Python.

`build.packages.hooks.pyprojectMakeVenvHook`

Create a virtual environment from buildInputs

Used internally by mkVirtualEnv.

`build.packages.hooks.pyprojectHook`

Meta hook aggregating the default pyproject.toml/setup.py install behaviour and adds Python.

This is the default choice for both pyproject.toml & setuptools projects.

`build.packages.hooks.pyprojectWheelHook`

Hook used to build prebuilt wheels.

Use instead of pyprojectHook.

build.hacks

`build.hacks.nixpkgsPrebuilt`

Use a package output built by Nixpkgs Python infrastructure.

Adapts a package by:

Stripping dependency propagation
Throwing away shell script wrapping
Filtering out sys.path dependency injection

This adaptation will of course break anything depending on other packages by $PATH, as these are injected by wrappers.

Example

nixpkgsPrebuilt {
  from = pkgs.python3Packages.torchWithoutCuda;
  prev = prev.torch;
}
=>
«derivation /nix/store/3864g3951bkbkq5nrld5yd8jxq7ss72y-torch-2.4.1.drv»

Type

nixpkgsPrebuilt :: AttrSet -> derivation

Arguments

from : Prebuilt package to transform output from

prev : Previous pyproject.nix package to take passthru from

`build.hacks.importCargoLock`

Build a Cargo (Rust) package using rustPlatform.importCargoLock to fetch Rust dependencies.

Uses IFD (import-from-derivation) on non-local packages.

Example

importCargoLock {
  prev = prev.cryptography;
  # Lock file relative to source root
  lockFile = "src/rust/Cargo.lock";
}
=>
«derivation /nix/store/g3z1zlmc0sqpd6d5ccfrx3c4w4nv5dzr-cryptography-43.0.0.drv»

Type

importCargoLock :: AttrSet -> derivation

Arguments

prev : Previous pyproject.nix package

importCargoLockArgs : Arguments passed directly to rustPlatform.importCargoLock function

cargoRoot : Path to Cargo source root

lockFile : Path to Cargo.lock (defaults to ${cargoRoot}/Cargo.lock)

doUnpack : Whether to unpack sources using an intermediate derivation

unpackDerivationArgs : Arguments passed directly to intermediate unpacker derivation (unused for path sources)

cargo : cargo derivation

rustc : rustc derivation

pkg-config : pkg-config derivation

build.util

`build.util.mkApplication`

Build applications without venv cruft.

Virtual environments contains many files that are not relevant when distributing applications. This includes, but is not limited to

Python interpreter
Activation scripts
pyvenv.cfg

This helper creates a new derivation, only symlinking venv files relevant for the application.

Example

util.mkApplication {
  venv = pythonSet.mkVirtualEnv "mkApplication-check-venv" {
    pip = [ ];
  }
  package = pythonSet.pip;
}
=>
«derivation /nix/store/i60rydd6sagcgrsz9cx0la30djzpa8k9-pip-24.0.drv»

Type

mkApplication :: AttrSet -> derivation

Arguments

venv : Virtualenv derivation created using mkVirtualEnv

package : Python set package

Hacking

This document outlines hacking on pyproject.nix itself, and lays out it's project structure.

Getting started

To start hacking run nix develop -c hivemind to run the project in watch mode.

This will start up two processes:

A Nix-unit test runner
A documentation server available at http://localhost:3000

Project structure & testing

All Nix code lives in lib/. Each file has an implementation and a test suite. The attribute path to a an attribute parseVersion in lib/pep440.nix would be lib.pep440.parseVersion.

A function in lib/test.nix maps over the public interface of the library and the test suite to generate coverage tests, ensuring that every exported symbol has at least one test covering it.

Integration tests meaning tests that perform environment constructions & builds lives in test/ and are exposed through Flake checks.

The manual you are reading right now is built from the doc/ directory. To edit a specific page see the "Edit this page on GitHub" link in the footer for each respective page.

Running tests

Run the entire unit test suite $ nix-unit --flake .#libTests
Run unit tests for an individual function $ nix-unit --flake .#libTests.pep440.parseVersion
Run integration tests $ nix flake check

Formatter

Before submitting a PR format the code with nix fmt and ensure Flake checks pass with nix flake check.

Support

Commercial support for the pyproject.nix ecosystem is offered by Bladis Limited.