Blog of Lalatendu Mohanty

Author: Lalatendu Mohanty

There are many existing projects/tools which builds python wheels, so the obvious question is: why do we need a new project i.e. Fromager to do the same thing. In this blog post, I will delve into the motivations behind Fromager and explain its unique role in the ecosystem.

The trust problem

When you run pip install numpy, a pre-built binary lands on your machine, and it works. That’s the happy path and for most development workflows, it’s fine.

But there are environments where the happy path is not enough.

A pre-built wheel is a binary artifact compiled by someone else’s CI system, from source code you may not have reviewed, with build tools you didn’t choose. If you work in an environment where you need to audit every binary you deploy e.g. financial services, government, defense, regulated AI, this is a non-starter. And after a string of supply chain attacks on Python packages (typosquatting, compromised maintainer accounts, malicious build hooks), this isn’t a theoretical concern anymore.

But security is only one piece. Organizations also end up building from source for other reasons:

• Custom hardware. If you need to link against a specific accelerator SDK i.e. a particular CUDA version, ROCm or you’re targeting an architecture nobody publishes wheels for, you have to build from source with the right flags.
• Regulatory compliance. Some frameworks require provenance: a documented chain from source code to deployed binary. A wheel downloaded from PyPI has no such chain.
• Reproducibility across platforms. When you manage multiple variants (x86_64, aarch64, CPU-only, CUDA), you need the same packages built consistently across all of them. Grabbing whatever wheels happen to exist on PyPI doesn’t give you that.

These requirements show up especially in the AI/ML ecosystem, where dependency trees are large, native code is everywhere, and hardware diversity is becoming the norm. If any of this sounds like your situation, you’ve probably already tried the obvious next step.

Why pip install –no-binary :all: doesn’t work

pip can build from source. Pass --no-binary :all: and it will compile every package instead of downloading wheels. But pip doesn’t solve the bootstrapping problem.

For example, to build numpy from source, pip needs setuptools. To build setuptools from source, pip needs… setuptools. This is a circular dependency. pip breaks this cycle by downloading pre-built wheels for build tools from PyPI. It has to because its architecture assumes that build tools are already available as binaries.

If your requirement is “every binary must be built from source in our environment,” pip cannot satisfy it. The build tools themselves are a gap. This bootstrapping challenge is well-known in the Python packaging community — it’s been discussed since 2020 and remains an open problem.

Also there’s no way to customize build flags per package, you can set global environment variables, but you can’t say “build numpy with CUDA support and scipy with OpenBLAS.” There’s no patching mechanism if upstream source doesn’t compile on your platform. There’s no cross-compilation support. And there’s no way to manage multiple platform variants (CPU, CUDA, ROCm) from a single build pipeline.

pip builds for the machine it’s running on, one package at a time, with whatever it finds.

uv has the same limitations. conda doesn’t build from source at all, it’s a binary-only package manager.

Why not Nix or Gentoo?

Nix and Gentoo do build everything from source, including build tools. They solve the bootstrapping problem. But they solve it by requiring a manually written build specification for every single package i.e. a Nix derivation or a Gentoo ebuild. Spack, widely used in HPC and scientific computing, takes the same approach: every package needs a handwritten package.py recipe that specifies how to fetch, configure, and build it.

For a Python dependency tree of hundreds of packages that changes frequently, writing and maintaining a build recipe per package doesn’t scale.

Bazel with rules_python provides hermetic, reproducible builds and can auto-generate targets for PyPI dependencies. But it primarily consumes pre-built wheels, building from source sdists with native extensions is still an open feature request. It’s designed for monorepos where you control your own code, not for bootstrapping an upstream ecosystem of packages entirely from source.

Here’s the thing: Python already has a standardized way for packages to declare their build requirements i.e. PEP 517 and pyproject.toml.

Every well-maintained Python package already says what it needs to build. The information is there. It just needs a tool that can use it, all the way down.

What fromager does

Fromager takes a list of top-level packages you want and discovers, resolves, and builds the entire dependency tree from source, including all build tools, all the way down to the bottom.

It does this entirely within the Python ecosystem. It reads pyproject.toml, calls PEP 517 hooks, consumes packages from PyPI, and produces standard wheels.

You don’t need to adopt Nix, learn Spack recipes, or write Bazel BUILD files. You need to provide the top-level packages. Fromager figures out what needs to be built and in what order.

It distinguishes build dependencies from install dependencies

This is the key design decision. The dependency graph has two fundamentally different kinds of edges:

• Build dependencies: packages needed to compile the software (setuptools, Cython, wheel). These must be fully built and installed before the package that needs them can be compiled.
• Install dependencies: packages needed to use the software at runtime (numpy, requests). These are discovered after the build phase completes, by reading metadata from the resulting wheel or sdist.

Build dependencies must be processed depth-first and compiled immediately. Install dependencies can be deferred. Fromager tracks this at a granular level, different categories of build requirements must be installed in a specific sequence (you need the build system before you can call PEP 517 hooks to discover what else is needed).

It discovers dependencies automatically

Fromager doesn’t require manually written build specifications. For each package, it:

1. Reads pyproject.toml to find build system requirements
2. Installs those into an isolated build environment
3. Calls PEP 517’s get_requires_for_build_wheel() hook to discover additional build requirements
4. Recursively applies the same process to every dependency it discovers

The build order emerges from the traversal i.e. an iterative depth-first loop over an explicit stack, not from a human writing it down.

It’s customizable without forking

This is one of the biggest strengths of fromager. When you are building hundreds of packages we will come across situations where the python project does not follow standard repository structure or other Python standards which would cause issues during the build process. For example a package may need a specific environment variable, a patch to its build system, or a pinned version that differs from what PyPI advertises.

Fromager handles this with a layered override system. Common adjustments, environment variables, patches, version pins go in per-package YAML settings files and don’t require any code.

While standard Python packaging currently lacks awareness of accelerators, fromager was specifically designed to fill this void.

A --variant flag lets you build for different targets (CPU, CUDA, ROCm) using the same pipeline with different settings per variant. For packages that don’t follow Python packaging standards at all, you can write a plugin to handle the edge case.

The system evolves over time: as common patterns emerge across packages, they get promoted into fromager itself, so what started as a plugin becomes a YAML setting.

It separates discovery from building

This is where the supply chain security story comes together. Fromager can split the build process into two stages separated by a data-only boundary:

Stage 1 (Discovery) resolves versions, queries package indexes, and runs PEP 517 hooks. Because those hooks are arbitrary Python code defined by each package’s build backend, this stage executes untrusted upstream code and requires network access. It produces:

• graph.json: every package, version, and typed dependency edge
• build-order.json: a topologically sorted build sequence
• Cached requirement files for each package
• Downloaded source archives

Stage 2 (Build) compiles packages using only the Stage 1 artifacts. It uses cached requirement files instead of re-running discovery hooks, and builds from pre-downloaded sources. On Linux, when network isolation is enabled, build commands run inside network namespaces (unshare --net) with no outbound connectivity.

Between the two stages, you can inspect everything, diff the graph.json to see what changed, review it in a pull request, or transfer the artifacts to an air-gapped system and build with no network access at all. No compilation happens until you’re satisfied with the plan.

It makes builds reproducible

The graph.json fromager produces isn’t a disposable build artifact, it functions as an ecosystem-wide lock file. On subsequent runs, fromager loads the previous graph and uses it for version resolution. If numpy resolved to 1.26.4 last time, it resolves to 1.26.4 again, even if 1.27.0 has been published since. The graph can be checked into version control and used to ensure that multiple platform variants resolve to the same versions.

It handles scale

For large dependency trees, serial building is slow. Fromager can schedule concurrent builds using topology-aware parallelism i.e. packages that don’t depend on each other for building can be compiled simultaneously. Resource-intensive packages like PyTorch can be marked for exclusive builds so they don’t run in parallel with other compilations.

When onboarding new packages, you don’t know how many will fail to build from source. Fromager’s test mode continues after failures by substituting a pre-built binary for any package that fails, so downstream packages can still be built. At the end, it produces a JSON report of every failure classified by type , giving you a clear map of what still needs fixing.

Fromager is already helping us build a fully documented chain from source code to deployed binary that directly addresses the core concerns of the “AI-ready enterprise”. However, Fromager is a relatively new project. We are just getting started and fromager is going to be our cornerstone in delivering a secure and optimized supply chain of python packages on which enterprise AI will be built.

Fromager is open source: github.com/python-wheel-build/fromager.

For introductory walkthroughs, check out the video series on YouTube.

Note: Written with AI assistance.