This post may require some preknowledge of Nix.

This has been an idea running through my head for a while, so I gotta write it down.

Nixpkgs' shell nut problem in a nutshell

If you've ever worked with Nixpkgs, you know that essentially all of Nixpkgs is a fancy wrapper around a bunch of Bash scripts. This is not exactly ideal for multiple reasons:

There are attempts to improve on the situation, such as a Nix RFC to change stdenv from Bash to oil, but that doesn't address the main purity issue of requiring imperative build steps in a declarative system.

This got me thinking: how would one design a "pure", functional approach to derivation-making, that's just as easy to use as a shell script, but avoids all of its pitfalls?

Before we can answer that question, we first have to figure out

Why is it like this in the first place?

I've been using "package", so it's time to get more specific with terminology: in Nix, packages are not called "packages", but rather "derivations". I'll use "packages" for the nebulous human understanding of software, and "derivations" for the technical Nix way of specifying it.

The way Nix derivations are structured is detailed way better in Nix Pills ch. 6 than I could ever write here, but here's a brief summary:

At the very bottom level, Nix derivations can be constructed with the derivation builtin function, which takes arguments as follows:

drv = derivation { 
    name = "derivation-name"; # name of the derivation
    builder = "builder-path"; # executable that's run to get the final package
    system = "system-specifier"; # e.g. x86_64-linux  
}

It creates an attrset which contains the "magic property" of type = "derivation", which the Nix tools treat differently than other kinds of attrsets.

One may immediately spot the problem here. drv.builder is... well... an executable. As soon as we bring in executables, we have to deal with all sorts of thorny issues with impure execution environments (environment variables, CPU inconsistencies, ...) As a general rule of thumb, imperative executable isolation is way harder than declarative isolation.1

So why is drv.builder an executable? Well, I have no idea. Eelco's original Nix PhD thesis does not seem to state a reason, and the only thing I can guess from it is that they originally wanted it to be an executable because it's easier to write build scripts invoking gcc.

But this is (checks PhD thesis date) 2006. We can do better than that!

Alright, so how do we fix this?

Derivations shouldn't have to use builder programs by default. It simply is too much of a hassle. Instead, how about we have a fully declarative file tree? Imagine if one could write a derivation that looks like:

derivation {
    name = "hello-world-1.0.0";
    system = "x86_64-linux";
    tree = dir {
        hello = dir {
            world = file "Hello World!";
        };
    };
}

Which produces a file tree like:

.
└── hello
   └── world

It would allow for easy creation of e.g. wrapper derivations:

derivation {
    name = "gcc-with-zlib-1.0.0";
    system = "x86_64-linux";
    tree = dir {
        bin = dir {
            gcc-with-zlib = fileWithPerms "0555" ''#!${pkgs.bash}/bin/bash
                ${pkgs.gcc}/bin/gcc -L${pkgs.zlib}/lib "$@"
            '';
        };
    };
};

It could be integrated with build scripts:

derivation = {
    name = "myproj-1.0";
    system = "x86_64-linux";
    tree = dir {
        bin = dir rec {
            myproj = fromCommand ["${pkgs.bash}/bin/bash" "-c" ''
                # produces `myproj` executable
                make
                # $entry corresponds to the fs entry we're editing
                mv myproj $entry
            ''];
            myproj-wrapper = fileWithPerms "0555" ''#!${pkgs.bash}/bin/bash
                ${myproj} --something-or-other "$@"
            '';
        };
    };
};

Footnotes

  1. Yes, I know that Nix takes into consideration that derivation builders are impure by default (if you read Eelco's thesis, he talks about this, and proposes the Nix intensional model as a solution.) But there should still be no need to pull in the full executable sandbox machinery for just a simple file tree, e.g. pkgs.writeShellScript or pkgs.writeText.