[RFC 0191] Lockfile Generation

[pyrox0]

Rendered

[pbsds]

I love the incentive, and this is indeed a problem we would like to solve.

This RFC describes a happy path, but does however not really go into many edge cases.
There are also a couple load-bearing "should"s, that IMO needs fleshing out.

My unresolved questions as so plentiful that I did not bother splitting them into separated threads. I think a second revision is needed before more a granular discussion becomes productive.

• lockfile generation
  • how do we handle edge cases, like the package.json not being at the root directory?
  • can we in any way patch the package.json file before the lockfile is generated?
  • can we in any way patch the lockfile? (Like changing an unsafe dependency.)
  • can package authors/maintainers in any way test the lockfile generation flow locally?
  • can we have multiple lockfiles, in the event of nixpkgs shipping multiple versions of a package? (These may be created through .override and not have a unique attribute path.)
  • can we have multiple lockfiles, in the event of a single package having multiple subpackages? (See pagefind.)
  • can we regenerate a lockfile for an older package version on release-xx.yy? (in the event of a dep that is insecure or that fails to build.)
• maintaining the lockfile monorepo
  • can we update a lockfile, e.g. in the event of a dep being unsafe, even if the package version is unchanged?
  • what should happen in the event of a package being renamed?
  • what should happen in the event of a false-detection of a package removal? (Eval failures will happen, those can be at the scope level.)
  • what should happen in the event of a package removal not being detected?
  • what should happen in the event of a package not needing a lockfile anymore? Can we automatically detect that?
  • should we even delete old lockfiles? what about reproducability?
• scaling and fetching the lockfile repo/store
  • should all lockfiles be put into a massive monorepo?
    • How should the folder layout look like?
    • Should a single revision of the whole lockfile monorepo be used throughout all expressions in nixpkgs? How do we prevent treewide rebuilds when bumping that rev?
    • Should multiple copies of the whole lockfile monorepo be used in nixpkgs? How do we prevent this from bloating the nix store of the user?
    • Should individual lockfiles from the lockfile monorepo be fetched per package expression in nixpkgs? How do we maintain all these fetchurl expressions? What should happen if we ever need to change the monorepo file layout?
  • should each lockfile be put into their own lockfile repo branch?
    • What should the branch name schema look like?
    • Will the github UI break if we have hundreds or thousands of branches?
  • should each lockfile be put into their own gist?
    • Do we have a limit on those?
    • How can a user browse and search in those?
  • should each lockfile be stored as GHA build artifacts?
    • Do we have a limit on those?
    • Do they last forever?
    • How can a user browse and search in those?
  • should we outsource the archival of lockfiles to some other actor than GitHub, like archive.org or the Software Heritage?
    • see https://archive.fosdem.org/2025/schedule/event/fosdem-2025-5897-guix-software-heritage-source-code-archiving-to-the-rescue-of-reproducible-deployment/
  • should each lockfile be stored and served by a website we self-host?
    • Should this website rely on being cached on cache.nixos.org, or need it take measures against ddos attacks?
    • Should we ever prune lockfiles from it? Do we consider it responible for reproducability or do we have cache.nixos.org for that?
• r-ryantm, nix-update, and incorrect maintanance
  • Can these automated load-bearing tools somehow correctly invoke the lockfile repo bot, retrieve the resulting lockfile url, and update the package expression correctly?
  • Can we prevent the case where fetchurl'ed lockfiles fail to get their hash updated in a package bump PR and go undetected?
  • How can a PR reviewer verify that a fetchurl'ed lockfile expression corresponds to the correct package?

I'm sorry if this wall of questions seems daunting. With a renewed view of various edge cases and considerations, I believe that focusing on how to we store the lockfiles will let the rest fall into place. It may also be benefitial to define scope early, on what cases we support and not, and what the fallback should be.

[pyrox0]

This RFC describes a happy path, but does however not really go into many edge cases.
There are also a couple load-bearing "should"s, that IMO needs fleshing out.

100%! I agree that there's a lot to be fleshed out here. This was my original design idea for it, and to be totally transparent, I created this over a few hours yesterday, which was kind of solidifying a general design I've had floating around in my head for a while. I wanted to put this out earlier in a more unfinished state, specifically so I could get input like this. It might take a while for me to answer all of those questions, but I do appreciate them!

[Ericson2314]

BTW, the idea of having a separate repo for verbose/generated data was also discussed as part of the #109 shepherd discussion.