Survey of Python-vendoring tools

[sethmlarson]

pip uses a project called "vendoring", it should be fairly straightforward to add SBOM generation to a project like this especially because the results live in the source tree (rather than being gathered at build-time).

• vendoring
• vendy

[di]

https://pypi.org/project/vendy/ exists as well, although I'm not sure it's widely used.

[di]

Thinking about this a bit: given that there is no standard for vendor-ing and that these two tools are not widely adopted, I think there's possibly a number of projects with vendor-ed dependencies might be missed by any SBOM standard.

For example, if a project is just copying one source tree into another and publishing it, this is exactly the thing we'd want an SBOM for: SCA will likely miss this, there's nothing declared in metadata about it, etc. But the project has no real incentive to do anything differently because there is no standard.

Given that, I wonder if it makes sense to pursue a standard for the use case that vendoring/vendy cover. Specifically:

• A way to declare vendored dependencies in project metadata (something like vendored_dependencies
• A standard way that tools like vendoring/vendy should bake these into the source tree

Feedback I've gotten before is that "fat distributions" isn't a use case we generally want to encourage, but the fact that these tools exist and that projects are likely vendor-ing other projects without using them as well leads me to think that it would be worthwhile (and would mean that any SBOM tool would not have to understand N different ways to vendor dependencies).

[sethmlarson]

@di Indeed, I've seen vendored Python dependencies used pretty frequently especially in the packaging space. Another future I could see is for one of those tools to become more widely adopted in projects where folks are vendoring dependencies.

I'm interested in seeing if there's a way to "detect" vendored projects, I used license detection as a part of my work on CPython's SBOMs to find bundled dependencies. Maybe that is effective here, too?

[ogrisel]

@sethmlarson now that pradyunsg/vendoring#66 is merged, how do you envision the process of including such a generated BOM file to the .dist-info/sboms reserved folder documented in PEP 770?

Is this the job of the build-backend to do this? Or should this be implemented first in generic wheel repair/post-processors such as auditwheel to avoid having to update all the build backends?

[sethmlarson]

@ogrisel For pip's case, the original plan was to have a user-specified key added to pyproject.toml. Then the SBOM could be added to pip/_vendor by vendoring and the project config points to it. This feature had contention and was holding up the definition of a .dist-info/sboms directory so that feature was factored out of the PEP.

I've thought about creating a simple script which adds an SBOM to an existing wheel and record to serve pip's use-case until a PEP is accepted. It would not be very tough to do.

For projects that pull in dependencies from their build backend (maturin, etc) then they can certainly add support! Auditwheel I am working on adding support already here: pypa/auditwheel#577

[ogrisel]

I've thought about creating a simple script which adds an SBOM to an existing wheel and record to serve pip's use-case until a PEP is accepted. It would not be very tough to do.

Sounds like a pragmatic workaround.