docs: new and shiny storage limit docs (#17716)

* docs: new and shiny storage limit docs

Signed-off-by: William Woodruff <[email protected]>

* add note about yanking

Signed-off-by: William Woodruff <[email protected]>

* `make translations`

Signed-off-by: William Woodruff <[email protected]>

* docs: add yanking

Signed-off-by: William Woodruff <[email protected]>

* docs/help: re-add help entries

Signed-off-by: William Woodruff <[email protected]>

---------

Signed-off-by: William Woodruff <[email protected]>

Author: woodruffw

dev/environment

@@ -5,6 +5,8 @@ WAREHOUSE_ENV=development
 WAREHOUSE_TOKEN=insecuretoken
 WAREHOUSE_IP_SALT="insecure himalayan pink salt"
 
+USERDOCS_DOMAIN="http://localhost:10000"
+
 TERMS_NOTIFICATION_BATCH_SIZE=0
 
 AWS_ACCESS_KEY_ID=foo

docs/mkdocs-user-docs.yml

@@ -61,6 +61,9 @@ edit_uri: blob/main/docs/user/
 
 nav:
   - "index.md"
+  - "Project Management":
+      - "project-management/storage-limits.md"
+      - "project-management/yanking.md"
   - "Organization Accounts":
       - "organization-accounts/index.md"
       - "organization-accounts/org-acc-faq.md"

docs/user/assets/project-size-and-limits.png

@@ -0,0 +1,104 @@
+---
+title: Storage Limits
+---
+
+PyPI imposes storage limits on the size of individually uploaded files,
+as well as the total size of all files in a project.
+
+The current default limits are **100.0 MB** for individual files and **10.0 GB**
+for the entire project.
+
+You can see your project's current size and storage limits on
+the project settings page (`https://pypi.org/manage/project/YOUR-PROJECT/settings/`):
+
+![](/assets/project-size-and-limits.png)
+
+## File size limits
+
+By default, PyPI limits the size of individual files to **100.0 MB**.
+If you attempt to upload a file that exceeds this limit, you'll receive
+an error like the following:
+
+```console
+Uploading sampleproject-1.2.3.tar.gz
+HTTPError: 400 Client Error: File too large. Limit for project 'sampleproject' is 100 MB.
+```
+
+### Requesting a file size limit increase
+
+!!! note
+
+    Note: All users submitting feedback, reporting issues or contributing to
+    PyPI are expected to follow the
+    [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md).
+
+If you can't upload your project's release to PyPI because you're hitting the
+upload file size limit, we can sometimes increase your limit. Make sure you've
+uploaded at least one release for the project that's under the limit
+(a [developmental release version number](https://packaging.python.org/en/latest/specifications/version-specifiers/#developmental-releases) is fine). Then,
+[file an issue](https://github.com/pypi/support/issues/new?assignees=&labels=limit+request&template=limit-request-file.yml&title=File+Limit+Request%3A+PROJECT_NAME+-+000+MB) and tell
+us:
+
+- A link to your project on PyPI (or TestPyPI)
+- The size of your release, in megabytes
+- Which index/indexes you need the increase for (PyPI, TestPyPI, or both)
+- A brief description of your project, including the reason for the additional size.
+
+## Project size limits
+
+By default, PyPI limits the total size of all files in a project to **10.0 GB**.
+If you attempt to upload a file that would exceed this limit, you'll receive
+an error like the following:
+
+```console
+Uploading sampleproject-1.2.3.tar.gz
+HTTPError: 400 Client Error: Project size too large. Limit for project 'sampleproject' total size is 10 GB.
+```
+
+### Freeing up storage on an existing project
+
+!!! important
+
+    Deleting and [yanking](./yanking.md) are two different actions. Yanking a release or file
+    does **not** free up storage space.
+
+!!! warning
+
+    Deleting releases and files from your project is permanent and cannot be undone
+    without administrative intervention.
+
+!!! warning
+
+    Deletion can be very disruptive for downstream dependencies of your project,
+    since it breaks installation for
+    [pinned versions](https://pip.pypa.io/en/stable/topics/repeatable-installs/).
+
+    Before performing a deletion, we **strongly** recommend that you
+    consider the potential impact on your downstreams.
+
+If you're hitting the project size limit, you can free up storage by removing
+old releases or individual files from your project. To do this:
+
+1. Navigate to the release management for your project: `https://pypi.org/manage/project/YOUR-PROJECT/releases/`;
+2. Click on `Options` next to the release you wish to delete from;
+    - If you wish to delete the entire release, click `Delete`;
+    - If you wish to delete individual files from the release, click `Manage`,
+      then use each file's `Options` menu to delete it.
+
+### Requesting a project size limit increase
+
+!!! note
+
+    Note: All users submitting feedback, reporting issues or contributing to
+    PyPI are expected to follow the
+    [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md).
+
+If you can't upload your project's release to PyPI because you're hitting the project size limit,
+first [remove any unnecessary releases or individual files](#freeing-up-storage-on-an-existing-project)
+to lower your overall project size.
+
+If that is not possible, we can sometimes increase your limit. [File an issue](https://github.com/pypi/support/issues/new?assignees=&labels=limit+request&template=limit-request-project.yml&title=Project+Limit+Request%3A+PROJECT_NAME+-+00+GB) and tell us:
+
+- A link to your project on PyPI (or TestPyPI)
+- The total size of your project, in gigabytes
+- A brief description of your project, including the reason for the additional size.

docs/user/assets/release-options-yank.png

@@ -0,0 +1,51 @@
+---
+title: Yanking
+---
+
+!!! note
+
+    PyPI currently only supports yanking of *entire releases*, not individual files.
+
+PyPI supports *yanking* as a non-destructive alternative to deletion.
+
+A *yanked release* is a release that is always ignored by an installer, unless it
+is the only release that matches a [version specifier] (using either `==` or `===`).
+See [PEP 592] for more information.
+
+[version specifier]: https://packaging.python.org/en/latest/specifications/version-specifiers/
+
+[PEP 592]: https://peps.python.org/pep-0592/
+
+## When should I yank a release?
+
+Like deletion, yanking should be done sparingly since it can be disruptive to
+downstream users of a package.
+
+Maintainers should consider yanking a release when:
+
+- The release is broken or uninstallable.
+- The release violates its own compatibility guarantees. For example, `sampleproject 1.0.1`
+  might be yanked if it's *unintentionally* incompatible with `sampleproject 1.0.0`.
+- The release contains a security vulnerability.
+
+## How do I yank a release?
+
+To yank a release, go to the release management page for your project:
+`https://pypi.org/manage/project/YOUR-PROJECT/releases/`.
+
+Click on the `Options` button next to the release you wish to yank, then click `Yank`:
+
+![](/assets/release-options-yank.png)
+
+A modal dialogue will appear, asking you to confirm the yank and provide an
+optional reason for yanking:
+
+![](/assets/yank-confirm-modal.png)
+
+The reason, if provided, will be displayed on the release page as well
+as in the [index APIs](../api/index-api.md) used by installers.
+
+!!! tip
+
+    Providing a reason for yanking is **strongly encouraged**, as it can
+    help downstream users determine how to respond to the yank.

docs/user/assets/yank-confirm-modal.png

@@ -25,6 +25,9 @@ def test_includeme(forklift_domain, monkeypatch):
     _help_url = pretend.stub()
     monkeypatch.setattr(forklift, "_help_url", _help_url)
 
+    _user_docs_url = pretend.stub()
+    monkeypatch.setattr(forklift, "_user_docs_url", _user_docs_url)
+
     config = pretend.stub(
         get_settings=lambda: settings,
         include=pretend.call_recorder(lambda n: None),
@@ -56,7 +59,10 @@ def test_includeme(forklift_domain, monkeypatch):
         ),
     ]
 
-    assert config.add_request_method.calls == [pretend.call(_help_url, name="help_url")]
+    assert config.add_request_method.calls == [
+        pretend.call(_help_url, name="help_url"),
+        pretend.call(_user_docs_url, name="user_docs_url"),
+    ]
     if forklift_domain:
         assert config.add_template_view.calls == [
             pretend.call(
@@ -90,3 +96,16 @@ def test_help_url():
     assert request.route_url.calls == [
         pretend.call("help", _host=warehouse_domain, _anchor="foo")
     ]
+
+
+def test_user_docs_url():
+    docs_domain = "http://example.com"
+    request = pretend.stub(
+        registry=pretend.stub(settings={"userdocs.domain": docs_domain}),
+    )
+
+    assert forklift._user_docs_url(request, "/foo") == f"{docs_domain}/foo"
+    assert (
+        forklift._user_docs_url(request, "/foo", anchor="bar")
+        == f"{docs_domain}/foo#bar"
+    )

docs/user/project-management/storage-limits.md

@@ -856,9 +856,7 @@ def test_fails_with_invalid_names(self, pyramid_config, db_request, name):
 
         assert resp.status_code == 400
         assert resp.status == (
-            "400 The name {!r} isn't allowed. "
-            "See /the/help/url/ "
-            "for more information."
+            "400 The name {!r} isn't allowed. See /the/help/url/ for more information."
         ).format(name)
 
     @pytest.mark.parametrize(
@@ -1622,29 +1620,25 @@ def test_upload_fails_with_deprecated_classifier(
             {"md5_digest": "bad"},
             {
                 "sha256_digest": (
-                    "badbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbad"
-                    "badbadb"
+                    "badbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadb"
                 )
             },
             {
                 "md5_digest": "bad",
                 "sha256_digest": (
-                    "badbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbad"
-                    "badbadb"
+                    "badbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadb"
                 ),
             },
             {
                 "md5_digest": _TAR_GZ_PKG_MD5,
                 "sha256_digest": (
-                    "badbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbad"
-                    "badbadb"
+                    "badbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadbadb"
                 ),
             },
             {
                 "md5_digest": "bad",
                 "sha256_digest": (
-                    "4a8422abcc484a4086bdaa618c65289f749433b07eb433c51c4e37714"
-                    "3ff5fdb"
+                    "4a8422abcc484a4086bdaa618c65289f749433b07eb433c51c4e377143ff5fdb"
                 ),
             },
         ],
@@ -1722,9 +1716,7 @@ def test_upload_fails_with_invalid_file(self, pyramid_config, db_request):
         resp = excinfo.value
 
         assert resp.status_code == 400
-        assert resp.status == (
-            "400 Invalid distribution file. " "File is not a zipfile"
-        )
+        assert resp.status == ("400 Invalid distribution file. File is not a zipfile")
 
     def test_upload_fails_end_of_file_error(
         self, pyramid_config, db_request, project_service
@@ -1769,9 +1761,7 @@ def test_upload_fails_end_of_file_error(
         resp = excinfo.value
 
         assert resp.status_code == 400
-        assert resp.status == (
-            "400 Invalid distribution file. " "File is not a tarfile"
-        )
+        assert resp.status == ("400 Invalid distribution file. File is not a tarfile")
 
     def test_upload_fails_with_too_large_file(self, pyramid_config, db_request):
         user = UserFactory.create()
@@ -1800,14 +1790,21 @@ def test_upload_fails_with_too_large_file(self, pyramid_config, db_request):
                 ),
             }
         )
-        db_request.help_url = pretend.call_recorder(lambda **kw: "/the/help/url/")
+        db_request.user_docs_url = pretend.call_recorder(
+            lambda *a, **kw: "/the/help/url/"
+        )
 
         with pytest.raises(HTTPBadRequest) as excinfo:
             legacy.file_upload(db_request)
 
         resp = excinfo.value
 
-        assert db_request.help_url.calls == [pretend.call(_anchor="file-size-limit")]
+        assert db_request.user_docs_url.calls == [
+            pretend.call(
+                "/project-management/storage-limits",
+                anchor="requesting-a-file-size-limit-increase",
+            )
+        ]
         assert resp.status_code == 400
         assert resp.status == (
             "400 File too large. Limit for project 'foobar' is 100 MB. "
@@ -1847,14 +1844,21 @@ def test_upload_fails_with_too_large_project_size_default_limit(
                 ),
             }
         )
-        db_request.help_url = pretend.call_recorder(lambda **kw: "/the/help/url/")
+        db_request.user_docs_url = pretend.call_recorder(
+            lambda *a, **kw: "/the/help/url/"
+        )
 
         with pytest.raises(HTTPBadRequest) as excinfo:
             legacy.file_upload(db_request)
 
         resp = excinfo.value
 
-        assert db_request.help_url.calls == [pretend.call(_anchor="project-size-limit")]
+        assert db_request.user_docs_url.calls == [
+            pretend.call(
+                "/project-management/storage-limits",
+                anchor="requesting-a-project-size-limit-increase",
+            )
+        ]
         assert resp.status_code == 400
         assert resp.status == (
             "400 Project size too large."
@@ -1901,14 +1905,21 @@ def test_upload_fails_with_too_large_project_size_custom_limit(
                 ),
             }
         )
-        db_request.help_url = pretend.call_recorder(lambda **kw: "/the/help/url/")
+        db_request.user_docs_url = pretend.call_recorder(
+            lambda *a, **kw: "/the/help/url/"
+        )
 
         with pytest.raises(HTTPBadRequest) as excinfo:
             legacy.file_upload(db_request)
 
         resp = excinfo.value
 
-        assert db_request.help_url.calls == [pretend.call(_anchor="project-size-limit")]
+        assert db_request.user_docs_url.calls == [
+            pretend.call(
+                "/project-management/storage-limits",
+                anchor="requesting-a-project-size-limit-increase",
+            )
+        ]
         assert resp.status_code == 400
         assert resp.status == (
             "400 Project size too large."

docs/user/project-management/yanking.md

@@ -312,6 +312,7 @@ def __init__(self):
         "terms.revision": "initial",
         "terms.notification_batch_size": 1000,
         "warehouse.commit": "null",
+        "userdocs.domain": "https://docs.pypi.org",
         "site.name": "Warehouse",
         "token.two_factor.max_age": 300,
         "remember_device.days": 30,
@@ -355,17 +356,11 @@ def __init__(self):
                     "pyramid_debugtoolbar.panels.versions.VersionDebugPanel",
                     "pyramid_debugtoolbar.panels.settings.SettingsDebugPanel",
                     "pyramid_debugtoolbar.panels.headers.HeaderDebugPanel",
-                    (
-                        "pyramid_debugtoolbar.panels.request_vars."
-                        "RequestVarsDebugPanel"
-                    ),
+                    ("pyramid_debugtoolbar.panels.request_vars.RequestVarsDebugPanel"),
                     "pyramid_debugtoolbar.panels.renderings.RenderingsDebugPanel",
                     "pyramid_debugtoolbar.panels.session.SessionDebugPanel",
                     "pyramid_debugtoolbar.panels.logger.LoggingPanel",
-                    (
-                        "pyramid_debugtoolbar.panels.performance."
-                        "PerformanceDebugPanel"
-                    ),
+                    ("pyramid_debugtoolbar.panels.performance.PerformanceDebugPanel"),
                     "pyramid_debugtoolbar.panels.routes.RoutesDebugPanel",
                     "pyramid_debugtoolbar.panels.sqla.SQLADebugPanel",
                     "pyramid_debugtoolbar.panels.tweens.TweensDebugPanel",

tests/unit/forklift/test_init.py

@@ -349,6 +349,9 @@ def configure(settings=None):
     maybe_set(settings, "warehouse.domain", "WAREHOUSE_DOMAIN")
     maybe_set(settings, "forklift.domain", "FORKLIFT_DOMAIN")
     maybe_set(settings, "auth.domain", "AUTH_DOMAIN")
+    maybe_set(
+        settings, "userdocs.domain", "USERDOCS_DOMAIN", default="https://docs.pypi.org"
+    )
     maybe_set(settings, "warehouse.legacy_domain", "WAREHOUSE_LEGACY_DOMAIN")
     maybe_set(settings, "site.name", "SITE_NAME", default="Warehouse")
     maybe_set(settings, "aws.key_id", "AWS_ACCESS_KEY_ID")