Policy: Empower reviewers to reject burdensome PRs

[jieyouxu]

Proposal: Empower reviewers to reject burdensome PRs

Reviewer time and effort are very precious and limited resources. However, there are Pull Requests (PRs) with certain qualities that place undue work on reviewers. We are proposing a policy that reviewers can use as justification for rejecting a PR without them having to excessively defend their position or relitigate the reasons for closing.

Motivation

In recent times, there are rather frequent contributions that are fully or partially produced by generative AI (e.g. LLMs and similar tools) which exhibit characteristics (see below) that make reviewing difficult or just a plain waste of time. While we've seen such PRs produced entirely by humans, generative AI tools have significantly lowered the effort to produce "plausibly-looking" contributions that are entirely inadmissible and in some cases against our Code of Conduct. This policy is thus drafted as a response to this problem, to avoid reviewers figuring out themselves how to respond.

PR characteristics that are red flags

PRs subject to this policy have characteristics such as (but not limited to):

• Needlessly or overly verbose descriptions or interaction (even when used to overcome a language barrier).
• Being not internally coherent or even self-contradictory. The code in the PR is plain wrong or not related to its stated purpose.
• Demonstrated misunderstanding of important aspects of what the existing code being modified is doing, what the code added by the PR is doing, or the purpose of the change.

It is okay to not be fully confident in a proposed change, but this should be disclaimed on the PR itself.

Policy

To mitigate undue work when encountering such PRs, reviewers may choose to exercise one or more of the following options:

• Ask the PR author to rework their PR to make it more easily reviewable.
• Edit out verbose PR descriptions.
• Reject (close) the PR.

Mentors or Reviewers

None specifically; compiler team.

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Note

Concerns (0 active)

• wait-for-some-discussions-to-settle resolved in this comment
• make-sure-messaging-does-not-imply-discouragement-of-ai-tools-as-aid resolved in this comment
• drop-disclosure-requirement resolved in this comment
• focus-on-the-problem resolved in this comment
• ai-should-still-be-mentioned-in-motivation resolved in this comment
• ai-should-still-be-mentioned-in-motivation resolved in this comment
• wording-nits resolved in this comment

Managed by @rustbot—see help for details.

[rustbot]

Important

This issue is not meant to be used for technical discussion. There is a Zulip stream for that.
Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections can formally be registered here by adding a comment.

@rustbot concern reason-for-concern
<description of the concern>

Concerns can be lifted with:

@rustbot resolve reason-for-concern

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[workingjubilee]

@rustbot second

[jieyouxu]

This deserves some more baking to discuss how we should frame this, most importantly, what this does not try to reject/discourage.

@rustbot concern wait-for-some-discussions-to-settle

[jieyouxu]

it's less about the intent of the MCP, and more about whether there is messaging in there that people using AI as a dev aide (i.e. not as a spam tool), would be pushed away from contributing or made feel inadequate as devs

@rustbot concern make-sure-messaging-does-not-imply-discouragement-of-ai-tools-as-aid

[jieyouxu]

I don't personally feel very strongly about having a declaration or not -- that could be a separate proposal. My core intention for this proposal was to have a

backing policy like this [that] allows the reviewer to close the low-effort PR quickly, without needing to spend a lot of energy or effort arguing.

Whether requiring disclosure or not IMO doesn't matter as much for this purpose.

Probably consider dropping the disclosure requirement, as it's not clear what should require a disclosure -- does simply using copilot for auto-completion for a very small subset of a PR count? That is likely better as a separate proposal.

@rustbot concern drop-disclosure-requirement

[jieyouxu]

AI tools are simply a method to produce overly verbose descriptions/responses and invalid changes, but is not the only way to arrive at them. We should focus on the problems, since there are many methods to arrive at them.

@rustbot concern focus-on-the-problem

[RalfJung]

@Kivooeo please read this comment.

[jieyouxu]

I adjusted this policy proposal following feedback on the thread. This policy is now focused only on burdensome PRs, and intentionally does not mention how they are produced (AI or not). Any responsible AI usage or AI usage disclaimer policies are intentionally beyond the scope of this MCP (but may be follow-ups).

EDIT: Generate AI is mentioned but only in the motivation capacity as "lowering the barrier to produce plausibly-looking contributions" without being focused on how burdensome PRs are produced.

@rustbot resolve make-sure-messaging-does-not-imply-discouragement-of-ai-tools-as-aid
@rustbot resolve drop-disclosure-requirement
@rustbot resolve focus-on-the-problem

[jieyouxu]

We seem to have a convergence on focusing on the characteristics of burdensome PRs (which seems uncontroversial) instead of the methodology of producing those, so:

@rustbot resolve wait-for-some-discussions-to-settle