ci: confirm expected autogenerated reference exists

[zimeg]

Summary

This PR adds a check that confirms the expected autogenerated reference exists for a commit to confirm build previews compile and to reduce the overhead of a release 📚

For slackapi/slack-health-score#116 and investigation toward #1752!

Testing

Category

• /docs (Documents)
• tests/integration_tests (Automated tests for this library)

Requirements

• I've read and understood the Contributing Guidelines and have done my best effort to follow them.
• I've read and agree to the Code of Conduct.
• I've run python3 -m venv .venv && source .venv/bin/activate && ./scripts/run_validation.sh after making the changes.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.95%. Comparing base (83661b7) to head (b49b5b0).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1759   +/-   ##
=======================================
  Coverage   84.95%   84.95%           
=======================================
  Files         113      113           
  Lines       12905    12905           
=======================================
  Hits        10964    10964           
  Misses       1941     1941           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mwbrooks]

✅ Love love love this! I'm so excited to see this run. If it works well, perhaps we add it to the Node SDK too?

[zimeg]

@mwbrooks I share your excitement! 📚 ✨

An above check shows that the current changes of main don't match the expected reference, which I think we're hoping to find?

diff --git a/docs/reference/index.html b/docs/reference/index.html
index 8b9b9e3..9b6a27e 100644
--- a/docs/reference/index.html
+++ b/docs/reference/index.html
@@ -1856,7 +1856,7 @@ and message responses using response_url in payloads.</p></div>
     def admin_users_list(
         self,
         *,
-        team_id: str,
+        team_id: Optional[str] = None,

I'll add these changes to this PR for continued testing and polish 📠

[zimeg]

🗣️ note: We're hiding generated documentation with this change!

🔗 https://docs.github.com/en/repositories/working-with-files/managing-files/customizing-how-changed-files-appear-on-github

[zimeg]

🔍 question: I'm unsure if this is alright to add to this file. At the moment this script might create uncommitted changes without causing this script to error.

🗣️ ramble: I can imagine another environment perhaps being used to run the diff commands that exist in the GitHub workflows at the moment, but I found this implementation to be alright for remembering to generate these changes alongside the PR requirements to run tests!

[WilliamBergamin]

I fear that including the docs generation here may lead to more work for code reviewers 😅 Since we ask developers to run scripts/run_validation.sh before opening PRs, this might double the average length of our PRs.

This may add more complexity for reviewers review code changes and may make it harder for maintainers to resolve merge conflicts when collaborating on similar features.

Updating the generated docs as a release step has been working well, maybe we could clean up the release instructions to make it more clear. Let me know what you think

[zimeg]

@WilliamBergamin Thank you so much for sharing your findings with releases so far 🙏 ✨

I agree the generated reference causes a significant diff but we're hiding those from the Files changed tab of a PR with a new ".gitattributes" file. I don't believe these pages require review at all if the CI check passes to confirm these match the expected generated files.

This change overall was motivated in reducing maintenance across development - reference pages of development branches were either missing notes or included new changes from commits since the last release, which was confusing in previews I find.

Merge conflicts hadn't yet happened for me in recent changes either but I do think regenerating reference from the related code might be a suitable fix. I'm hoping to discuss keeping docs with the related commit more since we can now release docs according to a particular release tag instead of main 🤖

[WilliamBergamin]

This does all make sense 💯
I am still hesitant, there has been situation where I've notices doc generation issue when cutting releases. Those were found when reviewing the diff of the release PR. Configuring the ".gitattributes" file so that the diff around generated references is always hidden would prevent reviewers from catching errors around the docs generation. We would be blindly trusting the generation script 😅

IIRC at some point the path of the generation script was changes by mistake, this caused the generation script to generate the docs in a new directory while leaving the old docs unchanged, this was caught in the release PR through the diff

It may still be useful to allow reviewers to view the diff of the generated docs at some critical points in our development cycle

[WilliamBergamin]

thanks for putting this together 💯

I left a comment on including the docs generation in the validation script, let me know if you want to discuss more on this

[WilliamBergamin]

I fear that including the docs generation here may lead to more work for code reviewers 😅 Since we ask developers to run scripts/run_validation.sh before opening PRs, this might double the average length of our PRs.

This may add more complexity for reviewers review code changes and may make it harder for maintainers to resolve merge conflicts when collaborating on similar features.

Updating the generated docs as a release step has been working well, maybe we could clean up the release instructions to make it more clear. Let me know what you think