Create fetch_job_status.py #833

araravik-psd · 2025-06-13T15:19:27Z

Creating a fetch_job_status.py python script to obtain Github actions job status output in the Workflow.

Adding a python helper script to be used as part of the Github actions worfklow to fetch job status for different jobs run as part of the workflow.

Sample Workflow changes to be added

- name: Determine job status for workflow run if: '!cancelled()' env: run_id: ${{ github.RUN_ID }} attempt: ${{ github.ATTEMPT }} run: python build_tools/github_actions/fetch_job_status.py

Creating a fetch_job_status.py python script to obtain Github actions job status output in the Workflow.

geomin12

Couple of questions in comments:

Will this be included in any workflow? This feels somewhat incomplete as it isn't tied in anywhere

geomin12 · 2025-06-13T16:07:54Z

build_tools/github_actions/fetch_job_status.py

+
+def run():
+    # TODO: Remove hardcoding for ATTEMPT once github variables fetch is fixed
+    github_release_url = f"https://api.github.com/repos/RoCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"


will this be used after the job is completed and getting the github status? or will this be a cronjob?

This will be used as part of the workflow and be added in after the job is completed to get the status of different steps in each job

This is the example of the workflow step

- name: Determine job status for workflow run if: '!cancelled()' env: run_id: ${{ github.RUN_ID}} attempt: ${{ github.ATTEMPT }} run: python build_tools/github_actions/fetch_job_status.py

Got it! Should we include that change now? And make sure this works properly?

geomin12 · 2025-06-13T16:08:11Z

build_tools/github_actions/fetch_job_status.py

+
+def run():
+    # TODO: Remove hardcoding for ATTEMPT once github variables fetch is fixed
+    github_release_url = f"https://api.github.com/repos/RoCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"


update variable github_release_url to be more relevant :)

sure will update the variable to "githib_run_url".

geomin12 · 2025-06-13T16:10:30Z

build_tools/github_actions/fetch_job_status.py

+            )
+
+        job_data = json.loads(response.read().decode("utf-8"))
+        if job_data["jobs"].keys() >= 0:


should this be checking len(job_data["jobs"].keys())?

I am fairly certain job_data["jobs"].keys() will result into a list of strings, and comparing it to >= won't be valid and will error in type issues

Hi Geo, this is a typo, wanted to to use len and missed it out at the end. Let me make the change and resubmit.

if len(job_data['jobs']).keys()>0

geomin12 · 2025-06-13T16:11:32Z

build_tools/github_actions/fetch_job_status.py

+
+        job_data = json.loads(response.read().decode("utf-8"))
+        if job_data["jobs"].keys() >= 0:
+            # Determine is number of jobs run in the workflow is atleast 1


can we clarify this comment? reading it is a bit difficult to understand

I am checking if the workflow's Github API status output shows atleast 1 job as run and getting populated in API the output.

# Determine if number of jobs run in the workflow is atleast 1

Ah, can you correct that? I see "is", it should be "if", and can you specify what kind of jobs?

geomin12 · 2025-06-13T16:12:20Z

build_tools/github_actions/fetch_job_status.py

+        job_data = json.loads(response.read().decode("utf-8"))
+        if job_data["jobs"].keys() >= 0:
+            # Determine is number of jobs run in the workflow is atleast 1
+            set_github_output({"append": json.dumps(job_data)})


set_github_output sets an output variable for other jobs to use. will this data used in other places?

I am planning to use the set_github_output variable in the next step of the workflow file where I write it to a Json file in S3, which will then be used by the Redshift database service to build a dashboard with the same.

Can you write down that plan somewhere (e.g. a GitHub issue)? I'd like to see what pieces you plan on building to make reviewing them easier.

If you are working within the context of a running workflow, there are ways to get job status without making an API request, like this: https://github.com/iree-org/iree/blob/main/.github/workflows/workflow_summary.yml (see iree-org/iree#19444 for context there)

geomin12 · 2025-06-13T16:13:16Z

also pls fix pre-commit :)

then pre-commit run --all-files

geomin12 · 2025-06-13T19:57:50Z

build_tools/github_actions/fetch_job_status.py

+        job_data = json.loads(response.read().decode("utf-8"))
+        if job_data["jobs"].keys() >= 0:
+            # Determine is number of jobs run in the workflow is atleast 1
+            set_github_output({"append": json.dumps(job_data)})


also noticed this on the call, but append is vague. can we call it a better variable name?

ScottTodd · 2025-06-13T21:31:00Z

build_tools/github_actions/fetch_job_status.py

Couple of questions in comments:

Will this be included in any workflow? This feels somewhat incomplete as it isn't tied in anywhere

For the file path, this script is simlar: https://github.com/ROCm/TheRock/blob/main/build_tools/hack/delete_workflow.sh

Copying one of my comments discussing that:

That sounds similar to some other utility scripts I've used like https://github.com/iree-org/iree/blob/main/build_tools/scripts/git/prune_old_tags.sh .

Could go in https://github.com/ROCm/TheRock/tree/main/build_tools/github_actions , or a similar subfolder under build_tools/ . It be -1 on a top level hack/ folder, but top level scripts/ would be okay, though there would be some overlap with build_tools/

Once we have more similar files we can decide where best to put them. Hard to commit with just a few.

ScottTodd · 2025-06-13T22:32:54Z

build_tools/github_actions/fetch_job_status.py

+  - ATTEMPT
+"""
+
+from configure_ci import set_github_output


We should pull set_github_output into a utils file instead of sharing across files this way.

ScottTodd · 2025-06-13T22:34:45Z

build_tools/github_actions/fetch_job_status.py

+        if response.status == 403:
+            raise Exception(
+                f"Error when retrieving GitHub response. This is most likely a rate limiting issue, so please try again"
+            )
+        elif response.status != 200:
+            raise Exception(
+                f"Error when retrieving GitHub response assets for {RUN_ID} tag with status code {response.status}. Exiting..."
+            )


Is there any more information in the response that could be logged here? Could also raise more specific Exception subclasses.

ScottTodd · 2025-06-13T22:38:20Z

build_tools/github_actions/fetch_job_status.py

+        job_data = json.loads(response.read().decode("utf-8"))
+        if job_data["jobs"].keys() >= 0:
+            # Determine is number of jobs run in the workflow is atleast 1
+            set_github_output({"append": json.dumps(job_data)})


Can you write down that plan somewhere (e.g. a GitHub issue)? I'd like to see what pieces you plan on building to make reviewing them easier.

If you are working within the context of a running workflow, there are ways to get job status without making an API request, like this: https://github.com/iree-org/iree/blob/main/.github/workflows/workflow_summary.yml (see iree-org/iree#19444 for context there)

ScottTodd · 2025-06-13T22:38:34Z

build_tools/github_actions/fetch_job_status.py

+
+def run():
+    # TODO: Remove hardcoding for ATTEMPT once github variables fetch is fixed
+    github_workflow_jobs_url = f"https://api.github.com/repos/RoCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"


nit: caps

Suggested change

github_workflow_jobs_url = f"https://api.github.com/repos/RoCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"

github_workflow_jobs_url = f"https://api.github.com/repos/ROCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"

ScottTodd · 2025-06-13T22:40:01Z

build_tools/github_actions/fetch_job_status.py

+RUN_ID = os.getenv("RUN_ID")
+ATTEMPT = os.getenv("ATTEMPT")


Relating to my other comment about seeing the whole plan before reviewing individual scripts, if this script runs outside of the workflow in question, I would expect arguments to be passed through the CLI with argparse. Since these are environment variables that GitHub provides, I'm wondering if you even need to make an API request to get the job results.

ScottTodd · 2025-06-16T20:19:00Z

Lots of commit activity here that looks unrelated to the code review. Can you convert to a draft PR and/or switch your debugging to another branch and use workflow_dispatch instead of pull_request triggers to test workflows?

ScottTodd · 2025-06-18T15:00:59Z

Lots of commit activity here that looks unrelated to the code review. Can you convert to a draft PR and/or switch your debugging to another branch and use workflow_dispatch instead of pull_request triggers to test workflows?

Bump. @araravik-psd every time you push a commit, it sends a github notification, posting on Discord and sending emails to repository watchers. If these commits are not relevant for the code review, please batch them or push them to a different branch.

Create fetch_job_status.py

207b957

Creating a fetch_job_status.py python script to obtain Github actions job status output in the Workflow.

araravik-psd requested review from ScottTodd, marbre, amd-chrissosa and geomin12 June 13, 2025 15:19

github-project-automation bot added this to TheRock Triage Jun 13, 2025

github-project-automation bot moved this to TODO in TheRock Triage Jun 13, 2025

Fixing Pre-commit errors

a81266a

geomin12 reviewed Jun 13, 2025

View reviewed changes

Addressed the PR comments in fetch_job_status.py

1257c5d

ScottTodd reviewed Jun 13, 2025

View reviewed changes

araravik-psd added 17 commits June 16, 2025 10:43

Job_summary dict key update in github fetch_job_status.py

f428abd

Added json copy to upload_build_logs_to_s3.py

53b6d3a

Local workflow changes for test

24dda2a

Local changes 1

0747576

fixing typo upload_build_logs_to_s3.py

9f8aa56

test local changes1

6e9ad93

Local test1

8acaa78

Local test2

c8f3697

local_test3

c531e66

local_test4

98f662e

local test

7afce46

local_test5

8154d14

local_test6

35944b2

local_test7

c1a2f98

Commenting return for log files upload_build_logs_to_s3.py

3879640

local_update build_linux_packages.yml

cb8033b

Update copy location build_linux_packages.yml

68d29d3

araravik-psd added 4 commits June 16, 2025 15:59

Update src job status file location in upload_build_logs_to_s3.py

ca375c0

fixed typo build_linux_packages.yml

a675264

Typo update upload_build_logs_to_s3.py

bf02993

local changes

ee21dc0

araravik-psd added 11 commits June 16, 2025 16:27

added job_status_dir var build_linux_packages.yml

392135d

Added job_status_dir to the script

5a55be8

fixing typo

35b29e9

modified attribute names

9ca5d4e

Update upload_build_logs_to_s3.py

be0b41b

Restoring build_linux_packages.yml for full run

14fa238

print github.run_number

ed07953

change to json.load()

9551a5f

local test workflow run

18937ed

removing json loads/dump

0e9d6a3

Reducing run time for test

9a452c2

araravik-psd added 7 commits June 18, 2025 11:01

fixed typo

7f31b09

Create convert_dict_to_json.py

56e0903

added output_file convert_dict_to_json.py

578b22e

Update build_linux_packages.yml

c6dea8d

Create write_json_to_file.py

faa11d4

Update build_linux_packages.yml

2034e7a

Update build_linux_packages.yml

8c81dff

araravik-psd marked this pull request as draft June 18, 2025 18:18

	github_workflow_jobs_url = f"https://api.github.com/repos/RoCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"
	github_workflow_jobs_url = f"https://api.github.com/repos/ROCm/TheRock/actions/runs/{RUN_ID}/attempts/1/jobs"

Create fetch_job_status.py #833

Are you sure you want to change the base?

Create fetch_job_status.py #833

Uh oh!

Conversation

araravik-psd commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geomin12 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geomin12 commented Jun 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ScottTodd commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottTodd commented Jun 18, 2025

Uh oh!

Uh oh!

araravik-psd commented Jun 13, 2025 •

edited

Loading

ScottTodd commented Jun 16, 2025 •

edited

Loading