Skip to content

Conversation

win5923
Copy link
Collaborator

@win5923 win5923 commented Sep 8, 2025

Why are these changes needed?

Since #3923 being merged, metrics related to deleted CRs are also removed.

Currently, if users want to see historical information for metrics such as:

  • kuberay_cluster_provisioned_duration_seconds

Using the instant query option, we can only see metrics that are still present in the /metrics endpoint. If a target scrape or rule evaluation no longer returns a sample for a previously existing time series, that time series is marked as stale.
螢幕擷取畫面 2025-09-08 212504

Changed to the range option so we can see the kuberay_cluster_provisioned_duration_seconds metric of deleted RayCluster.
image

I only updated the RayCluster Provisioned Duration table.

Related issue number

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@win5923 win5923 marked this pull request as ready for review September 8, 2025 12:05
@win5923 win5923 marked this pull request as draft September 8, 2025 12:25
@win5923 win5923 force-pushed the Grafana/time-series branch from b269462 to a297fe2 Compare September 8, 2025 13:30
@win5923 win5923 changed the title [Grafana] Replace table panel with time series panel [Grafana] Use Range option instead of instant Sep 8, 2025
@win5923 win5923 force-pushed the Grafana/time-series branch from a297fe2 to bc4bddd Compare September 8, 2025 13:31
@win5923
Copy link
Collaborator Author

win5923 commented Sep 8, 2025

Not working with the RayJob Execution Duration panel. so i didn't change this panel.
image

image

Maybe we can switch the table into a time series.
image

@win5923 win5923 marked this pull request as ready for review September 8, 2025 13:48
@seanlaii
Copy link
Contributor

seanlaii commented Sep 9, 2025

I like the idea of using the Range query, so we can see all the data points across the period instead of losing the data when clusters are deleted. This is very useful for monitoring and historical analysis.

@win5923 win5923 changed the title [Grafana] Use Range option instead of instant [Grafana] Use Range option instead of instant for RayCluster Provisioned Duration panel Sep 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants