Trino Gateway module QueryCountBasedRouterProvider not working properly

[Dens1002]

Recently I've implemented the aforementioned load balancing module in my team's trino gateway instance, having set all of the parameters according to the documentation: I've set the backendStateSecret, with a user and password that have access to our DB, monitorType is set to JDBC, taskDelaySeconds to 10 and of course the module itself is listed under the matching configuration. Now, the pod does run, but the only log I get is "ActiveClusterMonitor is getting stats for all active clusters" over and over again. And when I try to connect to the gateway through a python script I get the error: "RouterException: did not find any cluster for the adhoc routing group". We do have an adhoc group defined in our routing rules, and I did check that there is an yaml file within the pod with our routing rules, but the gateway seems to not be able to access it somehow. We've been having a bit of a hard debugging this issue, I don't know whether the problem lies in the fact that the routing rules are being ignored or if the connection details provided in the backendStateSecret are not being used. Any help or clarity is appreciated!

[andythsu]

did not find any cluster for the adhoc routing group meaning the Trino clusters failed the healthchecks performed by Trino-Gateway and Trino-Gateway has internally treated the clusters as unhealthy. Some steps to troubleshoot:

• Do you have any clusters in adhoc routing group?
  • You can navigate to the "cluster" tab in your Trino-Gateway UI and see if there are any clusters in "adhoc" routingGroup
• Are the clusters healthy and deemed healthy by Trino-Gateway?
  • You can add a log line log.info("clusters health: %s", stats); in https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ActiveClusterMonitor.java#L81

[Dens1002]

Yes, we do have a cluster in the adhoc routing group, and at any rate all of our clusters are healthy

[andythsu]

in that case you can add a log line in

trino-gateway/gateway-ha/src/main/java/io/trino/gateway/ha/router/QueryCountBasedRouter.java

Line 240 in d654094

clusterStats = stats.stream().map(a -> new LocalStats(a)).collect(Collectors.toList());

to verify the ClusterStats are correct. Specifically, you want to check

• the cluster indeed belongs to adhoc routing group
• the cluster is healthy