Help integrating Apache Kyuubi with Apache Ranger on Docker

[medina325]

Hello,

I'm working on integrating Apache Kyuubi with Apache Ranger to enable authorization and auditing capabilities for my data platform.

Here’s an overview of my current setup:

• Standalone Spark cluster using the Docker image bitnami/spark:3.5.5
  • 1 master node
  • 1 worker node
• Hive Metastore running in a PostgreSQL container
• Apache Ranger instance created using the ranger_in_docker script from the official Apache Ranger GitHub repo (branch ranger-2.5)
• Apache Kyuubi instance based on the official Docker image: apache/kyuubi:1.9.4-all

All containers are part of the same Docker overlay network and can communicate with each other.
Due to infrastructure constraints, the data is stored on a mounted volume instead of HDFS.

So far, I have:

• Successfully connected to Kyuubi using both DBeaver and dbt
• Verified that Kyuubi is correctly reading from the Hive Metastore

I now have a few questions that I’m hoping someone can help me with:

• How can I configure Kyuubi to use my external Spark cluster, instead of the default local Spark binaries located at /opt/kyuubi/externals/spark-3.5.2-bin-hadoop3/?
• Where should I place the authorization (authz) shaded JAR file? I followed the official guide here: Installing Ranger Spark Plugin, but the documentation isn't very clear on this point when using Docker.
• When I test the Hive connection in the Ranger UI, I get the following error:

org.apache.ranger.plugin.client.HadoopException: Unable to connect to Hive Thrift Server instance. Could not initialize class org.apache.hive.jdbc.Utils$JdbcConnectionParams.

Any idea what might be causing this? Could it be a missing dependency or configuration?

Any help or pointers would be greatly appreciated. I'm willing to share all the Dockerfiles and docker-compose file I've used.

Thank you in advance!