Kyuubi engine automatically exits after running for a period of time

[conorzhao]

[conorzhao]

From the logs, it seems that AuthZ may cause kyuubi engine to crash

[pan3793]

Please provide more logs with text format

[conorzhao]

ERROR SparkSOLEngine:RECEIVED SIGNAL 15:TERM
ERROR ApplicationMaster:RECEIVED SIGNAL TERM
INFO SparkRangerAdminPlugin$:clean up ranger plugin,appId:sparkSql 
INFO PolicyRefresher:PolicyRefresher(serviceName=hdp hive).run():interrupted!Exiting thread
java.lang.InterruptedException 
  at java.util.concurrent.locks.AbstractQueuedsynchronizer$Conditionobject.reportInterruptAfterWait(AbstractQueuedsynchronizer.java:2014)
  at java.util.concurrent.locks.AbstractQueuedsynchronizer$conditionobject.await(AbstractQueuedsynchronizer.java:2048)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:207ROR
INFO RangerTagEnricher$RangerTagRefresher:RangerTagRefresher(RangerTagRefresher(serviceName=hdp hive)-49290).run():Interrupted!Exiting thread
java.lang.InterruptedException
  at java.util.concurrent.locks.AbstractQueuedsynchronizer$Conditionobject.reportInterruptAfterWait(AbstractQueuedsynchronizer.java:2014)
  at java.util.concurrent.locks.AbstractQueuedsynchronizer$Conditionobject.await(AbstractQueuedsynchronizer.java:2048)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
  at org.apache.ranger.plugin.contextenricher.RangerTagEnricher$RangerTagRefresher.run(RangerTagEnricher.]ava:886)

[user-GitHub-user]

hi,I had the same problem in 1.7.0, any suggestions?

23/12/21 09:37:01 ERROR SparkSQLEngine: RECEIVED SIGNAL 15: TERM
23/12/21 09:37:01 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
23/12/21 09:37:01 INFO SparkRangerAdminPlugin$: clean up ranger plugin, appId: sparkSql
23/12/21 09:37:01 INFO PolicyRefresher: PolicyRefresher(serviceName=metastore).run(): interrupted! Exiting thread
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:160)
23/12/21 09:37:01 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
23/12/21 09:37:01 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
23/12/21 09:37:01 WARN ZookeeperDiscoveryClient: This Kyuubi instance cqdev01:41229 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/12/21 09:37:01 INFO EngineServiceDiscovery: 2 connection(s) are active, delay shutdown
23/12/21 09:37:01 INFO CuratorFrameworkImpl: backgroundOperationsLoop exiting
23/12/21 09:37:01 INFO ZooKeeper: Session: 0x200000ea4be5384 closed
23/12/21 09:37:01 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped.
23/12/21 09:37:01 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
23/12/21 09:37:01 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
23/12/21 09:37:01 INFO SparkSQLEngine: Service: [SparkSQLBackendService] is stopping.
23/12/21 09:37:01 INFO SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping.
23/12/21 09:37:01 INFO SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping.
23/12/21 09:37:01 INFO SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped.
23/12/21 09:37:01 INFO SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped.
23/12/21 09:37:01 INFO SparkSQLBackendService: Service[SparkSQLBackendService] is stopped.
23/12/21 09:37:01 INFO SparkSQLEngine: Service[SparkSQLEngine] is stopped.
23/12/21 09:37:01 INFO ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)
23/12/21 09:37:01 INFO DiskBlockManager: Shutdown hook called
23/12/21 09:37:01 INFO ShutdownHookManager: Shutdown hook called
23/12/21 09:37:01 INFO ShutdownHookManager: Deleting directory /data/module/hadoop-3.1.4/yarn/local/usercache/hadoop/appcache/application_1699966942074_0696/spark-1fe5ccf6-89f6-474c-ba42-5b49dc6e206f
23/12/21 09:37:01 INFO ShutdownHookManager: Deleting directory /data/module/hadoop-3.1.4/yarn/local/usercache/hadoop/appcache/application_1699966942074_0696/spark-1fe5ccf6-89f6-474c-ba42-5b49dc6e206f/userFiles-35f8d57d-613f-460b-ae6f-31ccd380fa7d
23/12/21 09:37:01 INFO AuditProviderFactory: ==> JVMShutdownHook.run()
23/12/21 09:37:01 INFO AuditProviderFactory: JVMShutdownHook: Signalling async audit cleanup to start.
23/12/21 09:37:01 INFO AuditProviderFactory: RangerAsyncAuditCleanup: Starting cleanup
23/12/21 09:37:01 INFO AuditProviderFactory: JVMShutdownHook: Waiting up to 30 seconds for audit cleanup to finish.
23/12/21 09:37:01 INFO AuditAsyncQueue: Stop called. name=sparkSql.async
23/12/21 09:37:01 INFO AuditAsyncQueue: Interrupting consumerThread. name=sparkSql.async, consumer=sparkSql.async.batch
23/12/21 09:37:01 INFO AuditProviderFactory: RangerAsyncAuditCleanup: Done cleanup
23/12/21 09:37:01 INFO AuditProviderFactory: RangerAsyncAuditCleanup: Waiting to audit cleanup start signal
23/12/21 09:37:01 INFO AuditProviderFactory: JVMShutdownHook: Audit cleanup finished after 0 milli seconds
23/12/21 09:37:01 INFO AuditProviderFactory: JVMShutdownHook: Interrupting ranger async audit cleanup thread
23/12/21 09:37:01 INFO AuditProviderFactory: <== JVMShutdownHook.run()
23/12/21 09:37:01 INFO AuditProviderFactory: RangerAsyncAuditCleanup: Interrupted while waiting for audit startCleanup signal!  Exiting the thread...
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.apache.ranger.audit.provider.AuditProviderFactory$RangerAsyncAuditCleanup.run(AuditProviderFactory.java:495)
	at java.lang.Thread.run(Thread.java:748)
23/12/21 09:37:01 INFO AuditAsyncQueue: Caught exception in consumer thread. Shutdown might be in progress
23/12/21 09:37:01 INFO AuditAsyncQueue: Exiting polling loop. name=sparkSql.async
23/12/21 09:37:01 INFO AuditAsyncQueue: Calling to stop consumer. name=sparkSql.async, consumer.name=sparkSql.async.batch
23/12/21 09:37:01 INFO AuditBatchQueue: Stop called. name=sparkSql.async.batch
23/12/21 09:37:01 INFO AuditBatchQueue: Interrupting consumerThread. name=sparkSql.async.batch, consumer=sparkSql.async.batch.log4j
23/12/21 09:37:01 INFO AuditAsyncQueue: Exiting consumerThread.run() method. name=sparkSql.async
23/12/21 09:37:01 INFO AuditBatchQueue: Caught exception in consumer thread. Shutdown might be in progress
23/12/21 09:37:01 INFO AuditBatchQueue: Exiting consumerThread. Queue=sparkSql.async.batch, dest=sparkSql.async.batch.log4j
23/12/21 09:37:01 INFO AuditBatchQueue: Calling to stop consumer. name=sparkSql.async.batch, consumer.name=sparkSql.async.batch.log4j
23/12/21 09:37:01 INFO BaseAuditHandler: Audit Status Log: name=sparkSql.async.batch.log4j, interval=01:05.757 minutes, events=108, succcessCount=108, totalEvents=154, totalSuccessCount=154
23/12/21 09:37:01 INFO BaseAuditHandler: Audit Status Log: name=sparkSql.async.batch, finalDestination=sparkSql.async.batch.log4j, interval=29.756 seconds, events=5, succcessCount=2, totalEvents=2806, totalSuccessCount=154
23/12/21 09:37:01 INFO AuditBatchQueue: Exiting consumerThread.run() method. name=sparkSql.async.batch
23/12/21 09:37:01 INFO SparkUI: Stopped Spark web UI at http://cqdev01:32833
23/12/21 09:37:01 INFO YarnClusterSchedulerBackend: Shutting down all executors
23/12/21 09:37:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down

[pan3793]

your first line of logs indicates that the Spark engine was killed from the external

SparkSQLEngine: RECEIVED SIGNAL 15: TERM

[user-GitHub-user]

thinks, but I'm not sure if the development killed the process，I'll watch it for a while