Make exceptions visible to the developer and potentially terminate the GRCPServer

[Jerry-Carter]

Improve the handling of exceptions in GRPCServer such that fatal errors terminate the server and are reported to the developer.

Is your feature request related to a problem? Please describe it.

In my application, when GRPCServer.serve() is executed, an error occurs while setting up TLS in the transport layer causing an exception to be thrown. This exception is caught and rethrown by GRPCServer.serve() but it does not ever appear to be printed or otherwise made available to the developer; instead the exception appears to be silently swallowed. The GRPC server appears to be running fine when, in fact, it has failed.

Describe the solution you'd like

Fatal errors should cause the server to terminate, gracefully when possible. The developer should be informed about the exception to facilitate debugging and, in production, application monitoring. Silent failures are very bad.

Describe alternatives you've considered

If necessary, a Logger could be passed to GRPCServer.serve, but I presume there are good reasons for not doing so or it would already be the case.

Additional context

In my application, the TransportServices flavor of HTTP2ServerTransport is being used, but the error handling appears to be more general and within GRPCCore. Additionally, I'm using the ServiceLifetime run() code from grpc-swift-extras.

And, finally, this is my first grpc-swift 2.x server so it is possible that an RTFM answer is in order. Should that be the case, I apologize in advance.

[glbrntt]

In my application, when GRPCServer.serve() is executed, an error occurs while setting up TLS in the transport layer causing an exception to be thrown. This exception is caught and rethrown by GRPCServer.serve() but it does not ever appear to be printed or otherwise made available to the developer; instead the exception appears to be silently swallowed. The GRPC server appears to be running fine when, in fact, it has failed.

That's surprising. If GRPCServer.serve() throws then that error should be propagated into your program and handled there.

Are you sure the error is actually thrown from serve()?

I know the http2 transports do swallow some errors (certainly at the stream layer) so my hunch is that actually it's being incorrectly swallowed in the transport rather than in GRPCServer.

Fatal errors should cause the server to terminate, gracefully when possible. The developer should be informed about the exception to facilitate debugging and, in production, application monitoring. Silent failures are very bad.

I agree wholeheartedly that silent errors are usually very bad!

If necessary, a Logger could be passed to GRPCServer.serve, but I presume there are good reasons for not doing so or it would already be the case.

The reason is that there's no one-true-logger in Swift: there's swift-log and OSLog. This is an issue we want to address though. I think in all likelihood we'll end up with a different abstraction and offer an integration with swift-log and OSLog, possibly behind a package trait.

[Jerry-Carter]

As always, glbrntt, thank you for your insight.

Bypassing for a second the question of whether this report belongs against grcp-swift, let me summarize the results of my investigation. Your instincts are good and the exception from the transport layer is indeed reaching GRPCServer.serve(). Unfortunately, the error description is lost when the exception is rethrown: "Server transport threw an error." This is suboptimal but I can appreciate that there are higher immediate priorities. I can file a new issue against grpc-swift-2 for potential resolution at a future date.

Here's where things get interesting and where my ignorance shows. Once the error is rethrown, the exception propagates to the run() function from grpc-swift-extras (Sources/GRPCServiceLifecycle/GRPCServer+Service.swift) and then into ServiceGroup from swift-service-lifecycle where the exception appears to vanish.

        do {
            try await serviceGroup.run()
        } catch {
            print("EXCEPTION: \(error.localizedDescription)")
        }

The catch is never triggered and my server code happily proceeds as if nothing is wrong. Is the problem here with grpc-swift-extras as I thought earlier in the week, with swift-service-lifecycle, or (perhaps most likely) with how I'm using ServiceLifecycle? I don't currently know.

In any event, I propose closing this issue against grpc-swift. I can then write a narrowly focused issue against grpc-swift-2 about rethrowing transport exceptions. And I can continue to investigate.

[glbrntt]

Unfortunately, the error description is lost when the exception is rethrown: "Server transport threw an error." This is suboptimal but I can appreciate that there are higher immediate priorities. I can file a new issue against grpc-swift-2 for potential resolution at a future date.

The error should include a description of the underlying error too. See here and here.

Here's where things get interesting and where my ignorance shows. Once the error is rethrown, the exception propagates to the run() function from grpc-swift-extras (Sources/GRPCServiceLifecycle/GRPCServer+Service.swift) and then into ServiceGroup from swift-service-lifecycle where the exception appears to vanish.

        do {
            try await serviceGroup.run()
        } catch {
            print("EXCEPTION: \(error.localizedDescription)")
        }

The catch is never triggered and my server code happily proceeds as if nothing is wrong. Is the problem here with grpc-swift-extras as I thought earlier in the week, with swift-service-lifecycle, or (perhaps most likely) with how I'm using ServiceLifecycle? I don't currently know.

I'm a bit surprised that the error isn't thrown by the run() from the service group. Can you reproduce this? (I tried with a basic example but it did throw the error from the service.)