Can Kyuubi Spark Engine show the output of python print statement in stdout?

[HaoYang670]

Currently when submitting a python operation to Spark, only the source code is shown, but not the stdout. For instance, submitting a one-line statement print(f"{1+1}") to Spark, and the driver logs will be like

25/06/24 07:02:53 INFO ExecutePython: Processing *'s query[9a817538-1138-4e7f-bdc6-1c6a55259b45]: PENDING_STATE -> RUNNING_STATE, statement:
print(f"{1+1}")
25/06/24 07:02:53 INFO ExecutePython: 
           Spark application name: kyuubi_GROUP_SPARK_SQL_group1_cluster-20250624070039-5rpybi2j_c3a2e761-0fff-4f65-9f53-a917e7311d57
                 application ID: spark-899ff5f6f41c42568f0bf963556846fb
                 application web UI: http://192.168.40.231:4040
                 master: k8s://https://kubernetes.default:443
                 deploy mode: cluster
                 version: 3.5.3
           Start time: 2025-06-24T07:01:01.368
           User: spring
25/06/24 07:02:53 INFO ExecutePython: Processing *'s query[9a817538-1138-4e7f-bdc6-1c6a55259b45]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.001 seconds

The output 2 is not shown here. How can I let 2 be shown? Does it need source code change in execute_python.py ?

[HaoYang670]

I replace sys_stdout with sys_stderr in https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/resources/python/execute_python.py#L534, but seems it doesn't work

[HaoYang670]

cc @cfmcgrady any insight?

[HaoYang670]

We want the print statement can be shown in Spark driver log for easier debugging. As the python operation works in batch way. Users cannot see the print output until the whole statement finishes, which is not friendly when the operation itself needs long time to run, for instance a for loop

[HaoYang670]

I see we have the result printed at https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/resources/python/execute_python.py#L534? But doesn't find anything get really printed in the spark driver

[cfmcgrady]

We want the print statement can be shown in Spark driver log for easier debugging.

If you want to print logs on the driver side, you can try applying the following patch.

def execute_request(content):
    try:
        code = content["code"]
    except KeyError:
        return execute_reply_internal_error(
            'Malformed message: content object missing "code"', sys.exc_info()
        )

    try:
        nodes = parse_code_into_nodes(code)
    except SyntaxError:
        exc_type, exc_value, tb = sys.exc_info()
        return execute_reply_error(exc_type, exc_value, None)

    result = None

    try:
        for node in nodes:
            result = node.execute()
    except UnknownMagic:
        exc_type, exc_value, tb = sys.exc_info()
        return execute_reply_error(exc_type, exc_value, None)
    except ExecutionError as e:
        return execute_reply_error(*e.exc_info)

    if result is None:
        result = {}

-   stdout = sys.stdout.getvalue()
-   stderr = sys.stderr.getvalue()
-
-   clearOutputs()

    output = result.pop("text/plain", "")

-   if stdout:
-       output += stdout
-
-   if stderr:
-       output += stderr

    output = output.rstrip()

    # Only add the output if it exists, or if there are no other mimetypes in the result.
    if output or not result:
        result["text/plain"] = output.rstrip()

    return execute_reply_ok(result)

[HaoYang670]

But this will change the result, right?

[HaoYang670]

Oh, I just find the stdout of python process is for communication with the java process? Does it mean any print in the python operation cannot be seen in the driver log? Maybe the only way to support this is to change the way of how java reads result from Python (maybe by socket)?