[SPARK-52934][PYTHON] Allow yielding scalar values with Arrow-optimized Python UDTF

[ueshin]

What changes were proposed in this pull request?

Allows yielding scalar values with Arrow-optimized Python UDTF.

Also the error class for the case where the number of columns is different from the return schema is fixed.

Why are the changes needed?

There is a behavior difference in Arrow-optimized Python UDTF between legacy and new serialization.

• legacy allows yielding scalar values
• new one doesn't

@udtf(returnType="a: int")
class TestUDTF:
    def eval(self, a: int):
        yield a

The behavior should be consistent.

Does this PR introduce any user-facing change?

Yes, the new code path allows yielding scalar values.

How was this patch tested?

Updated the related tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[allisonwang-db]

Nice!

[ueshin]

Thanks! merging to master.