Python Dictionary to Class Handling (How/Why)

[slominskir]

Description

Is there any discussion on how to handle serde transformation from a Python Dictionary to a Class? The Consumer API for example currently accepts a callable function from_dict and is demonstrated in the examples.

This appears to preclude using a generator such as avro-gen, since it apparently simply wraps the Dictionary instead of transform it and toss it out as is done with from_dict. Maybe avro_gen wrapping is a bad approach? Maybe classes aren't "Pythonic" anyways? Just trying to figure out how best to use the confluent_kafka_python API in my own code and figure someone has already thought about this.

How to reproduce

This is an API / Documentation question.

Checklist

None

[mhowlett]

avro-gen was not on our radar (seems to be new). It may be a good approach, we haven't looked at it.

We need to re-visit all of this and get Python serdes in a better state. I plan on doing this, but have competing priorities. Feel free to leave any opinions you may have, we'll take note.

[slominskir]

I haven't formed an opinion yet, but digging a little I found some additional information to consider:

• https://gitlab.com/Jaumo/pyavro-gen
• https://github.com/gudjonragnar/fastavro-gen
• Any guidance on implementing serialization from/deserialization to custom classes? fastavro/fastavro#287

@scottbelden is probably more knowledgeable about this.

The advantages of a class over a dict include:

• documentation (both IDE and developers can more easily discover the API)
• type safety

The disadvantages may include:

• complexity
• performance

[scottbelden]

If I were to re-write fastavro from scratch I would probably have it use classes (or dataclasses) instead of regular dictionaries. It would solve some of the ambiguity problems that currently exist in fastavro and could theoretically allow for faster serialization/deserialization since the classes could be generated with custom, schema specific encoders/decoders.

However, that's all a large enough change that it would almost certainly have to be in a completely new version two library and so I don't imagine fastavro ever moving away from using dictionaries for things like records.

[slominskir]

So when does fasteravro (fastavro2) come out? 👍

[scottbelden]

Whenever someone decides to write it 😛