Explore where `pydantic` might help with UX

[fselmo]

Explore the idea of using pydantic dependency to better validate request object types and their params. This could help with field validation and error messages. There may be some UX improvements that are worth exploring there.

[e3243eric]

I think this idea is great for runtime type checking. The objects need to inherit from BaseModel of pydantic to be parsed by pydantic. Does this mean we need to update all the classes in types.py?

[XiaoHuiHui233]

I'm very much looking forward to this feature. In my work, I encapsulate a bunch of pydantic Models outside of web3.py to represent some on-chain data structures. This is extremely friendly to the serialization. It can check various forms of data structure input and serialize them into different outputs in different situations. If web3.py can provide this capability, it will greatly reduce the workload in serialization, which is very effective for encapsulating high-level web interfaces and other tasks.

But actually there are some problems here. In my work, I use orjson as json serialization tool. But it was developed in Rust and converted to Python with pyo3. In orjson, integer data exceeding 64 bits cannot be processed, but most integer types in web3.py (such as Wei) will definitely exceed this limit. This is not a big problem in the json standard library (although it does not conform to the json specification). And pydantic uses the json standard library by default, this cannot be counted as a problem with pydantic. But the json standard library is notoriously slow, so I still think it's going to be a problem.

Also, pydantic V2 is expected. V2 provides more powerful capabilities that can save a lot of development costs and redundant code. It will be more efficient due to also using Rust for refactoring. Since Rust's json library is used (rather than relying on other Python json libraries), it is expected to solve the json problem just mentioned. However, there is no release version of V2 yet, and it is still in the testing process, and some fatal bugs will be found in actual use. In its documentation, the expectation for the official release of V2 is 2023Q1. The discussion on this aspect may have to wait for the official release of V2.

[fselmo]

Nice to see some discussion on this already!

@e3243eric I opened this as more of an exploratory "research" issue to track pros, cons, possible improvements, etc. There's not a real void to be filled with a PR quite yet but happy to see the interest coming in for it!

@XiaoHuiHui233, thanks for getting the conversation started and providing some solid input here. Very much appreciated.

Looks like some benchmarking could be useful. It might be nice to benchmark on a subset of calls / types to compare against the current state of things. Yeah, looks like V2 release is slated for Q1 of 2023 - will be good to keep an eye out for it. I don't quite have the bandwidth to dive in here at the moment but being as this would likely be part of the next major version of web3.py (v7), I think this is a good time to start the conversation and see how it evolves until then.

[DefiDebauchery]

That's funny, I was literally thinking about this earlier but was worried about pushback on adding another dependency.

Would the idea be that this replaces AttributeDicts into structured request and response classes based on submitted RPCEndpoint?

[johnny-emp]

I've built a library that takes an approach similar to FastAPI, where the library is built on top of Pydantic and all types are BaseModels. https://github.com/empyrealapp/eth-packages is the repo if you want to check it out.

I've found using Pydantic has a lot of benefits, especially in indexing contract events and creating typed function bindings. I think if the team finds this as a P4 priority, I'm happy to offer the work I've been doing as a friendly alternative to web3.py that addresses some of these requirements. I'd also like to share some of my findings if there is interested in integrating Pydantic.

The biggest benefits I found was utilizing Generics for contract functions to remove the need for contract ABIs, and having typed Events for indexing so it's much less error prone when you're building a larger indexing project. For example, I had much better success with my indexing of UniswapV2/V3 using my library because I was able to catch most of my bugs using mypy.

[DefiDebauchery]

Another potential suggestion is msgspec. Their Structs seem to be similar to Pydantic Models, as far as defining structured dataclasses, and appears to be more performant.

Edit: Apparently one reason it's more performant is that it doesn't perform validation on object creation, only on export. That may violate the intent of this issue's goal, but I leave it here just in case.