[EPIC] Iceberg Cache

[Xuanwo]

What's the feature are you trying to implement?

Cache is an essential component of an Iceberg table, and different types of cache are needed at various levels.

For example, for our table metadata, we will need a Manifest cache so that we don't have to read and deserialize the same manifest files repeatedly. For our Parquet files, we will need a FileMetadata cache to avoid parsing the metadata from the Parquet files each time. We could even implement a raw data cache to store portions of data files, eliminating the need to download them from S3 again.

As the foundation for various query engines, iceberg-rust should be designed to simplify integration while still allowing each engine to fully optimize performance. This applies whether they are using iceberg-rust on a single machine or within a distributed cluster.

I plan to add a set of cache APIs to meet all those needs. My current plan is:

• ObjectCache: an object cache trait that can hold objects like Manifest or FileMetadata
• BytesCache: a bytes cache that can hold row content of files, like table_metadata.json files.
• In FileIO Cache like opendal's CacheLayer, but the API is not decided yet.

Tasks

• ObjectCache
  • feat: Add trait for ObjectCache and ObjectCacheProvider #1222
  • feat: Implement ObjectCache for moka #1225
  • refactor: Adopt ObjectProvider in Table #1227
• BytesCache
• OpenDAL CacheLayer (TBD)

Willingness to contribute

I can contribute to this feature independently

[kyteware]

I'm aware that there is some progress being made on possibly restructuring FileIO, but I'd still like to take a go at writing an in-FileIO cache. I'll be happy to rebase my changes onto whatever new API is made!

[kyteware]

@Xuanwo I implemented a cache for data files in FileIO! I'd love for you to look at it and give feedback. #1430