Cache Invalidation of Result Storage Objects

[schwarmco]

Hi there!

I'm using Thumbor in front of a Minio instance. The Loader fetches images from a Bucket, and the processed images are stored via Result Storage in this bucket on a /cache folder. To generate these, i am using secure URLs with the HMAC key to protect the requests.

The Result Storage has a generous TTL of one month since, in most cases, the resulting images are essentially "permanent." However, there are scenarios where I delete or modify the source image in Minio. In these cases, I'd like to invalidate the corresponding images in the Result Storage so they can be regenerated based on the new or updated source.

Currently, my (non-scalable) solution is to iterate through all the Result Storage objects (i.e., /cache//) and manually check for matches based on the image path as i do not store all the generated URLs of the images' variations. While this works, it's inefficient and becomes cumbersome as the number of images grows.

A more scalable and elegant approach would be to leverage object tags. By tagging the Result Storage objects, we can easily identify and invalidate specific cached images. There are two potential solutions I see:

Use the same tags as the source image: This would allow us to align the Result Storage object with its source image's metadata.
Preferred option: Assign a specific tag, such as thumbor-aws-result: , to each Result Storage object. This way, when the source image is updated or deleted, it becomes straightforward to locate and invalidate the corresponding cache objects.

It could also be the case that i am completely missing a better approach or solution to this - if so, please point me in the right direction!

Best,
Joachim

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 5 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.

[schwarmco]

i know, this is more of a conceptual question - if you could just let me know if this is something to consider or if i should search for another/indvidual solution @heynemann @guilhermef

[guilhermef]

@schwarmco Since the Loader and the ResultStorage can be used independently, your second approach seems more feasible.

Do you mind creating a PR to add a tag to the ResultStorage objects?

Just be mindful with the character limit in some object storages.