[SwiftLexicalLookup] Unqualified lookup caching

[MAJKFL]

This PR introduces optional caching support to SwiftLexicalLookup. In order to use it, clients can pass an instance of LookupCache as a parameter to the lookup function.

LookupCache keeps track of cache member hits. In order to prevent the cache from taking too much memory, clients can call the LookupCache.evictEntriesWithoutHit function to remove members without a hit and reset the hit property for the remaining members. Calling this function every time after lookup effectively maintains one path from a leaf to the root of the scope tree in cache.

Clients can also optionally set the drop value:

/// Creates a new unqualified lookup cache.
/// `drop` parameter specifies how many eviction calls will be
/// ignored before evicting not-hit members of the cache.
///
/// Example cache eviction sequences (s - skip, e - evict):
/// - `drop = 0` - `e -> e -> e -> e -> e -> ...`
/// - `drop = 1` - `s -> e -> s -> s -> e -> ...`
/// - `drop = 3` - `s -> s -> s -> e -> s -> ...`
///
/// - Note: `drop = 0` effectively maintains exactly one path of cached results to
/// the root in the cache (assuming we evict cache members after each lookup in a sequence of lookups).
/// Higher the `drop` value, more such paths can potentially be stored in the cache at any given moment.
/// Because of that, a higher `drop` value also translates to a higher number of cache-hits,
/// but it might not directly translate to better performance. Because of a larger memory footprint,
/// memory accesses could take longer, slowing down the eviction process. That's why the `drop` value
/// could be fine-tuned to maximize the performance given file size,
/// number of lookups, and amount of available memory.
public init(drop: Int = 0) {
  self.dropMod = drop + 1
}

[MAJKFL]

swiftlang/swift#81209

@swift-ci Please test

[ahoppen]

Without diving too deeply into the details: I am a little concerned about the cache eviction behavior and the fact that you need to manually call evictEntriesWithoutHit (which incidentally doesn’t seem to be called in this PR or swiftlang/swift#81209) and I think it’s easy for clients to forget to call it. Does this more complex cache eviction policy provide significant benefits over a simple LRU cache that keeps, say 100, cache entries? We could share the LRUCache type that we currently have in SwiftCompilerPluginMessageHandling for that. Curious to hear your opinion.

[ahoppen]

I’m not a fan of the drop naming here. I don’t have a better suggestion yet, maybe I’ll come up with one.

[MAJKFL]

Yes, I agree it is a bit ambiguous. What about skip?

[MAJKFL]

Without diving too deeply into the details: I am a little concerned about the cache eviction behavior and the fact that you need to manually call evictEntriesWithoutHit (which incidentally doesn’t seem to be called in this PR or swiftlang/swift#81209) and I think it’s easy for clients to forget to call it.

Hi Alex, thank you for the suggestions and sorry for the late reply. I got quite busy with school. Thank you for pointing out evictEntriesWithoutHit is not called in the other PR. Originally, I called the method inside SyntaxProtocol.lookup after performing lookup, but ended up passing eviction to the client for extra flexibility. I must’ve forgotten to put it there. I think there’s enough evidence that it was a bad idea :).

Does this more complex cache eviction policy provide significant benefits over a simple LRU cache that keeps, say 100, cache entries? We could share the LRUCache type that we currently have in SwiftCompilerPluginMessageHandling for that. Curious to hear your opinion.

The current implementation assumes subsequent lookups happen in close proximity to the previous lookup. Like e.g. in the compiler in a single top-bottom scan (best case). The algorithm follows the intuition that for any (close) subsequent lookup, we shouldn’t recompute more than one scope. In top-bottom scan and maintaining one path to the root, we always have a guaranteed cache hit in the first common ancestor. I think a sufficiently big LRU cache would have a similar behavior, but it would require more memory than this approach and not provide additional speedup. I’ve also noticed that growing the cache too big leads to diminishing returns. I suppose it could happen because less of the data structure can remain cached in memory.

I attach below a sketch I used when pitching the idea to @DougGregor that visualizes an optimal top-bottom scan. In each step, blue represents contents of the cache, red represents evicted entries and green arrows point at the lookup position.

I think SwiftLexicalLookup could still benefit from an LRU cache though. The current implementation lacks an ability to arbitrarily lookup previously evaluated names without reevaluating a great part of the syntax tree below. What if we still used the optimal and small cache from the current implementation for subsequent lookups and maintain a large LRU cache for symbols/leaves that would fill up alongside it? This way we would have the best out of two worlds without blowing up the size of LRU with intermediate scope results. What do you think about this idea?

[ahoppen]

Would it be possible to use an LRU cache and provide an eviction method that can be called to clean up the cache as we know that some parts of it are no longer relevant (what you described in the sketch above). That way we would get reasonable out-of-the-box behavior and don’t have an ever-growing cache but also have the ability to keep the cache size low in cases where the client (here the compiler) cares about it and knows the access patterns.

[MAJKFL]

That way we would get reasonable out-of-the-box behavior and don’t have an ever-growing cache but also have the ability to keep the cache size low in cases where the client (here the compiler) cares about it and knows the access patterns.

Ah yes, that’s a very good idea to have an upper bound for the size of the cache. I haven’t thought about it. I’ll try to look into how to extend LRUCache from SwiftCompilerPluginMessageHandling with the cleanup algorithm then. Should we hoist LRUCache to some other, shared place, or should it remain in SwiftCompilerPluginMessageHandling?

[ahoppen]

Should we hoist LRUCache to some other, shared place, or should it remain in SwiftCompilerPluginMessageHandling?

We should hoist it up. We could put it into a new module or just stick it in the SwiftSyntax target at the package access level – I haven’t quite decided on that yet but I think it’s something that we could also change easily once the rest of the PR has taken shape.

[MAJKFL]

swiftlang/swift#81209

@swift-ci Please test

[MAJKFL]

swiftlang/swift#81209

@swift-ci Please test Windows Platform

[ahoppen]

Thanks for addressing my review comments. I just had a chance to look at the PR again and left a few comments inline.

[rintaro]

As Alex suggests, I'm not sure this addition is needed.

But even if we want this API, I don't think this Set storage is necessary. This is essentially table.keys except Dictionary.Keys is not a Set. I think something like this should be fine.

package var keys: some Collection<Key> {
  table.keys
}

The caller can create a Set from this.

[MAJKFL]

Yes, I think this is a much better idea that avoids redundancy. Thanks!

[ahoppen]

One more comment about the automatic cache eviction, otherwise LGTM.

[ahoppen]

This doesn’t seem right to me. Once hits.count > capacity * 2, we pretend that there were no hits. Shouldn’t we be clearing ancestorResultsCache and sequentialResultsCache in that case?