Skip to content

[NPUW]Implement prefix caching. #31669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open

[NPUW]Implement prefix caching. #31669

wants to merge 35 commits into from
+1,328 −155

Conversation

intelgaoxiong
Copy link
Contributor

@intelgaoxiong intelgaoxiong commented Aug 11, 2025 •
edited
Loading

Details:

This PR implements prefix caching in NPUW.
In existing inference engines, the KV cache of a request is discarded after processing is completed, preventing the KV cache from being reused across multiple calls and significantly slowing down the execution.
https://arxiv.org/pdf/ 2312.07104 proposed a new technique to reuse KV cache automatically across multiple generation calls.

Tickets:

@github-actions github-actions bot added category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin labels Aug 11, 2025
@intelgaoxiong intelgaoxiong force-pushed the xiong/prefix_caching branch 4 times, most recently from a7a59db to 48f060e Compare August 13, 2025 07:03
@intelgaoxiong
Copy link
Contributor Author

build_jenkins

@intelgaoxiong intelgaoxiong marked this pull request as ready for review August 13, 2025 09:11
@intelgaoxiong intelgaoxiong requested review from a team as code owners August 13, 2025 09:11
@intelgaoxiong intelgaoxiong requested a review from dmatveev August 13, 2025 09:12
@intelgaoxiong intelgaoxiong force-pushed the xiong/prefix_caching branch 5 times, most recently from 719eac4 to c9ee5f6 Compare August 22, 2025 02:08
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
smirnov-alexey
smirnov-alexey reviewed Aug 25, 2025
View reviewed changes
@intelgaoxiong
Clang format.
24053c6 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Optimize algo: decrease hash calculation.
e0fc886 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Optimize algo: Create output to input name map.
af25f0c 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Add functions to restore and store cache.
ffa5b11 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Refine classes and functions.
20e49fd 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Refine cache print.
6623247 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Add option to toggle prefix caching.
32e202b 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Calculate and print block memory size.
a2b8a63 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Modify KVBlock to a class type.
7485837 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Fixed cache eviction.
64302d1 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Add get_block_unsafe.
0dcf30e 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Use uint64_t for hash.
36748c2 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Add options for block size and max block number.
a470090 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Variable rename in class.
d8971af 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Bug fix.
9ae0852 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Fixed cache eviction.
71e5f5a 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Efficient cache eviction.
d12c8e1 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
More fix for cache eviction.
6081bc2 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Add unit test and bug fix.
319885b 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Make private variable.
be09f72 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Disable cache print.
3b6b2fc 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Fixed for CI check.
e7c1903 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Fixed some accuracy issues.
4facf02 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Solved review comments.
02080d6 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Register properties for prefix caching.
5cc8a13 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Refine log.
c58281d 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Fixed typo.
4bfd0e2 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Changed to LOG_VERB and const methods.
b4be394 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Decouple functions with LLMInferRequest and add function description.
18877de 
Signed-off-by: intelgaoxiong <[email protected]>
@intelgaoxiong
Remove default cache size.
3259df6 
Signed-off-by: intelgaoxiong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AsyaPronina AsyaPronina AsyaPronina left review comments

@smirnov-alexey smirnov-alexey smirnov-alexey approved these changes

@dmatveev dmatveev dmatveev requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@dmatveev dmatveev

Labels
category: build OpenVINO cmake script / infra category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin
Projects
None yet
Milestone
2025.4
Development

Successfully merging this pull request may close these issues.
4 participants
@intelgaoxiong @smirnov-alexey @dmatveev @AsyaPronina