In PagedAttention, KV Cache can only be reused within one single round of inference. While in multi-round sessions, next round chat can be regarded as a concat of prompts and output from previous round.
Updated on: Posted on:
In PagedAttention, KV Cache can only be reused within one single round of inference. While in multi-round sessions, next round chat can be regarded as a concat of prompts and output from previous round.