Sparse attention fixes memory bandwidth, not memory capacity.
“The whole idea of HiCache is trying to offload KV cache from the HBM to a lower tier, higher capacity but lower bandwidth memory, and only load the most relevant set of tokens during computation.”
“Sparse attention
