This branch adds an experimental, block-granular KV cache streaming path to the CUDA llama-server. It is intended for running long contexts when model weights leave too little VRAM for the complete KV cache.
With --kv-stream-stage-mib N, the authoritative KV tensors are stored in pinned host memory while a bounded CUDA pool is shared by resident KV pages and a transfer ring. The runtime adapts that split as the context grows: it keeps as many pages resident as the budget allows, reclaims resident space for staging when more streaming is required, and prefetches later layers while the current layer computes. This avoids relying on uncontrolled Unified Memory page thrashing and preserves exact attention over the full context.
(Not my project)
I just built and tested the above llama.cpp fork and I can attest that it really works. My previous “best Qwen 3.8 27B” on my 5060Ti 16GB and host DDR5 RAM was UD-IQ4_XS beellama with KVarN KV cache 5/4 where I got 112000 in context at usable speeds (~900 PP, ~25 TG).
With this fork I’m now running the same model, KV cache 8.0/4.0 at 131072 context size at the same speed.
Is that the feature freetoken was boasting about?
Also, cuda only? Why?No I think this is completely novel actually, and afaik it’s CUDA only because that’s what the author have themselves.
I’ve been using it since I compiled it. I’d go so far as to claim this is a game changer for 12 and 16GB cards.
Was wondering because its cuda only, and just like freetokens is game changer for lower memory setups.
I vaguely remember some kv cache stuff in llama that could make performance similar to freetoken, but forgot what it was.
Was there a pr open for this? ( though i cant use it, no cuda )
