So Matt Pocock says models typically are smartest in the first 140k tokens. Hes using the big boy cloud models tho. Does that hold true for 27b-35b range? Or is it smaller and how much?
What would the ‘smart range’ be for tiny models that fit in 2gb? Would be it be like a sentence or two? I have an old 1050 was thinking of throwing in rig but im not sure what id even be able to do with such a small model
I believe Pocock only recommends going up to 100k now, last I heard. But for a lot of simpler tasks 10-20k will be plenty if you don’t have a super long prompt + documentation for it to parse first. Your inputs are really what decide how much context you need.
There’s scientific papers about that kind of degradation. Like this one from earlier this year.
In short, from what I read: No, I don’t think it holds true. It’d be weird if it was some fixed number like 150k. If at all, I’d expect it to be a percentage of the maximum context length of a specific model. And it probably also depends on the exact model and how the attention and the parameters are set up.
There are thousands of resources for this information
https://meattacker.github.io/local-ai-model-recommender/best-local-ai-models-4gb-vram.html
Wow amazing stuff! Goes to show the gulf of knowledge between a noob just getting into local llama and you. Ty! I dont even know what to search for lol
It depends on the model. For newer stuff, and Qwen3.8 27b specifically, its Qwen Sparse Attention/Gated DeltaNet architecture does much better than a lot of previous models.
The old rule of thumb was that the middle of the context window can get lost in old models, but Qwen3.8 27b and Qwen 3.8 Flash Next work fine, even when using most of the 262k native tokens of context… but this long context requires 8GiB of VRAM just for the context, and this is with FP8 KV quantization.
I still don’t trust much more than 128k though, and most harnesses automatically compact if a request would exceed a certain threshold (e.g., OpenCode is the higher of 15% of max or 16k tokens for its buffer).
For smaller models 16-32k is probably fine unless you’re using a model that is extremely old, like llama3 or phi 2.5. But you’re going to struggle to get anything other than a simple chatbot with very short conversation on a 4GB card. Maybe light translation or document work, but don’t expect anything useful in coding or agentic work.