On the bar chart I included what I think are the most popular models right now (not objectively, based on my anecdotal observations, link to include/exclude what you want).
What do you think? Is it fair? There was, and to some digree still is, a bit of controversy about AA score indexes after they updated their benchmarks to V5.0. I am mainly interested in your thoughts about QFN in relation to other models.
I don’t know what they are testing but I use QFN for everything.
Assuming it doesn’t have to rely on internal knowledge too much by giving it tools to retrieve information, it does everything I ask it to.
What sort of setup are you rocking? Which quant? I’m running Qwen 3.6 35B A3B Q2 for speed on my 16GB T4060 Ti and Qwen 3.8 27b Q5 on my R9700 32GB for coding. Unsure if I can run a half decent QFN quant on my setup along with my 128GB of sys RAM.
I’m running Gufo on a Strix Halo PC with 128GB shared memory.
The quant is unsloth’s UD-Q4_K_XL. Speed is really good with this configuration (1600 t/s prefill and 100 t/s decode). I wanted to try some higher quants as well but I’m waiting for Qwen 4 for now to do more testing.
You should be able to get QFN running quite well on your system as well when offloading unused experts to RAM.
Amazing, will give that a shot. Thanks!
Gemini 3.8 does not match its benchmarks.
It forgets system instructions, spends too much time thinking, and has this weird bug with reasoning token reprocessing that totally throws off the model.
Mistral Large should be higher up, it’s far more reliable than its benchmarks shows.
What is this bug you described? First time hearing about it… Mistral seems weirdly low, it’s a very big model, maybe they didn’t finetune it for benchmarks?
Gemini encrypts their thinking traces, so if you inject a thought in an early thinking trace it always gets added after user context on any later message in the conversation.
That means if the thought trace disagrees with a user message it goes with the original thought trace.
E.g imagine you start a conversation with the date, then come back the next day. The thinking trace will make the model always think it’s the same date, no matter how much you argue.
They were working on fixing this, so not sure if the new models today fixes that.
Interesting, maybe caused by design focused on overprotection against distillation of the thinking process, so “it’s a feature, not a bug” mentality. I don’t use gemini models so I can’t confirm by myself, thanks for pointing that out!
That is part of it.
The thinking trace is a base64 encoded proton if with an encrypted string inside
