--moe-cache-mib doesn’t seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.
Im actually more excited about the model manager here. To be able to load/unload/switch models easily is something I had scripts do, but now its built-in.
Model manager already exist for quite a while, you can load and unload models via webui. I personally uses preset file option. This is new version of the existing mgr
Ah, i must have missed it. Thanks for the info 🙂
Hmm, not seeing it directly in the llama.cpp 0.6.0 version. Where were you seeing it?
Should be added sometime early this year. I think it was mentioned in HF blog/article when i was looking for way to use multiple models without restarting llama
Just use llama-swap
FWIW, vLLM already had a lot of this. This makes it so you can run decent models on smaller devices like phones or Raspberry Pis.
Getting pretty good results with llama.cpp and Bonsai on laptops and older phones.
