I am looking at resintalling my OS (currently Mint) which I have been running for a few years.
Mint is great and I may end up going back; but was thinking of trying something new.
I want to be able to play with local models and agents; I have a pretty decent system.
- AMD 3700X
- 64 GB DDR4
- AMD RX9070 16GB GPU
- 2TB NVME SSD
- A few TB of HDD’s
What are some good options for easily running models and agents; I have been using ollama to run models; but I’m sure things have moved on since I begun playing with things a year ago.
Give me a few tips; I am going to have a bit of time on my hands over the next few months.
How is switching OS related to local models? Are there any OS that is more efficient? Is there some software that serves certain OS better? I’m curious because I’m also using Mint, have been for about a year now after ditching Windows and I was actually exploring potential benefits of other distros for the first time earlier this week. I’d love to learn if there is something to keep in mind, or is it related to your AMD hardware?
Maybe also consider moving away from Ollama:
I agree with the sentiment in general, ollama sucks and is problematic, but the obstacle that stands in the way of that is reflected pretty much right here:
llama-swap handles multi-model orchestration, loading, unloading, and hot-swapping models on demand behind a single API endpoint. Pair it with LiteLLM and you get a unified OpenAI-compatible proxy that routes across multiple backends with proper model aliasing.
The article is just vaguely asserting people can go ahead and install 3 different applications and then figure out how to configure them to talk to each other and do all the things you want it to do seamlessly, without ever even attempting to explain how to actually do that, just to get what ollama does in 1 installer.
That’s one alternative they describe. They also mention other all-in-one alternatives. Ollama is not the only one of its kind.
Yeah, pick your poison. And mind this article is a few months old. I’m not sure where we’re at as of today, but plain llama.cpp has multi model routing built in as well. Not sure if all of those extra steps are still necessary to do.
Ignoring the other comments around complexity. If anybody is considering running LiteLLM in their lab. I strongly recommend looking at GOModel instead. LiteLLM was nothing but bugs, slow page loads, and security issues for me. GOModel just worked and was simple.
Interesting; what do you suggest to run in its place?
I started with ollama then moved to llama.cpp directly and just use it through llama-server and my own custom code now. The lack of ability to use mmproj for community models (meaning things like vision processing in uncensored qwen did not work at all!) plus the fact that I had to edit a modelfile every time I want to change temperature pushed me over the edge to figure out the rough edges around llama.cpp. There have been a few bumps in the road, but it’s been worth doing overall.
The blog article recommends you go straight to the source and use llama.cpp and it’s llama-server. And they mention a few other options as well. I did that for a while, but I’m mostly using KoboldCPP. Guess it depends on what people are trying to accomplish, but it’s packed with features, fairly easy to set up and operate and does images and audio as well. (But then I don’t use AI all that much, so maybe don’t take advice from me.)
Just stay with mint. It’s fine.
deleted by creator
I used a mix of Ubuntu, Fedora, and Windows for a decade or so but recently switched over to CachyOS and it’s pretty nice, installing / updating graphics drivers is easy as and having access to the AUR is pretty neat.
Switching to a new distro would also be a good time to explore better alternatives to ollama too, plain llama.cpp is decent (got it running on my laptop rtx 3060) and Strata is quite neat for supported MoE models - runs a bit faster than llama.cpp on my oculink 9060 xt + 64gb ddr4.
No suggestions, but keep us updated on your experience with ROCm. I’d love to migrate over to AMD GPUs for local model training, but the architecture just doesn’t seem to have the toolkit support I need to feel secure about such a decision.
I have been running models on it for a year…seems to be getting better and better; I never ran on Nvidia so I can’t compare.
I’m pretty heavily invested into training image and video generation models, so my use cases might be more particular. I recently did a huge deepdive and decided I should just stick to CUDA for the time being, given the specific toolkits and software I leverage in my process.
A lot of the software targets Ubuntu compatibility so that would be the easy distro choice. But, you should be able to get this stuff running on any/most distros as long as you’re comfortable building/compiling the software as needed. Or using containers like Docker, etc. I doubt you’d have any issue doing this with Mint if that’s your preference.
I’m just tinkering with this stuff so other people here will have way more experience. But as a point of reference I have no trouble running llama.cpp / sd.cpp on current Debian Trixie (with/without an updated Linux Kernel). This is with an AMD R9700 (32 GB VRAM) in an older Intel based system. Everyone says that running with Vulkan is faster but I’ve found it’s good to test your own setup - My overall inference speed is actually faster using ROCm. ROCm gives me less tokens per second but for whatever reason the RAM-VRAM operations are much faster when using ROCm which leads to the overall time from prompt to result being shorter. There’s likely more tweaking I could try to get more speed but I’m not heavily using AI at the moment so am happy with how it works now.
Before that had the same setup with a old Nvidia gaming GPU (6 GB VRAM), it sort of worked but was slow as hell and I could only get it to run with Vulkan. That GPU supported an older CUDA version (7.5 I think) and I never figured out how to get that working with llama.cpp + Nvidia software + Debian.
I would say that the best os/distro is the one you’re comfortable with. With that said, I would really iron out why you want to switch, like what you’re looking for. Stability? Debian. Newest features asap? Arch. Mix? Fedora. General support? Probably any of them, but fedora or Debian if you want properly supported packages and not rely on the AUR. (You can use any of the derivatives, but I can’t say they do anything special for me, so I stick to the core distros)
