• 4 Posts
  • 107 Comments
Joined 3 years ago
cake
Cake day: September 6th, 2023

help-circle




  • Hardware: I have a custom setup with a bunch of amd graphics cards. I will do a writeup about that sometime this year.

    I use opencode exclusively. It’s probably not the best harness but it’s the one that gives users the most control and I like that.

    The actual models I used for this work are Qwen3.8 27b - FP8 and Qwen3.8 - Flash - Next UD_Q6_K.

    I wrote my own agents with duty separation. Flash is the orchestrator and long-horizon guide. While 27b is the faster, more expandable worker. This combination feels about as smart as an opus 4.8 with maybe not as great token efficiency. For simple tasks, i use the worker directly. I have a ton of repo documentation and my agents have access to read and write to Forgejo, both for git and api (issues and PRs respectively)

    Maybe I need to write an article about this…





  • what zener_diode said.

    when i was setting up this version of my homelab (this is v2), i had to contend with how to run docker. The general concensus is that the “best” way to run it is Proxmox>VM>Docker. Proxmox>LXC>Docker is supposed to be viable too. Proxmox>Docker is possible but is not recommend.

    I have 20 active docker services with about 30 containers running all sorts of services and they’re extremely reliable under this setup. The only time I’ve had problems is when I ran out of disk space (128gb) because stale docker images weren’t pruned.










  • I find llama.cpp with Vulkan EXTREMELY reliable. I can have it running for days at once without a problem. As far as tokens/sec that’s that’s a complicated question because it depends on model, quant, sepculative, kv quant, context length, and card distribution. Generally:

    Models’ typical speeds at deep context for agentic use. Simple chats will be faster

    Model Quant Prompt Processing (tok/s) Token Generation (tok/s) Hardware Quality
    Qwen 3.5 397B Q2_K_M 100-120 18-22 2 x 7900 + 4 x Mi50 ★★★★★
    Gemma4 31B or Qwen3.5 27B Q8_0 400-800 20-25 2 x 7900xtx ★★★★
    Qwen 3.6 35B Q5_K_M 1000-2500 60-100 2 x 7900xtx ★★★★
    Qwen 3.5 122B Q4_0 200-300 30-35 4 x MI50 ★★★★
    gpt-oss 120b mxfp4 (native) 500-800 50-60 3 x Mi50 ★★
    Nemotron 3 Nano 30B IQ3_K_XXS 2500-3000 150-180 1 x 7900xtx ★

  • I’ve used Gemma4-31B for agentic coding and its actually very good as far as local models go. Its less verbose than qwen3.5 so it ends up being faster too. Gemma4-26B can do agentic but its noticeably worse so you have to go slow with it. I haven’t had any coherence issues like other commenters mention but I’ve only been using higher quality quants from unsloth on llama.cpp



  • This is something I learned the hard way.

    Consumer hardware is limited by multiple factors when it comes to PCIe connectivity.

    • Physical layout. Easy how many slots you have to plu into, their size, and configuration.
    • Supported lanes from the CPU
    • chipset (motherboard) limitations

    Your graphics card might be a 16 lane card (referred to as “x16”), but sometimes, not all of them are used. Aforementioned 5060ti - I believe only uses x8. Some devices like graphics cards can use a physically smaller slot with an adapter for a loss in performance (a few frames in game play performance)

    Similarly, your motherboard might have a x16 slot and another x16 at the bottom. That second slot might only function as x8 or even x4. Does this matter? Sort of. Inta-card communication aka peer to peer communication can affect affect performance and that can compound with multiple cards.

    Even worse, some motherboards may have all sorts of connectivity but may have limitations like only 2 out of the bottom 4 slots, PCIe and m.2, can work at a time. ASK ME HOW I KNOW.

    Your CPU controls PCIe. It has a hard cap in how many PCIe devices it can handle and what speed. AMD tends to be better here.

    Enterprise gear suffers from none of this bs. Enterprise CPUs have a ton of PCIe lanes and enterprise motherboards usually match the physical size of their PCIe slots to their capacity and support full bifurcation*

    PCIe lanes are used up by and consumable by m.2, MCIO, and occulink to name a few. That means that you can connect a graphics card to either one is those of you can figure out the wires and power**

    • ** Bonus: bifurcation and how my $200 consumer motherboard runs 6 graphics cards.

    Bifurcation is a motherboard feature that lets you split PCIe capacity, so a 16x slot can support two x8 devices. My motherboard lets me do this on just the main slot and in a strange x8x4x4 configuration. I have an MCIO adapter (google it) which plugs into the PCIe and gives me 3 PCIe adapters with those corresponding speeds.

    it also has 2 m.2 slots which connect to the CPU. One is them, I use for a nvme ssd like a normal person. The other is an m.2 to PCIe adapter which gives me an x4 PCIe slot. For those keeping track, that’s 24 PCIe lanes so far. That’s the maximum my processor Intel 265k can handle

    But wait! The motherboard also has a kind of PCIe router and that thing can handle 8 more lanes! So I use the bottom 2 PCIe lanes on my motherboard for 2 cards at x4 each. The thing that kills me is that there are more m.2 ports. But the mobo will not be able to use any more than 2 devices at once. AND even though that bottom PCIe slot is sized at x16, electrically, its x4.

    Do your research (level1techs is great) and read the manuals to really understand this stuff before you buy

    My mobo for reference ASUS: TUF GAMING Z890-PRO WIFI