Local models are meaningfully useful* for agentic work now. This is a realization I came to over the last day or so.
I wrote a lot of custom mods for a game that recently switched engines. The migration is going rather smooth for most people because of wise architectural choices from the dev early on, and because they prioritized backwards compatibility in the update. But there are still some rough edges, especially if you had your hooks deep into the engine.
I recently vibeforked Unbiased Bob’s fork of Strata to make it compatible, easy to use and optimized for my OS and hardware (NixOS 5800x3d 64gb, 7900xtx 24gb). Sol 6.1 got it working with Qwen 3.8 Flash Next IQ3 XSS (~75GB) at roughly 46tps 164k ctx with enough headroom to launch a heavy browser along with a triple-A game set to low in 720p for testing when I need to (plus space for zram swap)
I set it up in deepseek harness, gave it a 14MB error log in the game’s working directory, then went to sleep. I woke up and it had run for 3 hours. The game launched with only two errors, which it quickly fixed from the new, vastly smaller logfile.
It hammered down every issue, making changes across 30 separate mod repos. Practically a one-shot, and all the features worked as originally intended.
I left a local coding agent running completely on its own for three hours across multiple repos across a dozen auto-compactions with a log file and no ability for it to verify its work besides a compiler. And it produced meaningful results. I can’t speak to the code quality, but it worked. This was completely unthinkable even just two years ago. Even one year ago, the cloud frontier wouldn’t be able to do this on its own, and now a local model can, running on my consumer gaming hardware from one or two gens back.
“Huh… So… We’re here now…" I said to myself, with a bit of stunned disbelief, wondering what the future one or two years from now might hold.
You must log in or register to comment.
I’ve been following r/decomps recently. Several times a day someone posts a new AI-generated decompile, seemingly by AI agents sent off on a mission without any guidance, effort, or proper setup. I expected the tools / harnesses to develop first before seeing results like this. It’s insane that models have these abilities while still being based on what is fundamentally just text generation.
Reminder that the transformer architecture, the thing all LLMs rely on, was originally built to translate between languages for the purpose of Google Translate.
And compilers are just translators, translating from a programming language (and/or an intermediary) into machine code. So the task of translating it back is very close to what LLMs specialize in.
So I always had it in the back of my mind that LLMs would be incredibly powerful for decompilation, I’m just surprised it took this long to be good enough to be a problem.
I had a similar experience this week. I tried ollama like last year and thought it was kinda cool but ultimately not that helpful.
Over the weekend I got Qwen3.6 35b running Hermes on my gaming PC and it made me a whole back testing application for trading. Everything from planning out the work, breaking it into tasks, spinning up agents for blocks of work, writing tests, the whole shebang.
It took like two work days of me prodding it once in awhile but it’s still impressive
Yeah, we’re here. I’m a professional programmer but I haven’t written an actual line of code in quite a while now, and I’ve been having so much fun with my local model (Qwen3.8-27B Q4) doing all sorts of cleanup tasks on my lifetime’s giant pile of accumulated files. For example, all my generated music - I ended up having Qwen build a GUI music manager and whipped up a website generator for my favourites so I can more easily share them through free hosts without having to hand them over to spotify or whatever.
I’m curious, is the game you mentioned modding Rimworld? I’ve been pondering modding it for some time now, it’s one of my favourite games, if you’ve got experience using AI for that I wouldn’t mind some pointers. It was always a “I know how to do this stuff but it takes way more time and attention than I’ve got available right now” sort of task but AI can help so much there. :) Your description of the “engine changed but compatibility is remarkably good” matches what I’ve seen of Rimworld’s recent 1.7 update.
Yes, it’s Rimworld, you’re right! But I don’t have many pointers for modding with AI because I’m a rimworld modder and have been since alpha. This was just a huge collection of random forks and one-off personal mods that I wanted to update for 1.7 for a personal run.
I have also been using frontier models to write game cheats for various games, replacing my miles of handwritten Lua code in cheat engine tables with native injection mods (easiest in unity and ue4, but I also modded an old dosbox game for a friend, and got a bunch of features for assassins creed black flag resynced on the anvil engine). I haven’t tried doing that with local models yet, but after seeing these results, I might. I suspect that’s code will be substantially worse, though…
As far as modding Rimworld, because most of its JIT C#/mono, and the game files have a lot of data in them already; it seems like models don’t need a lot of hand-holding. I bet downloading the wiki for some additional context would probably be wise, but as far as implementing simple changes, it seems to be perfectly capable with little to no prep. Also I’m using nix so it can get any tooling it needs and set up a dev environment on its own.
Okay, thanks. I was hoping it’d be relatively straightforward to get AI up to speed on it, I just wasn’t sure where to start or whether it’d be a waste of time if I tried. I just want to give my pawns some interesting diseases and it seemed like a daunting bit of setup to get to that point. I’ll take another run at it soon. :)
Rimworld’s an interesting battleground right now in terms of AI, the various fan forums came out strongly against AI and then the new BRUTAL engine came along that incorporates AI-generated code into the base game. So I didn’t think I’d get much help from the forums I’m aware of.
Most agents can read websites, you may not need to download it. Just tell it that that wiki has useful information. Maybe include it in the AGENTS.md file so it remembers.
I imagine my first task for the agent will be to tell it to compile a skill file from all the documentation I’m able to direct it to. I was just hoping that maybe someone had already done that to give it a leg up.
Maybe, skill files are usually for particular tasks though. I have one for code review, another for writing tests, etc… Documentation usually goes in a different file. It’ll depend on what agent you pick.
Yeah, the standards for all that are still new and shifting so I usually just leave it up to the AI itself to figure out how it wants to organize the data. It’s the one that knows where its harness tells it to find stuff, after all.
AI creates sounds, not music.
Define music.
I don’t argue with hacks or people defending using AI. Nor do I engage with whatabout bullshit.
Then why post your opinion in the first place?
Because I am allowed to. The rules don’t state that you can only ever speak if you’re willing to argue with kids about your point.
I made a DOS port of Tetris Attack for the first IBM PC. The audio hardware is a single buzzer. I created the engine for music and sound effects, by hand, from scratch… at 4.77 MHz. I had no idea what I was doing. I still don’t. The parts which aren’t that-sounds-right repetition are just turning column heights into notes.
Is that music? That slapdash bespoke effort, in the dire absence of any formal training, on toy hardware? Because if my bullshit counts then I’m not clear how any danceable formulaic realization of proper western music theory would not count… and if my bullshit doesn’t count then I’m not sure what the point is in puffing up human effort over perfectly fine artificial results.
No. lol! It’s not music. It’s computerized sounds. Did you write the music?? Score the different instruments? Wat it your original composition?
Creating music is a SKILL. A talent that someone sharpened over years of repetition, learning, practice, and love. So, no…. You didn’t make music. You programmed a computer to make sounds that OTHER people created.
I am the only human being responsible for the audio in that game. There are no other people, fool. Every beat and trill is me using the buzzer as an instrument. If using randomness to fill in some variety is disqualifying, you’re picking a fight with the entire medium of process music. Nobody wrote out each performance of “In C” beforehand, so it’s only Terry Riley and friends noodling around, yeah?
Do you think Bobby Prince wasn’t a musician, just because he worked in MIDI? He didn’t play an actual harpsichord on “Into Sandy’s City,” so nuh uh, doesn’t count. The computer did it! For some reason! Guess it was feeling spooky that morning.
Or do you simply think bad music is not music? No true Scotsman can make music unless they have skill! Straight up gatekeeping amateur effort. Can amateurs then play music? Does someone fumbling their way through a solo comprise some different process? Because if that’s where your philosophy lands, there’s a hilarious counterexample in James Hetfield working out “Master Of Puppets.”
You’re not a musician. Stop arguing.
I wrote music my-fucking-self in a sound engine I wrote my-fucking-self. Zero AI was involved, you illiterate troll. I built my own goddamn instrument to give my game a soundtrack, within some obscene technical constraints. Fuck your blinding prejudice against anything besides how you were taught.
LMAO… hack.
I can pick up a synthesizer which does nothing but produce prerecorded sounds and hit random things till it sounds good. If someone enjoys it they would call it music.
Slop generated music ain’t any different. Music and art as a whole is in the eye of the behold. Only a raging asshole tries to tell people what is or isn’t art to them.
It doesn’t have to be art to you. But no one else gives a fuck what you or I or anyone else thinks. Because only their own opinion matters when it comes to what they believe is art.
LMAO… the keyboard/synthesiser bullshit? Really?
No dude. Richard Wright of Pink Floyd was a classically trained pianist. He did not “hit random things till it sounded good”.
Ffs! What a fucking parhrtic argument!
No one hits random shit until it sounds good. That’s called noise. You clearly know absolutely nothing about music. Seriously…. Just go sit down.
Gooseworx hit random shit until it sounded good. She’s famously self-taught. Her soundtrack for The Amazing Digital Circus is fantastic.
What do you think classical training is, besides an effort to explain and share which noises sound good, in a particular context? You’d tell us the first humans to bang rocks together to form a beat weren’t really making music, because those cavemen hadn’t invented the art degree.
I’m not arguing this with you. AI is not music and those that use it are not musicians. You’re a hack
Music you know well was made from a preset sample. The new akais break copyrighted music into separate stems using generative AI. Illegal by pink pantheress has like a billion plays. That’s like three notes and it DOES sound like she just pressed a few. Listen to it. I’m not a musician but I’m married to one and know a lot. None of them are trained.
LMAO… You have no fucking idea how keyboards and synths used to work, do you?
Lean shit. Them come back to this and delete your embarrassing shit.
Alright, I used it to organize my AI-generated sounds.
Feel better? Swapping those words had exactly zero impact on what was actually done. The labels you use don’t make any practical observable difference.
It make the world of difference to people that actually create music. Imagine you spent your lifetime learning and perfecting a craft, then along comes a kid with a 3d printer and calls himself a craftsman.
Typically shit like this is said by people that don’t make art or if they do dont sell it.
And since you’ve got beef with 3d printers, too. You jelly?
Lmao… typically shit like this told by kids that think they’re musicians because they punched a few adjectives into a prompt. I play three instruments and have been on stage up and down both coasts of the US.
Talk about jelly some more.
But that’s not what he said. He didn’t call himself a musician. He said he created music. He did. It’s undeniably music.
A person 3d printing does not call themselves a craftsman. They say they created objects. And they undeniably create objects.
I’m sorry you’re struggling with the AI stuff. As a developer, I’ve gone through stages of grief too. I can’t deny reality tho, this is my livelihood. It’s a tool. I use it.
Right, the vaguely defensible ‘that’s not art!’ breaks down as we move away from that broad abstract term. People have looked at paintings made by actual human beings and asked: ‘but is it art?’ You cannot see a Github repository with Claude as an author and scoff, ‘that’s not code!’ For starters: the compiler disagrees.
An AI render is not a drawing, because nobody drew it, and it’s not a photo, because it didn’t happen, but anyone insisting it’s not an image is playing stupid word games. Like the pixels don’t look like anything to them. Saying a sequence of notes in western tuning and 4/4 time with a verse-chorus-verse structure is just “noises” is at least that silly.
More generally, nobody worth listening to can look at a page of printed words and say, ‘that’s not a text.’ Everything is a fucking text.
It’s not music. And it’s not stages of grief. You like it because it does your job for you. I despise it because it allows talentless hacks to pretend they are my equal when I spent my life on this craft. They’re ruining culture. It’s slop.
End of story.
I suggest therapy as it’s not going to stop.
Cool story. I suggest you get used to being called a hack. Because it’s not going to stop.
I despise it because it allows talentless hacks to pretend they are my equal when I spent my life on this craft
It really has been all downhill since the year 2000 when Hampton the Hampster’s “The HampsterDance Song” was released, this was PEAK music
Now just about anyone can release some garbage and it’ll go viral eh?
Jfc you hacks are insufferable with all this stupidity you think are points.
Imagining. In such a situation I’d resist the urge to join a 3d printing forum and interject myself into their conversations with nothing worthwhile to add aside from my “um, actually” semantic quibbling. Because I’m not an ass and if the kid’s having a good time more power to him, it’s no skin off my back.
But you do you I guess.
Yeah, however, imagine that those kids are ruining your industry. But let me guess. You’d be totally fine with that, right?
You see, It’s hacks like… it’s hacks- that pretend to know what they’re doing by punching adjectives into a prompt and producing artificial music, or artificial art that are taking jobs away from talented people that actually work hard to produce their product.
And for the record…. I’m not here to get internet points kid, as a musician of stage and studio for over 30 years….
I’m here to say what needs to be said.
It’s the AI that is “ruining” your industry and not the fucking record labels and Spotify who has made it a broke bitch career to be a musician. Sure.
The record labels aren’t pretending to be musicians.
Yeah, however, imagine that those kids are ruining your industry. But let me guess. You’d be totally fine with that, right?
In fact, I am.
I’m a professional programmer. 20 years experience, it’s where I earn my money, got a university degree and everything. And we’re here in a thread about how not only are AI models good at programming, even small local models are good at programming. I use them myself, I give advice on how to use them, I ask advice on how to use them. I think they’re really neat and look forward to using them more, and encourage everyone else to also use them.
Sure, some people have no idea what they’re doing and they end up writing not very good software. My reaction to that is to try to explain to them what they did wrong and how they can get better. Or, if their not-very-good software is still perfectly good for whatever they’re using it for, I give them a little thumbs up and say “good on you, just make sure you know what you’ve got there.”
I’m here to say what needs to be said.
No, you’re here to interject yourself into a conversation that was not at all about AI music to push your personal dislike of AI music and to insult people who don’t share your view of AI music.
It doesn’t need to be said. Certainly not here and now.
Okaybuddy. Not reading all that. AI is for hacks. End of discussion.
Bonus post: a subagent went off the rails in a subsequent (much harder: reverse engineering) task.
(First and last time I’ve seen that behavior, but I’ve only been running it for a few days)
Hmm.
Hmm.
Ok, let me think clearly.
Does Qwen still do all the lengthy reasoning steps? Last time I tried to vibe-code some moderately easy webapp, it went through something like 1 - 1.5 million tokens. Which should be about 9h with your 46t/s?
Use the “Qwen Swift” fine-tune; it drastically shortens the reasoning without harming the final outputs. Its authors were very intentional about only cutting the unnecessary parts of the long reasoning.
No, I didn’t have that experience. I know that Qwen is known for long reasoning traces, and it certainly does think and even investigate a little bit more than it needs to, but it wasn’t anything crazy (like I’ve seen with the 27b)
Off of memory, the first run was about (not counting cache aggregates) 1.3M in and about 250K out. I’ve since done a bunch of enhancements and changes using that session, so now it’s up to 17 turns, 1544 steps: 901 minutes of LLM time, 130 minutes of tool time. 7.8M in and 1.2M out (115M aggregate if you count cached io). Dropped to about 41tps. But it got work done.
This is absolutely orders of magnitude more than something like sol 6.1 would use, but it isn’t completely outlandish and it can actually perform work; which is what was surprising to me. It still hallucinates a ton too, but it seems to be able to correct itself and ground itself in new information from toolcalls etc much better than models I’ve tried previously, which I suspect is what unlocked this “long running” capability. It also seemed to respond well to steering (most models I’ve tried previously just shit the bed)
strata/qwen flash next is a big boost to internal knowledge that “explores less clueless” reasoning paths. Size does matter.
Sounds acceptable. I mean as long as you don’t have to intervene all that much. And you said electricity is affordable where you live… So all good.
Yeah, I think hallucinating isn’t all that bad in reasoning steps or in-between. As long as it manages to then home in on the correct thing.
Electricity is affordable anywhere there is space for a few PV modules. At least, as long as the sun shines.
I feel that could be a bit of an issue with letting the agent run overnight 😆
But generally, yes. If someone can afford a graphics card, I suppose they could afford some 450W (peak) solar modules at like 100€ as well.
Sadly it’s the cold season now. With lots of overcast or rainy days and the sun angle isn’t ideal either. Lots of atmosphere absorbing the light, and days are shorter.
I paid 82 EUR for a 500 Wp bifacial when picking up four of them about a week ago. A 2 kW Hoymiles microinverter was 202 EUR, 10 m plug in cable 45 EUR. Flat roof mount 145 EUR. I will need to buy some 36x concrete plate ballast for about 25 EUR. Some 4x 3 m and 4x 1 m DC4 module cables, cheap. Yes, winter months are thin, but my meter is old enough to run backwards. When they exchange it, I will buy a couple 10 kWh battery.
I intend to run a local LLM during daytime anyway.
Lucky you! They replaced my good old electricity meter right when I was willing to buy some solar panels. And now I keep postponing it because it’s not that attractive anymore. The street is the wrong way, so all I can easily do is place them eastwards, with the roof casting shadows at a bit after noon. And not sure about the cost of paying someone to bolt them to the top of the roof… Guess I could do some east/west configuration there.
Ornith is a nice fine-tune of Qwen that works reasonably well on reasonable hardware, Bonsai also seems to be often recommended but I haven’t tried it personally. That said, from everything I’ve seen local agentic code is pretty computationally wasteful in many cases, it iterates heavily and confuses itself a lot and self-corrects often repeatedly. It hurts my soul to imagine the energy waste and environmentally damage if large numbers of people started using this routinely and carelessly.
It is meaningfully useful, I’ll grant you that, and when paired with highly specific and well-bounded tasks it performs quite well. I hate the idea of relying on it too heavily though. If you just throw it at things blind and without you also providing assistance to the model, it will get the job done, eventually, just in a really painful and stupid way. It’s not Claude, but for a locally-running substitute it’s remarkably good, it handles coding assistance very effectively, but it’s a very poor coder when operating completely independently. I find the best and most efficient use is to ask it for suggestions, keeping the pieces as small as reasonably possible, and then implement them myself based on what it suggests (either adjusting the model’s edits or copy-pasting the model’s suggestions). It’s very good at locating the relevant code and it usually suggests a reasonable design, which I typically tweak based on my own experience. Maybe that makes me old-school, maybe I’m a luddite, but that’s what my experience tells me to do. If you don’t have any experience, I guess you’re on your own, good luck with the models. They’re decent, but they’re not brilliant programmers (granted neither am I most days). That’s my opinion and I’m sticking with it until I see evidence to suggest otherwise.
It doesn’t run my GPU any harder than watching anime with my realtime upscalers and my CPU not much more playing lategame modded Minecraft. And I live in a region with very cheap energy with lots of renewables and nuclear. So if it can do work, I would otherwise have to do, for fractions of pennies while I sleep: it’s totally worth it.
It takes electricity. I take moments of a finite life. This is the same reason I didn’t review the code; for this case, I just wanted the result, I just wanted it to work, and I was curious how far a local AI would get. I didn’t actually expect it to finish. I expected to come back to it looping or a complete mess and revert all my repos and then tell Sol to do it in 30 minutes.
They are far less token efficient, and dumber, and far slower than Frontier models running on dedicated hardware. It hallucinates and get things wrong all the time, but it seems to be able to correct itself and get back on track from new information much better than earlier generations of model. That really seems to be the key to its long-running success from what I’ve seen. When I tried this about half a year ago, once it started hallucinating or its context got over ~50k, it was basically game over.
My background is in low-level resource-constrained systems programming, mostly C and modern C++. Yeah, I can code laps around this thing even in languages I don’t know well (not true for frontier) and I’d never trust it to write code that I would own and put my name to… But it can get results. It could do something low-stakes for me, while I slept, that was literally impossible a year ago. And it does it locally. That was mindblowing. I thought this approach (LLMs) would plateau a lot sooner than this. The fact that it can run outside of a datacenter is just a massive cherry on top.
This is the same reason I didn’t review the code;
As long as you don’t publish it. Because then you (should) also have to take responsibility.
Well, I think a caveat to that is publishing source (no prebuilt release) as unvetted slop, so everybody knows. Though in this case, it was strictly for personal use.
But yes. Anything I release proper, for general consumption, is owned by me. And professionally, I never own even a line of code (or comment) that I don’t understand or wouldn’t have written myself. I may ask for a bunch of audits and slop tests for insights, but I’ll keep those offline as “extra” unless it’s something on par with I would have written. I’ll never hallucinate important code, and anything i get paid for or contribute to OSS is important. Personal game mods and small utilities, not so much.
Good human.
Given that (personally affordable, ingnoring the bubble premium) hardware progress is saturating, how much headspace for improvement do we still have do you think? Would you currently put it at linear-ish, saturating, or too early to tell/not enough data?
the bubble premium
You know, I can absolutely see a future where Nvidia and the other hardware makers turn their eyes back towards consumers as they see the build-out slowing and the upgrade budgets shrinking.
What if in one year, Nvidia puts out a GPU or all in one PC AI server thing that will let you run free models that are as good or better than current frontier ones, at a reasonable speed, and they charge like $3000 or $5000 for the thing?
Well, I guess if they want to be proper capitalists/enshittifiers, they would first sell them for 10x as much to corporate IT departments who can replace their ridiculous tech giant subscriptions with local compute augmentation. On-prem is back, baby!
Then they can rip off the gamers and the home lab enthusiasts for a while.
And finally, at long last, with the economy raked over the coals and a cool trillion in the bank, Jensen will briefly feel whole inside. Surely he will.
Jen-Hsun has certainly sold a lot of golden shovels, but he will find gamer gear market to be slim pickings in comparison to the fat enterprise checks. What I’m hoping is that some older enterprise gear will make it to ebay and such. Maybe some smaller AMD Instincts with HBM.
It’s really hard to tell because I didn’t even see it getting to this point. I’m running a 75 gigabyte model on a 24 gigabyte GPU and my friend with a 16 gigabyte GPU is actually getting better performance than me. Mostly just because it’s a mixture of experts and Strata’s storage tier caching works well.
I don’t think it’s saturated because we would have to, not quite saturate, but definitely wring out a lot of the gains from kernel and model architecture first before raw compute becomes a limitation.
I expected this approach to AI to plateau before now. There has also been evidence that scale is not the sole key, and that quality and approach and harness all play significant roles. I thought it would have died by now, but since it hasn’t, I suspect it still has a long way to go. Given that I couldn’t have predicted this a year ago, I don’t think I can predict a year from now.
Thanks, food for thought.
The gap between big-boy models and local fare has been closing. That could mean scale is not necessary for further improvement, or it could mean we’re asymptotically approaching some limit to this approach, but either way - existing consumer hardware is sufficient for some witchcraft.
Waiting for the bubble to pop to join the fray. Adding some surplus PV capacity meanwhile.
They’re also oretty great for stuff that’s simple to write like self-hosting infrastructure/config code like Ansible, SaltStack. I wouldn’t use what comes out without understanding it though.
I use NixOS and my agents seem to love it as much as I do. The ability to read the derivations, evaluate the configs, navigate the nix store, and use nixpkgs from the cmdl is quite awesome for host & environment tinkering/debugging.
Models seem to prefer nix and arch a lot. Don’t really know why. Arch I get but nix always confuses me.
How is this set up to run? What model? IDE? etc
Details all in the post. Modified Strata backend, qwen 3.8 flash next iq3, deepseek harness. But this isn’t how-to guide
How is DeepSeek vs open code I always see people say DeepSeek is either the best or worst harness by a mile.
Haven’t used opencode yet so can’t compare. Dsh is pretty nice and customizable, and its compaction seems to work well. Qwen doesn’t seem to like keeping the todo up to date and can’t easily differentiate harness vs user instructions. Not sure which is at fault.
