I’m curious as to which tools and technologies you all are using to keep track of all those services you are deploying, whether it be resource tracking, network traffic, logs, traces, or uptime.
As a bonus question, how have you organized your network or your services to reduce the overhead of implementing observability?
Prometheus, grafana, loki
Everything else is just those tools but you’re abstracted away from some level of control.
Or ELK.
I’ve got grafana and Prometheus, but I’m currently working on a generic log pulling script that will then feed to a local ollama instance to summarize the logs and let me know if there are any potential issues I should be aware of. Grafana is great for current state, but I’m a big believer in watching the logs to see trends, but I’m too lazy to do it by hand.
Once that’s up and running and steady, I’ll probably tie it into ntfy to actually alert me in real time
Loki?
Cool, sounds like a fun project.
Not sure if you have been looking into Jev or other System 1 models, but it might be a good fit since it can output a structured type-safe output. Most of the automation I’ve tried in the past with LLMs failed due to the output structure being hallucinated (ignoring my requested structure). But it sounds like System 1 models might fix that.
I’ve been looking at an open source model called Laya a little and it might work well for your project. Sounds like you’d want structured output.
What kind of rig do you have ollama running on?
Jev and ollaya are on my list to eventually add. I’m running ollama through an RTX 3080 with 10 gigs of RAM. Just because it’s what I had lying around. I was mostly using it for gaming but it works for local model use. I’m also planning on installing Hermes in a bit once the log manager project is done, but that’s going to take some careful tinkering because I’m technically underpowered for it. I’m probably just going to have some gate checks up to make sure nothing loads during a gaming session.
My main goal is to completely get off the frontier models if possible because I have concerns about the environmental impacts, especially. As a shameless pirate I mind intellectual theft less, but I really do disagree with the environmental impact, especially when a optimized local model plus a bit of scripting and filtering. Does the job 90% of the time
But we’ll see, I’m still tweaking everything, but I think I’m about 90% done with the log normalizer to do most of the heavy lifting act on the logs programmatically, to keep the context window for my local models as optimized as possible.
It’s definitely been a good fun learning project
Oh, one of the first things that I implemented was a hallucination checker. If it cites any data that’s not actively in one of the corpus I’ve supplied, it automatically gets flagged and filtered out. But most of what I do can be handled with internal documentation and material I explicitly provide.
Bezsel, simple and easy, constantly having new features added.
Beszel looks really convenient.
If you’re wanting to go hard in the paint here, Grafana has a great OSS stack with Grafana, Loki, Mimir, Tempo, and Alloy.
It’s curios if you used Prometheus as well
Can confirm it’s a great stack! We run it at my work and its quite nice to work with. I’ve been thinking of replacing Prometheus in my homelab with Alloy
It looks really compelling. Really flexible with deep options. Seems like I could spend 10 years building the dashboard alone!
I stood up grafana at work, and I make dashboards to keep my role secure.
Everyone uses my shitty little dashboards :)
Systemd
To add, journal-brief can e-mail you systemd logs. If your apps aren’t very noisy, this can be helpful without a need to set up a complex observability stack.
I use Prometheus with various exporters pushed to Grafana which I have displayed on a small screen running on a ROCK Pi in my office
I used to run dockprom but it was pretty ressource intensive and I have since moved to Beszel + Uptime Kuma.
I have yet to find a way to keep logs between container re-creation (I unsuccessfully tried to set up Loki back when I ran dockprom), both Beszel and Komodo only display logs from the container currently running in the stack.
Uptime Kuma if I want something to alert me (still trying to decide between ntfy & gotify)
Home Assistant is also checking some stuff
Watchdog to reboot my Raspberry Pi Zeros when they fall off the wifi
Smokeping for a general, long term feel of the network, which might answer your bonus question?
Smokeping? In 2026? Jesus…
What’s not to like?
Still alive & still maintained, and less resources than Grafana & Prometheus…
Portainer, beszel, and Whats up docker
I recently switched to Dockhand from Portainer and I like it a lot more, it’s much cleaner and straightforward. It also auto-updates so you might be able to ditch whats up docker too.
No auto updates thanks. 😅 I’ve just reset all my compose files to pin versions today after an unintended mariadb update cost me hours to repair. I’ll look into dockhand though. Was unimpressed at portainer’s announcement that new version wont have a community edition. Portainer is really just an ‘at a glance’ view for me since i try to do all the compose and troubleshooting stuff in terminal. Wud will probably become a thing i just review every few weeks to make sure i dont get too adrift from latest.
I ought to look into a unified dashboard as i dont have easy visibility or things like disk usage, sheduled backup success etc that id like to see alongside the container status and beszel activity graphs.
Im still quite new to it all though.
Me too lol I thought What’s Up Docker was for auto updates. What does it do?
For me, its just a board that knows which imagea i am running and tells me if there is a newer one so that i know how out of date i am 😂
Observawhat? My services just exist, if they fail and nobody complains… Well, they get removed. If the fail and somebody complains… The get restarted…
Pretty efficient but simple approach.
I set up homedex alongside portainer and dozzle
