The agent resets its WebSocket deadline to 70s, but the hub's default
collection interval is 60s, so a single slow collection cycle is enough to
trip the deadline and start a reconnect loop even though the hub is still
serving the agent. Raise the deadline to 120s and add a regression test that
keeps the slow-collection window from being lowered below two minutes.
Verified with go test -tags=testing ./agent (focused tests and the full agent
suite minus the container-only TestDirectoryIsWritable case), go vet, the
agent build and gofmt.
Closes#2294
Add a new "ContainerHealth" alert type that fires when a Docker container's
health check reports unhealthy, and resolves when it recovers. This mirrors
the existing Status (up/down) alert pattern: an alert can be armed per system
and honors the "min minutes" delay before firing.
When the alert fires, the notification (email and any configured webhook,
including Discord via shoutrrr) includes a log excerpt fetched live from the
agent for up to 2 of the unhealthy containers, prioritizing lines containing
"error" or "fatal" (falling back to the log tail if none match), capped to
keep the message well under Discord's size limit.
---------
Co-authored-by: hank <hank@henrygd.me>
Adds a user-configurable "Failed Services" alert that notifies when any
tracked systemd service enters the failed state, and again when all services
recover.
---------
Signed-off-by: Martin Stenröse <martin@stenrose.se>
Co-authored-by: henrygd <hank@henrygd.me>
- track pool capacity, health, I/O, scrub status, and vdev errors
- report dataset usage and correct ZFS filesystem metrics
- add pool charts, detail views, refresh controls, and health alerts
- persist pool details and include ZFS usage in disk alerts
- support configurable detail intervals and legacy agent compatibility
---------
Co-authored-by: hank <hank@henrygd.me>
refreshSystemDetails() takes NCPU and MemTotal from the Docker daemon's
/info response. That only describes this machine when the daemon shares its
kernel. On macOS and Windows Docker runs inside a Linux VM, so the system
details header shows the VM's memory as the host total, and the VM's CPU
count clamps both cores and threads through the lxc branch below it.
Only consult Docker's host info on platforms where the daemon runs natively.
Everything else already falls back to gopsutil, which reads this host.
SwapCached pages have been read back into memory but still occupy allocated swap slots. Subtracting them from SwapTotal - SwapFree underreported swap usage compared with free, Glances, and gopsutil's canonical SwapMemory metric.
Intel GPUs (intel_gpu_top) never report temperature or memory, so the
"suspended card" heuristic in calculateGPUAverage (temp == 0 &&
memoryUsed == 0) fired on every collection that landed between samples.
intel_gpu_top samples every 3.3s (intelGpuStatsInterval) while the hub's
realtime worker collects every 1s, so most realtime collections had no
new sample (delta count 0) and returned an empty GPUData with power
omitted (json "p"/"pp" are omitempty). The frontend derives the GPU
Power Draw series and legend from the latest sample, so the chart and
legend blanked on roughly two of every three or four one-second cycles.
NVIDIA/AMD were unaffected because they report temperature even when
idle, so the heuristic never fired and the last average was already
carried forward.
Gate the zero-return on non-engine (discrete) GPUs so Intel GPUs carry
the last average forward during between-sample gaps, matching the
existing NVIDIA/AMD behavior. Add a regression test.
Every other metric in getSystemStats is stored through utils.TwoDecimals.
The load averages were assigned straight from gopsutil, so whatever the
platform reported was recorded verbatim.
On Linux that goes unnoticed because /proc/loadavg is already two decimal
places. Everywhere else it is not. macOS and BSD divide a fixed point
value by fscale and produce numbers like 2.55322265625, and the Windows
implementation synthesises the average as a decaying EWMA over the
processor queue length counter, so an idle machine reports values like
1.3667392689044936e-73 instead of 0.
The hub already treats two decimals as the canonical precision for this
field, since records.go rounds the load average when it averages records.
That left the raw agent records as the only place carrying full precision.
Battery names come from firmware (sysfs model_name on Linux), which does not
guarantee valid UTF-8. The hub decodes agent payloads using the default
fxamacker/cbor decode mode, which rejects invalid UTF-8, so a single bad byte
in a battery name makes the hub drop the entire payload and mark the system
down until the agent is downgraded.
gopsutil's host.Uptime() calls the sysinfo(2) syscall. Inside an LXC
container lxcfs virtualizes /proc/uptime but cannot intercept a
syscall, so every container reported the host's uptime.
Reads /proc/uptime on linux and falls back to host.Uptime() if the file
is missing or unparseable, so other platforms are unchanged.
gopsutil's cpu.Info() does not parse the 'cpu model' field from
/proc/cpuinfo, which is the only source of CPU model names on MIPS.
Add a fallback that reads /proc/cpuinfo directly and combines
'cpu model' (e.g. 'MIPS 1004Kc V2.15') with 'system type'
(e.g. 'MediaTek MT7621 ver:1 eco:3') for a complete identifier.
The fallback only triggers when gopsutil returns an empty ModelName,
so x86/ARM/other architectures are unaffected.
- Report battery data for individual devices
- Select a representative battery for legacy fields and alerts
- Average named battery data independently
- Display multiple batteries in system charts
- Add cross-platform coverage and transport tests
intel_gpu_top does not support the xe driver, so skip it for xe devices
and let the existing nvtop last-resort collector handle them. nvtop leaves
device_name unset on xe, so name the GPU from its PCI device id ("Intel GPU
(<id>)"). Adds nvtop to the Intel agent image (gputop already ships with
igt-gpu-tools).
When SMART_DEVICES specifies an explicit type (e.g. /dev/sda:scsi), the
agent resolved the device type correctly but smartctlArgs dropped the -d
flag for scsi/ata (the #1345 scan-misdetection workaround), so smartctl
re-detected the wrong type (sat) and collection failed on USB drives
whose bridge does not support SAT passthrough.
Mark types that come from an explicit SMART_DEVICES hint and always pass
them through via -d, while still letting scan-detected scsi/ata
auto-detect as before. Adds regression tests for the arg building, the
full parse -> merge -> args path, and flag preservation across rescans.
Fixes#2072
CalculateCpuPercentLinux subtracted the stored previous counters from the
current ones without checking direction. When a stats response is processed
after a newer one for the same container, or an accounting counter resets,
the current total reads lower and the unsigned subtraction wraps to ~2^64
instead of going negative.
On the container counter that surfaces as the reported error, and the sample
is discarded along with the container's network stats:
cpu pct greater than 100: 1.15292150348562e+13
On the system counter it is quieter and worse: the wrapped value inflates the
divisor, so the percentage collapses toward zero and is stored as a healthy
sample rather than rejected. A synthetic rollback measures 2.7e-12 percent.
Both directions are now treated as a new baseline (0% for one sample), which
matches how the function already handles the first-run case.
CalculateCpuPercentWindows had the same unguarded subtraction and is fixed
the same way.
Fixes#2149
Co-authored-by: Ryan Chou <ryanchou1994@users.noreply.github.com>
Extract host memory derivation into calculateHostMemoryUsage and use
saturating subtraction to prevent uint64 underflow caused by race
conditions when reading /proc/meminfo. Also ensure UsedPercent guards
against division by zero.
Co-authored-by: Sven van Ginkel <svenvanginkel@icloud.com>
decode() reused a single json.Decoder across responses. Once a decode errored on a truncated or malformed body (e.g. a container removed mid-request), the decoder stayed desynced and every subsequent decode failed, silently stopping all container stats until the agent was restarted. Decode the buffered bytes with json.Unmarshal instead, which keeps the reusable read buffer but holds no cross-call state.
Regression from d67d638.
Some enterprise NVMe drives (e.g. Dell Ent NVMe CM7 U.2) report capacity
via nvme_total_capacity instead of user_capacity.bytes in smartctl output.
The NVMe SMART parser now falls back to nvme_total_capacity when
user_capacity.bytes is zero.
