feat(alerts): add container health alerts with log excerpt on notifications (#2225)

Add a new "ContainerHealth" alert type that fires when a Docker container's
health check reports unhealthy, and resolves when it recovers. This mirrors
the existing Status (up/down) alert pattern: an alert can be armed per system
and honors the "min minutes" delay before firing.

When the alert fires, the notification (email and any configured webhook,
including Discord via shoutrrr) includes a log excerpt fetched live from the
agent for up to 2 of the unhealthy containers, prioritizing lines containing
"error" or "fatal" (falling back to the log tail if none match), capped to
keep the message well under Discord's size limit.

---------

Co-authored-by: hank <hank@henrygd.me>
This commit is contained in:
Michał Mleczko
2026-09-02 18:46:23 +02:00
committed by GitHub
parent b1895247ba
commit 5969d36856
44 changed files with 1245 additions and 16 deletions

View File

@@ -29,7 +29,11 @@ func (stubHub) HandleSystemAlerts(systemRecord *core.Record, data *esystem.Combi
return nil
}
func (stubHub) HandleStatusAlerts(status string, systemRecord *core.Record) error { return nil }
func (stubHub) CancelPendingStatusAlerts(systemID string) {}
func (stubHub) HandleContainerAlerts(systemRecord *core.Record, data *esystem.CombinedData, fetchLogs func(containerID string) (string, error)) error {
return nil
}
func (stubHub) CancelPendingStatusAlerts(systemID string) {}
func (stubHub) CancelPendingContainerAlerts(systemID string) {}
// newTestSystemWithHub creates a System backed by a real (temp) database, along
// with a matching "systems" record, for tests that need to exercise DB reads/writes.