how to use Cockpit and/or Performance Co-Pilot (PCP) as alert system

[Jelle de Jong]

Hello everybody,

How can I use cockpit to sent me an alert when disk space goes low, CPU 
load stays high, systemd status has a failure etc?

Is it possible to do this in combination with Performance Co-Pilot? I 
use ansbile a lot and was hoping I could ansible to automatic add nodes 
to monitoring use SSH for data collection.

I had asked this question yesterday on IRC and got the bellow reply:

"I think there may be better channels sharing info about alerting and 
monitoring sysOps, but surely using an stack like grafana, prometheus, 
telegraf... could help you. Or more easy, NMS engines like librenms, 
zabbix... the variety options is endless"

"tuxcrafter: right, what GilObradors[m] said -- there is nothing running 
when you are not logged into cockpit; it's not a permanent monitoring 
solution."

Can I do permanent monitoring with cockpit? Can I set a few 
alerts/notifications? If not can these features be added to cockpit and 
what alternative would you recommend, I values an F/LOSS tool, nodes 
easily added with ansible, preferable no daemon/agents, just SSH.

Kind regards,

Jelle de Jong
The Netherlands

[Martin Pitt]

Hello Jelle,

Jelle de Jong [2022-07-13 12:46 +0200]:
> How can I use cockpit to sent me an alert when disk space goes low, CPU load
> stays high, systemd status has a failure etc?

You can't. This is not what Cockpit does -- there is literally nothing running
on a system (other than the systemd socket unit) until someone logs into
Cockpit, and afterwards everything goes away again. Also, Cockpit has no
notification/communication methods, and isn't going to get any.
> "I think there may be better channels sharing info about alerting and
> monitoring sysOps, but surely using an stack like grafana, prometheus,
> telegraf... could help you. Or more easy, NMS engines like librenms,
> zabbix... the variety options is endless"
> what alternative would you recommend, I values an F/LOSS tool, nodes easily
> added with ansible, preferable no daemon/agents, just SSH.

For monitoring you need *something* on the system that regularly probes it;
that can be a permanently running daemon, or a cron job, or a cron job on a
remote site which regularly SSHs into the system to collect a data snapshot and
sends it off somewhere else.

Gil already mentioned Grafana/prometheus as the de-facto standard monitoring
solution these days. They can both be deployed into a container somewhere
centrally, and the managed machines only have to run PCP (in particular,
pmproxy) and redis (also easily installed).

Cockpit supports that, and I recently wrote about this [1]. That also has a
link for how to automate the PCP/redis setup on the watched machines [2], it's
just three commands.

Another very popular monitoring software is https://www.nagios.org/ . I don't
have personal experience with that, though.

Martin

[1] https://cockpit-project.org/blog/pcp-grafana.html
[2] https://cockpit-project.org/guide/latest/feature-pcp.html