One of my machines ran out of disk due to a scraper during the week, so I finally set up:

• email alerting
• having alerts on disk space per machine
• having some alerts on services not running.
• having some probing.

Hopefully all will improve. I found this a useful collection of prom rules.