How to Train a Watchdog