Updated 28 Oct 07:23:03 BKK by kookwatch. Alert sender (leader): kookwatch. Open problems that every layer failed to fix: 5
- kookwatch: machine - disk 90% used, 5560 MB RAM free, load 5.20 (bad 76.0 h; no automatic fix exists)
- kookwatch: M runtrack snapshots freshness (Test Manager, from super) - 240.4 h old (max 75 min) (bad 239.1 h; 1395 fix(es), last: restart runtrack.service (rc 0))
- kookwatch: M 10 s holder and buy/sell tape freshness (Test Manager, from super) - 225.1 h old (max 30 min) (bad 224.5 h; 1310 fix(es), last: restart rhfast.service (rc 0))
- kookwatch: M Picker retrain freshness (Test Manager, from super) - 302.4 h old (max 36.0 h) (bad 266.4 h; 1554 fix(es), last: restart hist-retrain.service (rc 0))
- kookwatch: M Launch catch-up (reconcile) freshness (Test Manager, from super) - 264.5 h old (max 10 min) (bad 264.3 h; 1542 fix(es), last: restart reconcile.timer (rc 0))
Layer 1 = each box's own guard (twice a minute). Layer 2 = the other boxes call its entry over SSH every minute (revive guard, start dead units).
Telegram only when both fail. To stop a unit on purpose: systemctl disable --now X or touch /opt/guard/hold/X.
| firstmin-trader - local guard 34 s ago | |||
| AI pick feed reaches this box | OK | 0 s old | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 2 min old | |
| L live-change log cron job | OK | 1 jobs present | |
| LIVE bot fmmc5 (real money) | OK | running, heartbeat 2 s | |
| S stop rule: 10 losses in a row (paper/live test 2 only; Ben 8 Oct: none for fmmc5) | OK | 1 jobs present | |
| dashboard pnl.html (live) | OK | 17 s old | |
| dashboard pnl2.html | OK | 20 s old | |
| dashboard pnl2v.html | OK | 20 s old | |
| dashboard tests.json | OK | 22 s old | |
| gmgnd (live sells) running | OK | running | |
| gmgnd answers | OK | answers ping {"warmOk": 21830, "warmErr": 3, "lastWarmMs": 195} | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 37% used, 302 MB RAM free, load 3.47 | |
| outside-money card | OK | 2 min old | |
| paper test 1 fml2 | OK | running, heartbeat 0 s | |
| paper test 1b fml2v | OK | running, heartbeat 0 s | |
| safety + dashboard cron jobs | OK | 12 jobs present | |
| kookwatch - local guard 0 s ago | |||
| 60 s AI picker (feeds paper tests) | OK | running, heartbeat 2 s | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 5 min old | |
| L ROT chains.json from rot-box (Manager) | OK | 47 min old | |
| L ROT closes.json from rot-box (Manager) | OK | 17 min old | |
| L ROT map-data.json from rot-box (Manager) | OK | 3 min old | |
| L ROT rh_drill.json from rot-box (Manager) | OK | 8 min old | |
| L ROT rotation.json from rot-box (Manager) | OK | 91 s old | |
| L ROT rotmap.json from rot-box (Manager) | OK | 4.0 h old | |
| L ROT sync main<->rot-box (Manager) | OK | 55 s old | |
| L judge guard ok (PP Data) | OK | 2 s old | |
| L lane check ok (PP Data) | OK | 4 s old | |
| L lane price tracker (PP Data) | OK | 78 s old | |
| L lane stats page (PP Data) | OK | 2 min old | |
| L live-change log cron job | OK | 1 jobs present | |
| L nightly review cron job | OK | 1 jobs present | |
| L nightly review page | OK | 23.7 h old | |
| M 10 s holder and buy/sell tape freshness (Test Manager, from super) | BAD | 225.1 h old (max 30 min) | 1310 fix(es): restart rhfast.service (rc 0) |
| M AI picker round 5 (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| M Ben Telegram buttons (benbot) | OK | running | |
| M Ben Telegram line in cron | OK | 2 jobs present | |
| M Launch catch-up (reconcile) freshness (Test Manager, from super) | BAD | 264.5 h old (max 10 min) | 1542 fix(es): restart reconcile.timer (rc 0) |
| M Live job checker (sentinel) freshness (Test Manager, from super) | OK | 25 s old | |
| M PP Feeds layer-2 guard (ppfeedguard2) | OK | running, heartbeat 14 s | |
| M PP data cron lines (PP Data) | OK | 3 jobs present | |
| M PP data liquidity table (liqinc) (PP Data) | OK | 59 min old | |
| M PP data picker-box copy (pbsync pull) (PP Data) | OK | 12 s old | |
| M PP data ponstape (PP Data) | OK | running | |
| M PP data ppexit honest-exit tracker (PP Data) | OK | 0 s old | |
| M PP data venuepick paper picker (PP Data) | OK | 0 s old | |
| M PP feed GMGN board (price-predict-rh) | - | disabled on purpose | |
| M PP feed crewlive (crews) | OK | running, heartbeat 7 s | |
| M PP feed klines | OK | running, heartbeat 18 s | |
| M PP feed ponstape (Pons curve) | OK | running, heartbeat 14 s | |
| M PP feed rhledger | OK | running | |
| M PP feed runtrack (runners) | OK | heartbeat 240.4 h old but log live (not hung) | |
| M Picker guard (pickguard) freshness (Test Manager, from super) | OK | 22 s old | |
| M Picker retrain freshness (Test Manager, from super) | BAD | 302.4 h old (max 36.0 h) | 1554 fix(es): restart hist-retrain.service (rc 0) |
| M SW engine w2coder (paper) | OK | running, heartbeat 56 s | |
| M bot watchdog freshness (Test Manager, from super) | OK | 4 min old | |
| M cleaner freshness (Test Manager, from super) | OK | 8 min old | |
| M daily miss review in cron | OK | 1 jobs present | |
| M gmgn feeds recorder freshness (Test Manager, from super) | OK | 69 s old | |
| M instant miss watch ran | OK | 2 min old | |
| M lane price tracker (lanetrack) freshness (Test Manager, from super) | OK | 78 s old | |
| M lane radar v3 (laneradar3) freshness (Test Manager, from super) | OK | 0 s old | |
| M lane radar v3 beat (PP Data) | OK | 0 s old | |
| M lane stats page (lanestats) freshness (Test Manager, from super) | OK | 2 min old | |
| M meta radar (metaradar) freshness (Test Manager, from super) | OK | 95 min old | |
| M paper test (pp-paper) freshness (Test Manager, from super) | OK | 101 s old | |
| M picker learning loop cron (hourly) | OK | 1 jobs present | |
| M ppinc hourly new pools + supply (PP Data) | OK | 19 min old | |
| M rate-limit review in cron | OK | 2 jobs present | |
| M rate-limit review ran (heartbeat) | OK | 57 min old | |
| M runtrack snapshots freshness (Test Manager, from super) | BAD | 240.4 h old (max 75 min) | 1395 fix(es): restart runtrack.service (rc 0) |
| S AI picker (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| S P3 mirror cross (a1pull@cross) | OK | running | |
| S P3 mirror early (a1pull@early) | OK | running | |
| S backup checker (livewatch2) freshness (Test Manager, from super) | OK | 2 s old | |
| S fast checker (livewatch) freshness (Test Manager, from super) | OK | 0 s old | |
| S fast checker livewatch | OK | running, heartbeat 0 s | |
| S live scanner rhfast (feeds Price Predict) | - | disabled on purpose | |
| S live settings watchdog | OK | 1 jobs present | |
| S live settings watchdog ran | OK | 54 s old | |
| backup picker aipick5b stays off | OK | off (as decided) | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| machine | BAD | disk 90% used, 5560 MB RAM free, load 5.20 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 89 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| pick feed file l2picks.json | OK | 0 s old | |
| pick feed l2feed | OK | running, heartbeat 0 s | |
| published tests dashboard | OK | 52 s old | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |
| testspub cron job | OK | 1 jobs present | |
| picker-box - local guard 3 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 5 min old | |
| M lab paper books | - | on hold (/opt/guard/hold/labbooks) | |
| M shadow picker pbshadow | OK | running | |
| free RPC relay (2nd IP) | OK | running | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 34% used, 593 MB RAM free, load 0.62 | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| research-box - local guard 3 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 5 min old | |
| L kitb | OK | 9 min old | |
| L shadow grid report (Test Manager, nightly 02:30) | OK | 4.4 h old | |
| L trending feed | OK | 2 min old | |
| M KITFAST kit feed (PP Feeds) | OK | running, heartbeat 4 s | |
| M PP Feeds layer-1 guard (ppfeedguard) | OK | running, heartbeat 1 s | |
| M auditq ran | OK | 2 min old | |
| M crew tagger pretag | OK | running, heartbeat 17 s | |
| M datajob runner ran | OK | 59 s old | |
| M flowwatch ran | OK | 59 s old | |
| M ideaq ran | OK | 0 s old | |
| M killbot ran | OK | 12 min old | |
| M paper cards ran | OK | 2 min old | |
| M picker learning loop ran (hourtrain, hourly) | OK | 32 min old | |
| M v3 live feed v3live | OK | running, heartbeat 1 s | |
| M v3 mc5 paper book | OK | running, heartbeat 1 s | |
| S A1 early + cross feed (a1feed) | OK | running, heartbeat 0 s | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| history rebuild moving (one-time, paid) | BAD | no progress for 61 min | 1 fix(es): restart hist-rh (rc 0) |
| machine | OK | disk 89% used, 14530 MB RAM free, load 5.76 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 10 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| rebuild moves to free when done (cron) | OK | 1 jobs present | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |