Updated 23 Oct 13:02:30 BKK by kookwatch. Alert sender (leader): kookwatch. Open problems that every layer failed to fix: 7
- research-box: history rebuild moving (one-time, paid) - no progress for 6.8 h (bad 5.7 h; 12 fix(es), last: restart hist-rh (rc 0))
- research-box: M picker learning loop ran (hourtrain, hourly) - 349.2 h old (max 4.0 h) (bad 338.7 h; no automatic fix exists)
- kookwatch: machine - disk 89% used, 5146 MB RAM free, load 8.41 (bad 39.6 h; no automatic fix exists)
- kookwatch: M runtrack snapshots freshness (Test Manager, from super) - 126.0 h old (max 75 min) (bad 124.8 h; 728 fix(es), last: restart runtrack.service (rc 0))
- kookwatch: M 10 s holder and buy/sell tape freshness (Test Manager, from super) - 110.7 h old (max 30 min) (bad 110.2 h; 643 fix(es), last: restart rhfast.service (rc 0))
- kookwatch: M Picker retrain freshness (Test Manager, from super) - 188.1 h old (max 36.0 h) (bad 152.1 h; 887 fix(es), last: restart hist-retrain.service (rc 0))
- kookwatch: M Launch catch-up (reconcile) freshness (Test Manager, from super) - 150.2 h old (max 10 min) (bad 150.0 h; 875 fix(es), last: restart reconcile.timer (rc 0))
Layer 1 = each box's own guard (twice a minute). Layer 2 = the other boxes call its entry over SSH every minute (revive guard, start dead units).
Telegram only when both fail. To stop a unit on purpose: systemctl disable --now X or touch /opt/guard/hold/X.
| firstmin-trader - local guard 24 s ago | |||
| AI pick feed reaches this box | OK | -2 s old | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 113 s old | |
| L live-change log cron job | OK | 1 jobs present | |
| LIVE bot fmmc5 (real money) | OK | running, heartbeat 2 s | |
| S stop rule: 10 losses in a row (paper/live test 2 only; Ben 8 Oct: none for fmmc5) | OK | 1 jobs present | |
| dashboard pnl.html (live) | OK | 50 s old | |
| dashboard pnl2.html | OK | 53 s old | |
| dashboard pnl2v.html | OK | 53 s old | |
| dashboard tests.json | OK | 54 s old | |
| gmgnd (live sells) running | OK | running | |
| gmgnd answers | OK | answers ping {"warmOk": 1263, "warmErr": 1, "lastWarmMs": 213} | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 37% used, 381 MB RAM free, load 4.05 | |
| outside-money card | OK | 110 s old | |
| paper test 1 fml2 | OK | running, heartbeat 0 s | |
| paper test 1b fml2v | OK | running, heartbeat 0 s | |
| safety + dashboard cron jobs | OK | 12 jobs present | |
| kookwatch - local guard 0 s ago | |||
| 60 s AI picker (feeds paper tests) | OK | running, heartbeat 7 s | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 45 min old | |
| L ROT chains.json from rot-box (Manager) | OK | 27 min old | |
| L ROT closes.json from rot-box (Manager) | OK | 57 min old | |
| L ROT map-data.json from rot-box (Manager) | OK | 10 min old | |
| L ROT rh_drill.json from rot-box (Manager) | OK | 47 min old | |
| L ROT rotation.json from rot-box (Manager) | OK | 40 min old | |
| L ROT rotmap.json from rot-box (Manager) | OK | 97 min old | |
| L ROT sync main<->rot-box (Manager) | OK | 21 s old | |
| L judge guard ok (PP Data) | OK | 17 s old | |
| L lane check ok (PP Data) | OK | 4 s old | |
| L lane price tracker (PP Data) | OK | 20 s old | |
| L lane stats page (PP Data) | OK | 103 s old | |
| L live-change log cron job | OK | 1 jobs present | |
| L nightly review cron job | OK | 1 jobs present | |
| L nightly review page | OK | 5.4 h old | |
| M 10 s holder and buy/sell tape freshness (Test Manager, from super) | BAD | 110.7 h old (max 30 min) | 643 fix(es): restart rhfast.service (rc 0) |
| M AI picker round 5 (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| M Ben Telegram buttons (benbot) | OK | running | |
| M Ben Telegram line in cron | OK | 2 jobs present | |
| M Launch catch-up (reconcile) freshness (Test Manager, from super) | BAD | 150.2 h old (max 10 min) | 875 fix(es): restart reconcile.timer (rc 0) |
| M Live job checker (sentinel) freshness (Test Manager, from super) | OK | 19 s old | |
| M PP Feeds layer-2 guard (ppfeedguard2) | OK | running, heartbeat 7 s | |
| M PP data cron lines (PP Data) | OK | 3 jobs present | |
| M PP data liquidity table (liqinc) (PP Data) | OK | 39 min old | |
| M PP data picker-box copy (pbsync pull) (PP Data) | OK | 8 s old | |
| M PP data ponstape (PP Data) | OK | running | |
| M PP data ppexit honest-exit tracker (PP Data) | OK | 1 s old | |
| M PP data venuepick paper picker (PP Data) | OK | 1 s old | |
| M PP feed GMGN board (price-predict-rh) | OK | running, heartbeat 43 s | |
| M PP feed crewlive (crews) | OK | running, heartbeat 0 s | |
| M PP feed klines | OK | running, heartbeat 37 s | |
| M PP feed ponstape (Pons curve) | OK | running, heartbeat 18 s | |
| M PP feed rhledger | OK | running | |
| M PP feed runtrack (runners) | OK | heartbeat 126.0 h old but log live (not hung) | |
| M Picker guard (pickguard) freshness (Test Manager, from super) | OK | 17 s old | |
| M Picker retrain freshness (Test Manager, from super) | BAD | 188.1 h old (max 36.0 h) | 887 fix(es): restart hist-retrain.service (rc 0) |
| M SW engine w2coder (paper) | OK | running, heartbeat 37 s | |
| M bot watchdog freshness (Test Manager, from super) | OK | 77 s old | |
| M cleaner freshness (Test Manager, from super) | OK | 3 min old | |
| M daily miss review in cron | OK | 1 jobs present | |
| M gmgn feeds recorder freshness (Test Manager, from super) | OK | 34 s old | |
| M instant miss watch ran | OK | 2 min old | |
| M lane price tracker (lanetrack) freshness (Test Manager, from super) | OK | 20 s old | |
| M lane radar v3 (laneradar3) freshness (Test Manager, from super) | OK | 6 s old | |
| M lane radar v3 beat (PP Data) | OK | 6 s old | |
| M lane stats page (lanestats) freshness (Test Manager, from super) | OK | 103 s old | |
| M meta radar (metaradar) freshness (Test Manager, from super) | OK | 83 min old | |
| M paper test (pp-paper) freshness (Test Manager, from super) | OK | 9 s old | |
| M picker learning loop cron (hourly) | OK | 1 jobs present | |
| M ppinc hourly new pools + supply (PP Data) | OK | 21 min old | |
| M rate-limit review in cron | OK | 2 jobs present | |
| M rate-limit review ran (heartbeat) | OK | 37 min old | |
| M runtrack snapshots freshness (Test Manager, from super) | BAD | 126.0 h old (max 75 min) | 728 fix(es): restart runtrack.service (rc 0) |
| S AI picker (paper) freshness (Test Manager, from super) | OK | 7 s old | |
| S P3 mirror cross (a1pull@cross) | OK | running | |
| S P3 mirror early (a1pull@early) | OK | running | |
| S backup checker (livewatch2) freshness (Test Manager, from super) | OK | 17 s old | |
| S fast checker (livewatch) freshness (Test Manager, from super) | OK | 10 s old | |
| S fast checker livewatch | OK | running, heartbeat 10 s | |
| S live scanner rhfast (feeds Price Predict) | OK | running, heartbeat 37 s | |
| S live settings watchdog | OK | 1 jobs present | |
| S live settings watchdog ran | OK | 20 s old | |
| backup picker aipick5b stays off | OK | off (as decided) | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| machine | BAD | disk 89% used, 5146 MB RAM free, load 8.41 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 35 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| pick feed file l2picks.json | OK | 0 s old | |
| pick feed l2feed | OK | running, heartbeat 0 s | |
| published tests dashboard | OK | 18 s old | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |
| testspub cron job | OK | 1 jobs present | |
| picker-box - local guard 29 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 45 min old | |
| M lab paper books | - | on hold (/opt/guard/hold/labbooks) | |
| M shadow picker pbshadow | OK | running | |
| free RPC relay (2nd IP) | OK | running | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 33% used, 599 MB RAM free, load 0.16 | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| research-box - local guard 28 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 45 min old | |
| L kitb | OK | 58 s old | |
| L shadow grid report (Test Manager, nightly 02:30) | OK | 10.1 h old | |
| L trending feed | OK | 2 min old | |
| M KITFAST kit feed (PP Feeds) | OK | running, heartbeat 0 s | |
| M PP Feeds layer-1 guard (ppfeedguard) | OK | running, heartbeat 6 s | |
| M auditq ran | OK | 2 min old | |
| M crew tagger pretag | OK | running, heartbeat 2 s | |
| M datajob runner ran | OK | 29 s old | |
| M flowwatch ran | OK | 10 min old | |
| M ideaq ran | OK | 9 min old | |
| M killbot ran | OK | 51 min old | |
| M paper cards ran | OK | 2 min old | |
| M picker learning loop ran (hourtrain, hourly) | BAD | 349.2 h old (max 4.0 h) | |
| M v3 live feed v3live | OK | running, heartbeat 1 s | |
| M v3 mc5 paper book | OK | running, heartbeat 2 s | |
| S A1 early + cross feed (a1feed) | OK | running, heartbeat 0 s | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| history rebuild moving (one-time, paid) | BAD | no progress for 6.8 h | 12 fix(es): restart hist-rh (rc 0) |
| machine | OK | disk 88% used, 8439 MB RAM free, load 24.75 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 13 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| rebuild moves to free when done (cron) | OK | 1 jobs present | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |