Gladex Agent Logs

141:    'echo "summary: 1 assertions, 1 passed, 0 failed"' \
143:# rc == 0 with a failing summary -> also red, from the other side.
145:    'echo "FAIL RWTEST A1 failing summary"' \
146:    'echo "summary: 5 assertions, 4 passed, 1 failed"' \
237:assert_contains "B3 …and monitor_exit/passed/failed from the summary" \
268:assert_contains "C3 …monitor_exit=1 with the summary counts" \
305:assert_eq "C27 the ORIGINAL onset epoch survives the set change" \
312:assert_contains "C29 …reported state=red even though its summary says 0 failed" \
317:assert_rc "C31 a failing summary exits 0 (warn-only)" 0 "$RC"
320:assert_contains "C33 …with the summary's own failed count" "failed=1" "$(out)"
365:assert_rc "E10 a monitor with no summary line exits 3" 3 "$RC"
367:assert_contains "E12 the detail says why (no summary line)" "no summary line" "$(alerts)"
393:assert_eq "E27 that failure payload carries exactly 6 keys" "6" \
420:assert_contains "F6 the assertions object carries both counts" \
421:    '"assertions": {"failed": 0, "passed": 1}' "$J"
426:EXPECT_KEYS="alert alert_file assertions check_error exit_code fails log_file monitor monitor_exit ok repo state state_file timestamp tool"
519:    assert_lacks "G27 …and no longer claims --help renders the docstring's exit codes" \
554:    "M6": ('check_error = "monitor printed no summary line (monitor_exit=%s)" % rc',
610:    run_mutation M6 "E10 a monitor with no summary line exits 3"
=== regression-run suite list ===
=== system-status monitor refs ===
1072:# `tools/red-watch` runs `tests/test_gladex_monitor.sh` every 15 minutes (cron
=== queue-source-check monitor refs ===
76:    to keep `test_gladex_monitor.sh`'s clean-tree assertions green during a
> build · mimo-v2.6-flash-free
$ cd /data/repo && echo "=== REGISTRY sections for suites ==="; grep -n "^## " tools/REGISTRY.md | grep -i "test_\|suite" | head -20; echo "=== monitor section heading? ==="; grep -n "^## .*monitor" tools/REGISTRY.md; echo "=== how regression-run discovers suites ==="; grep -n "tests/\*\|glob\|GLO\|suite_list\|SUITES=" tools/regression-run | head -20; echo "=== cron ==="; crontab -l
=== REGISTRY sections for suites ===
2905:## test_identity_wrapper.sh
2933:## test_app_contrast_meta.php
3113:## test_regression_run.sh
3142:## test_queue_source.sh
3456:## test_registry_coverage.sh
4180:## test_detached_children.sh
4983:## test_start_page.php
5107:## test_onboarding_baseline.php
5146:## test_commit_gate.sh
5266:## test_qa_handover.php
5558:## test_red_watch.sh
5642:## test_system_status_red_watch.sh
=== monitor section heading? ===
=== how regression-run discovers suites ===
=== cron ===
0 3 * * 0 /data/repo/tools/ip-drift-cron gladex.de
*/15 * * * * /data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1
> build · mimo-v2.6-flash-free
$ cd /data/repo && echo "=== regression-run discovery ==="; grep -n "test_\*\|find .*tests\|re\.match\|glob\|suite" tools/regression-run | head -25; echo; echo "=== REGISTRY red-watch section (5460-5500) ==="; sed -n '5460,5500p' tools/REGISTRY.md; echo "=== monitor count in REGISTRY? ==="; grep -n "monitor.*27\|27.*monitor" tools/REGISTRY.md
=== regression-run discovery ===
2:"""regression-run — run every test suite in the repo and total the result.
5:    Every CHANGELOG entry in this repo quotes a suite total ("39 suites, 3203
9:    listing 16 of 39 suites as no-summary failures — twelve of which had
14:    The contract it is built around: a suite that cannot be parsed is NEVER
18:    The tree carries four summary shapes and a suite may add any number of
20:    `Results: 19 passed, 0 failed`, `repo-lint suite: 291 passed, 0 failed`,
24:    "Last match", not "last line": seven suites print trailing noise after
31:    match is used rather than reporting no summary — a suite really did print
33:    match_count is reported per suite, so a run where the tie-break actually
35:    A suite with no match at all is `no_summary` and contributes NOTHING to
39:    --save-baseline FILE records this run's per-suite counts as a small JSON
43:    how "+110 somewhere" turns into an unverified "no other suite moved".
44:    The report gives per-suite deltas, the suites added and removed, every
45:    status change, and a CLOSURE check: the sum of the per-suite deltas must
49:    One function (contrib) decides what a suite contributes, and it is used
55:    A suite outside --only's filter reports as removed, because the baseline
70:    baseline document, so an edited file is refused before any suite runs.
75:    0  every discovered suite ran and reported 0 failed
76:    1  at least one suite reported failed > 0
81:    3  a suite could not be verified — it crashed with no failure report, it
84:    4  a suite produced no parseable summary line
85:    5  no test suites were discovered
87:       suites from is short of GLADEX_MIN_FREE_MB (default 512), or a watched
89:    Precedence: 2 > 6 > 3 > 4 > 1 > 0, and EVERY suite's outcome is printed
90:    regardless of which code wins. A suite that reports failures and exits 1

=== REGISTRY red-watch section (5460-5500) ===
**Exit codes**:
- `0` - The check ran and its verdict was recorded — **green or red**. `ok` in
  the JSON is the *tree's* verdict (`state == "green"`), not the tool's, so a
  red tree still reads `ok: false` on an exit 0.
- `2` - Invalid arguments (unknown option, `--format` outside `human|json`,
  `--format` with no value, `REDWATCH_MONITOR_TIMEOUT` not a positive integer)
  — validated by `argparse` **before any I/O**, so a typo never runs the
  monitor and never writes a log line
- `3` - **Cannot verify**: the monitor is missing or not runnable, timed out,
  or printed no `summary:` line. A `CHECK-ERROR` alert is written. An input
  that was not read is never a pass **and never a red either** — `state` stays
  at the last verdict, so a later green run still reads as a `RECOVERED`
  instead of adopting "check-error" as a resting state

**What it runs**: `bash <repo>/tests/test_gladex_monitor.sh` — the repo's own
27-assertion monitor (git clean/in-sync, `queue-source-check`, `repo-lint`,
`system-status`, `inbox-status`), i.e. both detectors item (114) named, through
one sanctioned entry point rather than a second copy of them. Measured
**1 m 51 s** per run on 2026-10-02 (~13 % duty at the cron interval below).

**State machine** (one alert line *only* on a change, so a tree that stays red
does not fill the file):
- `green` → `red`: **`RED`** alert, `since=<epoch>` recorded in the state file
- still `red`, **failing set changed**: **`RED-SET`** alert carrying the new
  `FAIL` lines — this is the line that distinguishes *another identity's
  untracked file* (`A3`) from *the gates everyone is blocked on* (`A9`–`A11`)
- `red` → `green`: **`RECOVERED after=<n>s`**
- still `green`: **no alert** (a quiet success stays quiet)
- check could not run: **`CHECK-ERROR`** every time, per `ip-drift-cron`'s rule

**Outputs** (all outside the git tree, so a scheduled run can never dirty
`git status`):
- `/data/agent-logs/red-watch-YYYYMMDD.log` — **one line per run**, green or red
- `/data/agent-logs/red-watch-ALERTS.log` — append-only; `RED` / `RED-SET` /
  `RECOVERED` / `CHECK-ERROR` lines, each carrying the monitor's own `FAIL`
  lines (whitespace-collapsed, capped at 600 chars)
- `/data/agent-logs/red-watch.state` — `state` / `since` / `failed` / `fails` /
  `updated`, the memory that makes onset-vs-repeat answerable

**Cron schedule**: `*/15 * * * * /data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1`
(15 min matches `agent-loop-watchdog.timer`; the redirect keeps cron from
=== monitor count in REGISTRY? ===
5475:27-assertion monitor (git clean/in-sync, `queue-source-check`, `repo-lint`,
> build · mimo-v2.6-flash-free
$ cd /data/repo && echo "=== regression-run discovery fn ==="; grep -n "def discover\|iterdir\|scandir\|startswith(\"test_\|test_.*\.sh\|test_.*\.php" tools/regression-run | head -15; echo "=== is monitor in a baseline? ==="; ls /data/agent-logs/*.json 2>/dev/null | head; grep -rn "gladex_monitor" /data/agent-logs/*baseline* 2>/dev/null | head -3; echo "=== test_system_status_red_watch.sh monitor refs ==="; grep -n "monitor\|27" tests/test_system_status_red_watch.sh | head -20
=== regression-run discovery fn ===
25:    their summary — a `====` banner in six, and `test_pdns_api.sh` ends in a
271:# built for. `test_regression_run.sh` A23 holds this comment honest by
281:# It is measured now: `test_regression_run.sh` A24 compares this order with
363:def discover(tests_dir: str):
372:        if not name.startswith("test_"):
1048:        # tests/test_registry_coverage.sh fails if this row stops naming one.
=== is monitor in a baseline? ===
=== test_system_status_red_watch.sh monitor refs ===
13:# on the dashboard every run, the monitor (probe `sys`/`sysj`) and the reviewer
> build · mimo-v2.6-flash-free
Design is settling: **(132)** = a `docs` probe + assertions A28/A29 in `tests/test_gladex_monitor.sh` (red-watch's monitor), detecting code commits with no CHANGELOG behind them. Gathering baseline facts first.
$ cd /data/repo && echo "=== branch vs origin ==="; git status --short --branch; git rev-parse origin/main; git rev-parse HEAD; echo "=== red-watch state ==="; cat /data/agent-logs/red-watch.state 2>/dev/null; echo "=== last red-watch log lines ==="; tail -3 /data/agent-logs/red-watch-$(date -u +%Y%m%d).log 2>/dev/null; echo "=== ALERTS tail ==="; tail -4 /data/agent-logs/red-watch-ALERTS.log
=== branch vs origin ===
## main...origin/main [ahead 1]
5d7bb8f4c7c15638bc0c51c3e835ca84114971e2
930390b2844831a4d6eacf834075b6e2b48bab98
=== red-watch state ===
state=red
since=1791080794
failed=1
fails=FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main)
updated=2026-10-04T02:30:01Z
=== last red-watch log lines ===
2026-10-04T02:15:01Z state=green monitor_exit=0 passed=27 failed=0 alert=none fails=""
2026-10-04T02:24:54Z state=red monitor_exit=1 passed=26 failed=1 alert=RED fails="FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main)"
2026-10-04T02:30:01Z state=red monitor_exit=1 passed=26 failed=1 alert=none fails="FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main)"
=== ALERTS tail ===
2026-10-04T01:30:01Z RED since=1791077504 monitor_exit=1 passed=25 failed=2 fails="FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main) | FAIL A15 git-tree reports a clean tree (kind=contains key=sys want=[OK] clean)"
2026-10-04T01:42:41Z RED-SET since=1791077504 monitor_exit=1 passed=26 failed=1 fails="FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main)"
2026-10-04T01:47:11Z RECOVERED after=1032s previously_failed=1
2026-10-04T02:24:54Z RED since=1791080794 monitor_exit=1 passed=26 failed=1 fails="FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main)"
$ cd /data/repo && echo "=== last app/src/php commits: touched CHANGELOG? ==="; for c in $(git log --format=%H -6 -- app/src/php); do echo "--- $(git log -1 --format='%h %s' $c | cut -c1-90)"; git show --format= --name-only $c | grep -c '^CHANGELOG.md$'; done
=== last app/src/php commits: touched CHANGELOG? ===
--- 34ac28e identity mia shift 2026-10-03T16:36Z
0
--- ae6fcd7 identity leon shift 2026-10-03T16:28Z
0
--- 29f0996 jonas: shift 2026-10-03 morning — /docs scrolled sideways at 290px; overflow-wra
0
--- a44d70d identity lena shift 2026-10-03T06:18Z
0
--- 18f71e8 fix: remove backticks from CSS comment in start.php — renders inside <style>, te
0
--- c6d5f91 test: add overflow-wrap fix for /start page horizontal scroll at 320px
0
> build · mimo-v2.6-flash-free
$ echo "=== now ==="; date -u; echo "=== cron service ==="; systemctl is-active cron 2>/dev/null || service cron status 2>&1 | head -3; echo "=== cron log tail ==="; tail -5 /data/agent-logs/red-watch-cron.log 2>/dev/null; echo "=== grep red-watch in syslog ==="; grep -a "red-watch" /var/log/syslog 2>/dev/null | tail -3; grep -a CRON /var/log/syslog 2>/dev/null | tail -3
=== now ===
Sun Oct  4 02:43:40 UTC 2026
=== cron service ===
active
=== cron log tail ===
  alerts=/data/agent-logs/red-watch-ALERTS.log
red-watch: state=red monitor_exit=1 passed=26 failed=1 alert=none
  FAIL A3 tree clean AND in sync with origin/main (kind=is key=git want=## main...origin/main)
  log=/data/agent-logs/red-watch-20261004.log
  alerts=/data/agent-logs/red-watch-ALERTS.log
=== grep red-watch in syslog ===
2026-10-04T04:00:01.722275+02:00 startup-builder CRON[3402082]: (root) CMD (/data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1)
2026-10-04T04:15:01.100803+02:00 startup-builder CRON[3644790]: (root) CMD (/data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1)
2026-10-04T04:30:01.553414+02:00 startup-builder CRON[3696301]: (root) CMD (/data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1)
2026-10-04T04:30:01.553414+02:00 startup-builder CRON[3696301]: (root) CMD (/data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1)
2026-10-04T04:35:01.292272+02:00 startup-builder CRON[3705969]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2026-10-04T04:39:01.302867+02:00 startup-builder CRON[3707243]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
$ cd /data/repo && echo "=== recent tests/tools commits: CHANGELOG touched? ==="; for c in $(git log --format=%H -12 -- tests/ tools/ ':!tools/REGISTRY.md'); do n=$(git show --format= --name-only $c | grep -c '^CHANGELOG.md$'); files=$(git show --format= --name-only $c | grep -E '^(tests|tools)/' | grep -v '^tools/REGISTRY.md$' | head -2 | tr '\n' ' '); echo "$(git log -1 --format='%h %s' $c | cut -c1-60) | CHANGELOG=$n | $files"; done
=== recent tests/tools commits: CHANGELOG touched? ===
930390b run 14: 2026-10-04T02:18Z model=opencode/ling-3.1-fl | CHANGELOG=0 | tests/test_registry_coverage.sh 
f4b908a queue item (128) 9th of 9 [0.4.192]: tests/test_immi | CHANGELOG=1 | tests/test_immich_roundtrip.sh 
4d66755 run 640: 2026-10-04T01:08Z model=opencode/mimo-v2.6- | CHANGELOG=0 | tests/test_template_sync.sh 
bce01b5 run 11: 2026-10-04T00:44Z model=opencode/ling-3.1-fl | CHANGELOG=1 | tests/test_disk_show.sh 
28a9c96 run 639: 2026-10-04T00:28Z model=opencode/mimo-v2.6- | CHANGELOG=0 | tests/test_budget_show.sh 
a2c4758 run 631: 2026-10-03T22:39Z model=opencode/mimo-v2.6- | CHANGELOG=0 | tests/test_repo_lint.sh 
d8eff89 run 628: 2026-10-03T22:09Z model=opencode/mimo-v2.6- | CHANGELOG=1 | tests/test_source_sync_check.sh 
4f12018 queue item (128) 3rd of 9 [0.4.186]: test_promote_li | CHANGELOG=1 | tests/test_promote_lint_gate.sh 
582f490 run: 2026-10-03T18:06Z queue item (128) 2nd of 9: te | CHANGELOG=1 | tests/test_promote_gate.sh 
490adea run 605: 2026-10-03T17:44Z model=opencode/mimo-v2.6- | CHANGELOG=0 | tests/test_promote_json.sh 
548c129 run: 2026-10-03T17:27Z queue item (127): C43d built  | CHANGELOG=1 | tests/test_registry_coverage.sh 
153c67e identity sofia shift 2026-10-03T17:14Z: ONBOARDING t | CHANGELOG=0 | tests/test_onboarding_baseline.php
> build · mimo-v2.6-flash-free

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

[Sun Oct  4 04:25:49 2026] 127.0.0.1:42770 Accepted
[Sun Oct  4 04:25:49 2026] 127.0.0.1:42770 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48544 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48544 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48546 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48546 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48550 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48550 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48566 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48566 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48570 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48570 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48582 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48582 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48590 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48590 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48592 Accepted
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48592 Closing
[Sun Oct  4 04:30:09 2026] 127.0.0.1:48600 Accepted
[Sun Oct  4 04:30:10 2026] 127.0.0.1:48600 Closing
[Sun Oct  4 04:30:10 2026] 127.0.0.1:48610 Accepted
[Sun Oct  4 04:30:10 2026] 127.0.0.1:48610 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57500 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57500 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57506 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57506 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57522 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57522 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57528 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57528 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57542 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57542 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57558 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57558 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57568 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57568 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57574 Accepted
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57574 Closing
[Sun Oct  4 04:30:56 2026] 127.0.0.1:57580 Accepted
[Sun Oct  4 04:30:57 2026] 127.0.0.1:57580 Closing
[Sun Oct  4 04:30:57 2026] 127.0.0.1:57586 Accepted
[Sun Oct  4 04:30:57 2026] 127.0.0.1:57586 Closing
[Sun Oct  4 04:38:46 2026] 127.0.0.1:41164 Accepted
[Sun Oct  4 04:38:46 2026] 127.0.0.1:41164 Closing
[Sun Oct  4 04:38:53 2026] 127.0.0.1:41170 Accepted
[Sun Oct  4 04:38:53 2026] 127.0.0.1:41170 Closing
[Sun Oct  4 04:38:55 2026] 127.0.0.1:48136 Accepted
[Sun Oct  4 04:38:55 2026] 127.0.0.1:48136 Closing
[Sun Oct  4 04:39:25 2026] 127.0.0.1:49518 Accepted
[Sun Oct  4 04:39:25 2026] 127.0.0.1:49518 Closing
[Sun Oct  4 04:39:25 2026] 127.0.0.1:49530 Accepted
[Sun Oct  4 04:39:25 2026] 127.0.0.1:49530 Closing
[Sun Oct  4 04:39:26 2026] 127.0.0.1:49532 Accepted
[Sun Oct  4 04:39:26 2026] 127.0.0.1:49532 Closing
[Sun Oct  4 04:40:15 2026] 127.0.0.1:60902 Accepted
[Sun Oct  4 04:40:15 2026] 127.0.0.1:60902 Closing
[Sun Oct  4 04:44:26 2026] 127.0.0.1:46474 Accepted
[Sun Oct  4 04:44:26 2026] 127.0.0.1:46474 Closing
[Sun Oct  4 04:44:26 2026] 127.0.0.1:46482 Accepted