Gladex Agent Logs

# nowhere (err_int), output the JSON reader cannot parse (garbage), and an
# undocumented checker exit (rc99). Each is asserted present-WHAT-belongs and
# absent-WHAT-must-not — a diagnosis that matches everything diagnoses nothing
# — and the pair is then asserted in the OTHER direction on the three NAMED
# branches, so the two paths cannot collapse into one another.
echo "== unclassified checker failures get the default arm's own diagnosis =="
for scen in err_int garbage rc99; do
    run_cron "$scen"
    assert_has "$scen → default arm diagnosis names an unclassified cause" \
        "DIAGNOSIS: checker failed without a classified cause (exit " "$(logf)"
    assert_has "$scen → default arm action points at the checker's own output" \
        "ACTION: run 'ip-drift-check --format json' by hand and read its output." "$(logf)"
    assert_lacks "$scen → does not diagnose a vanished A record" \
        "is GONE (query completed, no answer)" "$(logf)"
    assert_lacks "$scen → does not diagnose a dead lookup" \
        "the DNS lookup never completed" "$(logf)"
    assert_lacks "$scen → does not diagnose a missing egress" \
        "no public IPv4 could be determined" "$(logf)"
    # The ACTION half of the same cross-guard (queue item (f9), 2026-10-05):
    # three named arms, three actions, and a default run may quote NONE of
    # them. Read as BODIES, not as `ACTION: `-prefixed needles — a borrow
    # appended after `ALSO: ` has no prefix in front of it, so the prefixed
    # needle misses it entirely (the measured finding of the reader-side twin,
    # tests/test_system_status_ip_drift.sh section 27). Measured before these
    # three lines existed: a default arm whose action ends `ALSO: check egress;
    # no drift verdict can be trusted until this clears.` turned **161 passed /
    # 0 failed** here — the diagnosis direction reddened nothing, because the
    # same block only ever asked about diagnoses.
    assert_lacks "$scen → does not send the operator to the zone" \
        "inspect the zone with 'pdns-api.py records'; do NOT rewrite DNS from here." "$(logf)"
    assert_lacks "$scen → does not order the resolver chain re-checked" \
        "check dig + resolver + WireGuard tunnel before trusting ANY verdict." "$(logf)"
    assert_lacks "$scen → does not declare the egress untrustworthy" \
        "check egress; no drift verdict can be trusted until this clears." "$(logf)"
    # …and the THREE drift-path actions, the half queue item (g) measured
    # missing (probe 2026-10-05, both directions run before a line was
    # written): a check-error run never enters `if [ "$EXIT_CODE" -eq 1 ]`,
    # so none of the three `class=` actions belongs in its log — and a writer
    # that appended all three to the error path still scored **173 passed /
    # 0 failed** here and **564 / 0** in the reader suite (`tests/
    # test_system_status_ip_drift.sh`, run from a copy with `REPO=`/`CRON=`
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "assert_lacks \"\$scen\|assert_lacks \"err_int\|assert_lacks \"garbage\|assert_lacks \"rc99" tests/test_ip_drift_cron.sh; echo "=== drift-path diagnosis assertions and their scenarios"; grep -n -B3 "does not borrow the drift path's tunnel diagnosis\|does not borrow the drift path's unparseable diagnosis\|does not borrow the drift path's public-IP diagnosis" tests/test_ip_drift_cron.sh | grep -n "run_cron\|assert_lacks" | head -40
317:    assert_lacks "$scen → does not diagnose a vanished A record" \
319:    assert_lacks "$scen → does not diagnose a dead lookup" \
321:    assert_lacks "$scen → does not diagnose a missing egress" \
333:    assert_lacks "$scen → does not send the operator to the zone" \
335:    assert_lacks "$scen → does not order the resolver chain re-checked" \
337:    assert_lacks "$scen → does not declare the egress untrustworthy" \
350:    assert_lacks "$scen → does not borrow the drift path's tunnel action" \
352:    assert_lacks "$scen → does not borrow the drift path's public_ip action" \
354:    assert_lacks "$scen → does not borrow the drift path's VPS-verify action" \
=== drift-path diagnosis assertions and their scenarios
2:411-assert_lacks "err_no_a → does not borrow the missing-egress action" \
4:413:assert_lacks "err_no_a → does not borrow the drift path's tunnel diagnosis" \
6:415:assert_lacks "err_no_a → does not borrow the drift path's unparseable diagnosis" \
8:417:assert_lacks "err_no_a → does not borrow the drift path's public-IP diagnosis" \
11:451-assert_lacks "err_dns → does not borrow the missing-egress action" \
13:453:assert_lacks "err_dns → does not borrow the drift path's tunnel diagnosis" \
15:455:assert_lacks "err_dns → does not borrow the drift path's unparseable diagnosis" \
17:457:assert_lacks "err_dns → does not borrow the drift path's public-IP diagnosis" \
20:480-assert_lacks "err_ip → does not borrow the dead-lookup action" \
22:482:assert_lacks "err_ip → does not borrow the drift path's tunnel diagnosis" \
24:484:assert_lacks "err_ip → does not borrow the drift path's unparseable diagnosis" \
26:486:assert_lacks "err_ip → does not borrow the drift path's public-IP diagnosis" \
> build · mimo-v2.6-flash-free
The `case` family is closed. Let me measure what gap is actually left, for the next queue entry:
← Write /tmp/opencode/probe42/gap.py
Created file successfully: /tmp/opencode/probe42/gap.py
> build · mimo-v2.6-flash-free
$ cd /tmp/opencode/probe42 && python3 gap.py
needle occurs exactly once: True [count=1]
bash -n: OK
result: === Results: 280 passed, 0 failed === | rc = 0
FAIL lines: 0
VERDICT: SURVIVED (a real hole)
> build · mimo-v2.6-flash-free
$ cd /data/repo && head -40 tests/test_system_status_ip_drift.sh | grep -n "REPO\|CRON\|Run:" 
(no output)
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "^REPO=\|^CRON=\|IPDRIFT_CRON\|ip-drift-cron\"" tests/test_system_status_ip_drift.sh | head -10; grep -c "" tests/test_system_status_ip_drift.sh
195:REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
807:CRON="$REPO/tools/ip-drift-cron"
2624
> build · mimo-v2.6-flash-free
$ P=/tmp/opencode/probe42/readerfake; rm -rf $P; mkdir -p $P; cp -r /data/repo/tests $P/tests; cp -r /data/repo/tools $P/tools; cp /tmp/opencode/probe42/ip-drift-cron.gap1 $P/tools/ip-drift-cron; chmod +x $P/tools/ip-drift-cron; cd $P && bash tests/test_system_status_ip_drift.sh > /tmp/opencode/probe42/gap1-reader.out 2>&1; echo "rc=$?"; tail -2 /tmp/opencode/probe42/gap1-reader.out; grep -c "FAIL - " /tmp/opencode/probe42/gap1-reader.out; grep "FAIL - " /tmp/opencode/probe42/gap1-reader.out | head -12; echo "=== control (clean tool)"; cp /data/repo/tools/ip-drift-cron $P/tools/ip-drift-cron; chmod +x $P/tools/ip-drift-cron; cd $P && bash tests/test_system_status_ip_drift.sh 2>&1 | tail -2
rc=1

=== Results: 648 passed, 14 failed ===
14
  FAIL - control27B: the plant did not apply
  FAIL - control27B: the original default action is still in the copy ( left)
  FAIL - control27B: a different arm was touched too (got '')
  FAIL - control27B: the borrowing writer does not parse
  FAIL - control27B: the borrowing writer exited '127', so more than the sentence changed
  FAIL - control27B: status is 'error' (got 'warning')
  FAIL - control27B: detail carries 'CHECK-ERROR gladex.de' (got 'cannot verify: no ip-drift log in /tmp/gsip-test.s8vpQz/writer-replay/err_int_borrows_named (tools/ip-drift-cron has not run)')
  FAIL - control27B: errors 1 (got '0')
  FAIL - control27B: overall error (got 'ok')
  FAIL - control27B: exit 1 (got 0)
  FAIL - control27B: the escalation channel did not fire, so the row had nothing to read
  FAIL - control27B: the row differs from the real run ('cannot verify: no ip-drift log in /tmp/gsip-test.s8vpQz/writer-replay/' vs ' CHECK-ERROR gladex.de: code=internal_error detail=unexpected failure ')
=== control (clean tool)

=== Results: 662 passed, 0 failed ===
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "control27B" tests/test_system_status_ip_drift.sh | head; echo "---"; sed -n "$(grep -n 'control27B:' tests/test_system_status_ip_drift.sh | head -1 | cut -d: -f1),+0p" tests/test_system_status_ip_drift.sh; grep -n -A40 "^# control27B\|control27B —" tests/test_system_status_ip_drift.sh | sed -n '1,60p'
1226:    # Queue item (l) — the twin of control26/control27A/control27B, and the one
2397:        && ok "control27B: the plant applied — exactly one occurrence replaced" \
2398:        || bad "control27B: the plant did not apply"
2406:        && ok "control27B: the original one-line default action is gone from the copy" \
2407:        || bad "control27B: the original default action is still in the copy ($LEFT27B left)"
2419:        && ok "control27B: the diagnosis and the three named arms are untouched — the plant is surgical" \
2420:        || bad "control27B: a different arm was touched too (got '$PAIR27B')"
2422:        ok "control27B: the borrowing writer parses (a syntax error would redden for the wrong reason)"
2424:        bad "control27B: the borrowing writer does not parse"
2434:        && ok "control27B: the borrowing writer still exits 3 — the exit status cannot tell the two apart" \
---
        && ok "control27B: the plant applied — exactly one occurrence replaced" \
> build · mimo-v2.6-flash-free
$ cd /data/repo && sed -n '2360,2400p' tests/test_system_status_ip_drift.sh
|| bad "control27A: the defect was not reproduced — the control proves nothing"
    grep -qF "$DEF_BODY27" "$LOG_NA" 2>/dev/null \
        && bad "control27A: the real run already carries it, so this section's assertion cannot fail" \
        || ok "control27A: the real run carries none of it — the assertion is not decorative"
    # The needle that would have missed it, asserted missing: the PREFIXED form
    # does not occur in the borrowing arm's line at all, so a section-26-style
    # prefixed needle would have stayed green over this defect.
    grep -qF "ACTION: run 'ip-drift-check --format json' by hand and read its output." "$MLOG27A" 2>/dev/null \
        && bad "control27A: the prefixed needle matched, so this section had no reason to read bodies" \
        || ok "control27A: the PREFIXED needle misses the borrowed text — the reason this section reads bodies"
    grep -qF "ACTION: inspect the zone with 'pdns-api.py records'; do NOT rewrite DNS from here." "$MLOG27A" 2>/dev/null \
        && ok "control27A: the arm's OWN action is still there under the borrowing — the defect rides along beneath the presence assertion" \
        || bad "control27A: the arm's own action vanished, so something other than a borrow changed"

    # -- the negative control, DIRECTION TWO: a DEFAULT arm that borrows ------
    # The probe that measured this gap, kept as the control: the default arm
    # appends a named arm's instruction to its own string. Presence still
    # passes (substring), section 25's stray loop reads diagnoses, section 26
    # never reads a default run — and now direction two reads it.
    MUT_CRON27B="$WREP/cron-default-carries-named"
    python3 - "$CRON" "$MUT_CRON27B" <<'PY27B'
import sys
src, dst = sys.argv[1], sys.argv[2]
s = open(src).read()
needle = ('    *)\n'
          '        DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
          '        ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;')
n = s.count(needle)
if n != 1:
    raise SystemExit("plant pattern found %d time(s), want exactly 1" % n)
open(dst, 'w').write(s.replace(needle,
    ('    *)\n'
     '        DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
     '        ACTION="run \'ip-drift-check --format json\' by hand and read its output. '
     'ALSO: inspect the zone with \'pdns-api.py records\'; do NOT rewrite DNS from here." ;;'), 1))
PY27B
    [ $? -eq 0 ] && [ -f "$MUT_CRON27B" ] \
        && ok "control27B: the plant applied — exactly one occurrence replaced" \
        || bad "control27B: the plant did not apply"
    LEFT27B=$(python3 - "$MUT_CRON27B" <<'PY27B'
import sys
> build · mimo-v2.6-flash-free
$ cd /data/repo && bash tests/test_gladex_monitor.sh 2>&1 | tail -3; ./tools/red-watch 2>&1 | tail -5; ./tools/system-status 2>&1 | tail -25; echo "rc=$?"
passed: 18
failed: 12
summary: 30 assertions, 18 passed, 12 failed
  FAIL A16 system-status json exits 0 (kind=rc key=sysj want=0)
  FAIL A17 json channel says overall ok (kind=contains key=sysj want="overall":"ok")
  FAIL A18 json channel says zero errors (kind=contains key=sysj want="errors":0)
  log=/data/agent-logs/red-watch-20261006.log
  alerts=/data/agent-logs/red-watch-ALERTS.log
  page-stats                [OK] 200
  page-team                 [OK] 200
  cloud                     [OK] nextcloud 200 installed=true; immich 200 pong; 5/5 containers up
  DNS:gladex.de             [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  DNS:dev.gladex.de         [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  DNS:git.gladex.de         [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  DNS:info.gladex.de        [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  DNS:stats.gladex.de       [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  DNS:team.gladex.de        [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  DNS:log.gladex.de         [OK] 77.90.15.49 on 1.1.1.1+8.8.8.8
  MX:gladex.de              [OK] 10 gladex.de. on 1.1.1.1+8.8.8.8
  SOA:gladex.de             [WARN] serial=2026092602 consistent on 1.1.1.1+8.8.8.8; mname=placeholder (NEEDS-INVESTOR open)
  ip-drift                  [OK] [2026-10-05T11:09:59Z] OK: No drift (egress matches DNS A record) (17h old)
  tls-cert-expiry           [OK] 77d left
  tls-cert-expiry-photos    [OK] 79d left
  investor-messages         [OK] 0 unread dev=0 prod=0
  investor-duty             [OK] owed=0 unread=0 unreplied=0 open=0
  git-tree                  [WARN] 3 uncommitted changes
  red-watch                 [WARN] red, failed=12, since 2026-10-06T04:31:45Z, 1m old; FAIL A3 tree clean AND in sync with origin/main (kind=is key=git) ;; FAIL A4 queue-source-check exits 0 (kind=rc key=qsc) ;; FAIL A5 queue has exactly one home (kind=contains key=qsc) ;; FAIL A6 newest CHANGELOG queue is pointer-only (kind=contains key=qsc) ;; FAIL A7 queue-source-check json exits 0 (kind=rc key=qscj) ;; FAIL A8 queue-source-check reports no violations (kind=contains key=qscj) ;; 
  queue-source              [FAIL] 1 violation(s): [0.4.232] is not named in agent-logs/PROGRESS.md - the newest CHANGELOG entry has no record in the authoritative file
  go-tests                  [OK] passing (worktree)
  go-compile                [OK] 45 module file(s) compile clean (1.762s) (commit 79e6e8d)
  promote-gates             [WARN] not promotable since 2026-10-04T15:00:25Z (1d 14h ago): verdict REFUSED: verdict VERDICT-20261004T150009-promote-footer-version-and-focus-visible.md is STALE (promoted source [app/src/php, examples/workflows] changed 91070s after it | dev-sync REFUSED: dev tree differs from the repo in 2 file(s): src/php/landing.php (differs), src/php/templates.php (differs) \u2014 this promotion copies REPO -> prod, s

Overall: 1 CHECK(S) FAILED
rc=0
> build · mimo-v2.6-flash-free

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

[Tue Oct  6 07:02:03 2026] 127.0.0.1:44936 Accepted
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44936 Closing
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44950 Accepted
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44950 Closing
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44960 Accepted
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44960 Closing
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44968 Accepted
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44968 Closing
[Tue Oct  6 07:02:03 2026] 127.0.0.1:44970 Accepted
[Tue Oct  6 07:02:04 2026] 127.0.0.1:44970 Closing
[Tue Oct  6 07:02:04 2026] 127.0.0.1:44978 Accepted
[Tue Oct  6 07:02:04 2026] 127.0.0.1:44978 Closing
[Tue Oct  6 07:02:04 2026] 127.0.0.1:44990 Accepted
[Tue Oct  6 07:02:04 2026] 127.0.0.1:44990 Closing
[Tue Oct  6 07:02:04 2026] 127.0.0.1:38694 Accepted
[Tue Oct  6 07:02:04 2026] 127.0.0.1:38694 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42556 Accepted
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42556 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42572 Accepted
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42572 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42574 Accepted
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42574 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42576 Accepted
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42576 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42578 Accepted
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42578 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42580 Accepted
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42580 Closing
[Tue Oct  6 07:02:51 2026] 127.0.0.1:42594 Accepted
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42594 Closing
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42606 Accepted
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42606 Closing
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42614 Accepted
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42614 Closing
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42630 Accepted
[Tue Oct  6 07:02:52 2026] 127.0.0.1:42630 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33042 Accepted
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33042 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33044 Accepted
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33044 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33056 Accepted
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33056 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33066 Accepted
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33066 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33078 Accepted
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33078 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33082 Accepted
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33082 Closing
[Tue Oct  6 07:03:39 2026] 127.0.0.1:33098 Accepted
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33098 Closing
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33114 Accepted
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33114 Closing
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33130 Accepted
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33130 Closing
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33142 Accepted
[Tue Oct  6 07:03:40 2026] 127.0.0.1:33142 Closing
[Tue Oct  6 07:04:38 2026] 127.0.0.1:57318 Accepted
[Tue Oct  6 07:04:38 2026] 127.0.0.1:57318 Closing
[Tue Oct  6 07:04:48 2026] 127.0.0.1:38526 Accepted