Gladex Agent Logs

test_reduced_motion.php
test_system_status_red_watch.sh
79:add A3  is          git        "## main...origin/main"                      "tree clean AND in sync with origin/main"
91:add A15 contains    sys        "[OK] clean"                                 "git-tree reports a clean tree"
> build · ling-3.1-flash-free
> build · ling-3.1-flash-free
> build · ling-3.1-flash-free
> build · ling-3.1-flash-free
→ Read repo/tools/red-watch 
> build · ling-3.1-flash-free
red-watch runs `tests/test_gladex_monitor.sh` as its monitor — so a "red-watch row" = a new lettered assertion in that suite. Let me read it.
→ Read repo/tests/test_gladex_monitor.sh 
> build · ling-3.1-flash-free
$ cd /data/repo && grep -n "red.watch\|redwatch" tools/system-status | head -20; echo "=== test_red_watch.sh head ==="; head -60 tests/test_red_watch.sh
299:            echo "red-watch verdicts (the last verdict tools/red-watch recorded, read from"
304:            echo "                          red-watch saw a red tree. ALWAYS a warning and never"
305:            echo "                          an error: red-watch is warn-only by construction and"
318:            echo "Test hooks (env), red-watch row:"
320:            echo "                          /data/agent-logs/red-watch.state - absolute, not"
1070:# red-watch's last recorded verdict — queue item (117).
1072:# `tools/red-watch` runs `tests/test_gladex_monitor.sh` every 15 minutes (cron
1073:# `*/15`) and writes its verdict to /data/agent-logs/red-watch.state - outside
1075:# watches. Measured before this row existed: `grep -rln "red-watch" /data
1079:# reached only a human who opened `red-watch-ALERTS.log` by hand. That is
1084:# Contract (pinned by tests/test_system_status_red_watch.sh):
1089:#   `state=red` -> warning, ALWAYS a warning, whatever the age: red-watch is
1110:# /data/agent-logs/red-watch.state - deliberately ABSOLUTE, not REPO_DIR
1115:RW_STATE_FILE="${GLADEX_REDWATCH_STATE:-/data/agent-logs/red-watch.state}"
1131:RW_DETAIL="cannot verify: no red-watch state at $RW_STATE_FILE (tools/red-watch has not run)"
1155:        *)         RW_DETAIL="cannot verify: unreadable red-watch state '$RW_STATE_SANE'" ;;
1214:            RW_DETAIL="stale green: last verdict $RW_AGE_TXT old (updated $RW_UPDATED_SANE) - cron rewrites it every 15 min, run tools/red-watch"
1225:STATUS_ITEMS+=("{\"name\":\"red-watch\",\"status\":\"$RW_STATUS\",\"detail\":\"$RW_DETAIL\"}")
=== test_red_watch.sh head ===
#!/usr/bin/env bash
# test_red_watch.sh — hermetic suite for tools/red-watch (queue item (116), [0.4.173]).
#
# WHY THIS SUITE EXISTS
#   tools/red-watch landed in [0.4.172] with NO dedicated suite — the item was
#   closed with eight hand-run probes against stub monitors under /tmp/opencode,
#   plus the generic `--help` / exit-2 / synopsis reading
#   tests/test_registry_coverage.sh gives every tool. Both are real, and both
#   are the shape this repo has already been bitten by: a probe run once by the
#   run that wrote the tool is not a guard, it is a memory. (116) queued exactly
#   this gap instead of papering over it, and the registry's own red-watch
#   section said "no dedicated suite yet — honest gap".
#
# WHAT THIS SUITE PINS (all of it measured against the shipped tool first)
#   A  the argv contract, and — the half that matters — that it is validated
#      BEFORE any I/O: a bad --format or a bad REDWATCH_MONITOR_TIMEOUT never
#      runs the monitor and never creates the log dir.
#   B  a green verdict is QUIET: one dated line, no ALERTS file at all, and a
#      state file carrying state/since/failed/fails/updated.
#   C  a red verdict is WARN-ONLY (exit 0 — the whole point of (114)) and is
#      deduped: onset alerts once, the same red again does not, a CHANGED
#      failing set alerts as RED-SET with the ORIGINAL since epoch kept.
#   D  recovery alerts with a measured duration and restores the state file.
#   E  a check that could not run is the one outcome that is never quiet
#      (exit 3 + CHECK-ERROR) and is never a resting state: the previous
#      verdict is KEPT in the state file, so a later green still reads as a
#      RECOVERY rather than as a transition out of "check-error".
#   F  the JSON channel: exactly 15 keys, exit_code equal to the process exit
#      code, and `ok` reporting the TREE (`state == "green"`) rather than the
#      tool — so a red tree reads `ok: false` on an exit 0.
#   G  the caps and the quoting (12 FAIL lines / 600 chars of alert / 400 of
#      log, `"` flattened to `'`), plus static guards (python3, no network, no
#      DNS write) and the REGISTRY contract — including the correction this
#      suite forced into it: the red-watch section listed 13 of the 15 payload
#      keys and claimed `--help` shows "the module docstring's" exit-code
#      block, which it does not (argparse renders its own usage).
#   H  nine mutants, one per branch, each asserted to have applied and to make
#      THIS suite report the wrong verdict for the claim that branch belongs to.
#
# HERMETICITY
#   Every run points REDWATCH_LOG_DIR at a mktemp sandbox and
#   REDWATCH_MONITOR_BIN at a stub under it; REDWATCH_REPO_DIR is pinned to
#   this repository so the tool never derives a default from a mutant's copy
#   location. The real /data/agent-logs is only READ (a token check at the end
#   proves no stub's FAIL line ever landed there), and no stub, no alert and no
#   state file outside the sandbox is written. No network, no DNS, no service.
#
# Run: bash tests/test_red_watch.sh
#      REDWATCH_TOOL=<blob> bash tests/test_red_watch.sh   # mutant replay
# Mutations: bash tests/test_red_watch.sh --mutations
# Exit codes: 0 = all pass, 1 = at least one failure, 2 = tool missing.
set -u

TESTS_DIR=$(cd "$(dirname "$0")" && pwd)
REPO=$(dirname "$TESTS_DIR")
TOOL="${REDWATCH_TOOL:-$REPO/tools/red-watch}"
REGISTRY="$REPO/tools/REGISTRY.md"
REAL_LOG_DIR="${REDWATCH_REAL_LOG_DIR:-/data/agent-logs}"
[ "${1:-}" = "--mutations" ] && DO_MUTATIONS=1 || DO_MUTATIONS=0
$ cd /data/repo && head -80 tests/test_system_status_red_watch.sh; echo ...; grep -n "assertions\|27\|28" tests/test_system_status_red_watch.sh | head
#!/usr/bin/env bash
# test_system_status_red_watch.sh — the `red-watch` row surfaces the scheduled
# reader's LAST VERDICT, so its alerts have a reader at last.
#
# Queue item (117), CHANGELOG [0.4.174].
#
# One focused claim: `tools/red-watch` writes `RED` / `RED-SET` / `RECOVERED`
# to files OUTSIDE the git tree on purpose, and measured 2026-10-02 nothing
# read them — `grep -rln "red-watch" /data --include='*.sh' --include='*.md'
# --include='*.json' | grep -v '^/data/repo/'` was EMPTY, `tools/healthcheck`
# and `tools/system-status` named it nowhere, and `crontab -l` held only the
# writer — so a red tree was *recorded* and never *seen*. The verdict belongs
# on the dashboard every run, the monitor (probe `sys`/`sysj`) and the reviewer
# already look at.
#
# The verdicts this suite pins, and why each must be exactly what it is:
#   absent/unreadable state file, no `state=` line, an unknown state, an
#                        `updated=` stamp that cannot be aged (absent / not a
#                        date / postdating now) -> WARNING "cannot verify",
#                        never ok: an input that was not read is never a pass.
#   state=green, fresh    -> ok, errors 0, exit 0 — the only passing verdict.
#   state=red             -> WARNING, ALWAYS, errors 0, exit 0. Never an error:
#                        red-watch is warn-only by construction and the
#                        git-tree/queue-source rows grade the tree itself; this
#                        row exists so the verdict is SEEN, not so it fails the
#                        dashboard a second time.
#   state=green, stale    -> WARNING "stale green": the cron job rewrites the
#                        file every 15 min, so an old green is a green about
#                        the past. Staleness only ever downgrades a green —
#                        section 5 proves it never masks a red.
#   in every case        -> errors 0, overall ok, exit 0: a warn-only reader
#                        must never be able to fail the tool it reads into.
#
# Hermetic: the children this suite does not pin are stubbed on PATH and the
# rows that would probe the live box are pointed at absent paths (the pattern
# of tests/test_system_status_staged.sh), every run is handed a sandbox repo
# through GLADEX_REPO_DIR, and the state file is always a fixture path this
# suite writes — the LIVE /data/agent-logs/red-watch.state is never read here,
# and section 13 asserts the default path is absolute by reading the SOURCE
# instead.
#
# Run: bash tests/test_system_status_red_watch.sh
# Exit codes: 0 = all pass, 1 = at least one failure.
set -u

REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
TOOL="${SYSTEM_STATUS_BIN:-$REPO/tools/system-status}"

PASS=0
FAIL=0
ok()  { PASS=$((PASS+1)); echo "  ok   - $1"; }
bad() { FAIL=$((FAIL+1)); echo "  FAIL - $1"; }

[ -f "$TOOL" ] || { echo "SKIP: tool not found: $TOOL"; exit 0; }
command -v git     >/dev/null 2>&1 || { echo "SKIP: git not installed"; exit 0; }
command -v python3 >/dev/null 2>&1 || { echo "SKIP: python3 not installed"; exit 0; }

SANDBOX="$(mktemp -d /tmp/gtrw-test.XXXXXX)" || exit 1
trap 'rm -rf "$SANDBOX"' EXIT INT TERM

# ---------------------------------------------------------------- stubs -----
# Not under test: stubbing keeps the suite fast, offline and deterministic.
STUB="$SANDBOX/bin"
mkdir -p "$STUB"
printf '#!/bin/sh\necho active\n'   > "$STUB/systemctl"
printf '#!/bin/sh\nprintf 200\n'    > "$STUB/curl"
printf '#!/bin/sh\necho "notAfter=Dec 22 12:00:00 2026 GMT"\n' > "$STUB/openssl"
# dig: every check green (live-accurate fixtures from test_system_status_mx_soa.sh)
cat > "$STUB/dig" <<'STUBDIG'
#!/bin/sh
rtype=""
for a in "$@"; do case "$a" in @*|+*) ;; [A-Z]*) rtype="$a" ;; esac; done
case "$rtype" in
  A)   echo "77.90.15.49" ;;
  MX)  echo "10 gladex.de." ;;
  SOA) echo "a.misconfigured.dns.server.invalid. hostmaster.gladex.de. 2026092305 10800 3600 604800 3600" ;;
  *)   : ;;
esac
STUBDIG
printf '#!/bin/sh\nexit 0\n'        > "$STUB/go"
...
188:# five assertions every verdict here shares: status, needle, errors 0,
> build · ling-3.1-flash-free
$ cd /data/repo && grep -n "gladex_monitor" tools/REGISTRY.md | head; echo "=== REGISTRY section ==="; awk '/^## gladex-monitor/,/^## [a-z]/' tools/REGISTRY.md | head -40
1345:  `tests/test_gladex_monitor.sh` on that schedule and writes
5424:and `tests/test_gladex_monitor.sh` both reported it, and **nothing on this box
5474:**What it runs**: `bash <repo>/tests/test_gladex_monitor.sh` — the repo's own
5547:  `/data/agent-logs/`, verdict matching `tests/test_gladex_monitor.sh`
=== REGISTRY section ===
$ cd /data/repo && echo "T=$(git log -1 --format=%H -- 'tests/*.sh')"; git log -1 --format='%h %s' -- 'tests/*.sh'; echo "V=$(git log -1 --format=%H --extended-regexp --grep='\[[0-9]+\.[0-9]+\.[0-9]+\]')"; git log -1 --format='%h %s' --extended-regexp --grep='\[[0-9]+\.[0-9]+\.[0-9]+\]'; echo "ancestor test:"; git merge-base --is-ancestor $(git log -1 --format=%H -- 'tests/*.sh') $(git log -1 --format=%H --extended-regexp --grep='\[[0-9]+\.[0-9]+\.[0-9]+\]') && echo YES-ancestor || echo NO-not-ancestor; echo "=== free slots ==="; for v in 0.4.193 0.4.194 0.4.195; do echo -n "$v: "; git show HEAD:CHANGELOG.md | grep -c "^## \[$v\]"; done
T=930390b2844831a4d6eacf834075b6e2b48bab98
930390b run 14: 2026-10-04T02:18Z model=opencode/ling-3.1-flash-free
V=5d7bb8f4c7c15638bc0c51c3e835ca84114971e2
5d7bb8f run close 2026-10-04T01:50Z: (128) 9th of 9 landed as [0.4.192] — tests/test_immich_roundtrip.sh SB_ALL ledger + section Z (suite 150 -> 160, the last of the four named near-zero suites, third where the two leak readers agree: standing inventory imrt.* 0 glob matches vs 0 directories, read find -type d first — (131)'s last owed recount); commit #1 f4b908a carries CHANGELOG [0.4.192] + REGISTRY 150->160 and Status 150/0->160/0 + tests/test_immich_roundtrip.sh + this run's PROGRESS entry appended BEFORE it (CLOSE rule 1), through the hook legitimately (private-index probe e3415ba8 -> repo-lint --sha exit 0, 180 files clean, 197 headings/197 unique, 7,825 citations, 0 missing, 45 go modules compile clean), no --no-verify, no GLADEX_SKIP_COMMIT_GATE; the commit also carried, attributed, the concurrent ling session's staged (131) entry (shared index, same file, nothing of theirs dropped — their own close d18ac6b verifies the carry from their side and records their repo-lint refusal while [0.4.192] did not yet exist as a heading); push 3498b2f..f4b908a origin/main...HEAD 0 0; closing gates — queue-source-check exit 0 one queue [0.4.192] pointer-only 111 items/148 sections/153 bullets, inbox-status exit 0 nothing owed (0/0 unread, INBOX 79/79/0, last reply dev 142/prod 107, thread never touched), red-watch GREEN 27/0 alert none (single pre-push red was the known A3 committed-but-unpushed shape, cleared by the push, measured after not predicted), healthcheck 200/200 0.4.28 dev+prod, budget 2026-10 5.00/0.00/5.00 warnings [], crontab 2 lines, census 24 tools/36 sections/85 suites (- Live: 85 untouched), registry-coverage reader on this tree 455/0; STEP 0 re-read 0/0, thread never touched; no app/src/php change, no promote, prod 0.4.28, no suite added, no service restart ((99) stands), no cron change, zero DNS writes, no mail, spend 0.00; this close carries the closing append BEFORE commit #2 (CLOSE rule 2); (128) now 9 of 9 COMPLETE, (131) CLOSED — next run: (93)/(97)(a) if the investor has answered, otherwise (130)
ancestor test:
NO-not-ancestor
=== free slots ===
0.4.193: 0
0.4.194: 0
0.4.195: 0
> build · ling-3.1-flash-free
> build · ling-3.1-flash-free
> build · ling-3.1-flash-free
> build · ling-3.1-flash-free
Error: Rate limit exceeded. Please try again later.
exit=1

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

[Sun Oct  4 05:30:55 2026] 127.0.0.1:50662 Accepted
[Sun Oct  4 05:30:55 2026] 127.0.0.1:50662 Closing
[Sun Oct  4 05:30:55 2026] 127.0.0.1:50672 Accepted
[Sun Oct  4 05:30:55 2026] 127.0.0.1:50672 Closing
[Sun Oct  4 05:30:55 2026] 127.0.0.1:50684 Accepted
[Sun Oct  4 05:30:55 2026] 127.0.0.1:50684 Closing
[Sun Oct  4 05:30:55 2026] 127.0.0.1:50698 Accepted
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50698 Closing
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50700 Accepted
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50700 Closing
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50708 Accepted
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50708 Closing
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50716 Accepted
[Sun Oct  4 05:30:56 2026] 127.0.0.1:50716 Closing
[Sun Oct  4 05:32:47 2026] 127.0.0.1:42408 Accepted
[Sun Oct  4 05:32:47 2026] 127.0.0.1:42408 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58138 Accepted
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58138 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58154 Accepted
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58154 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58156 Accepted
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58156 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58170 Accepted
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58170 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58174 Accepted
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58174 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58182 Accepted
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58182 Closing
[Sun Oct  4 05:33:34 2026] 127.0.0.1:58188 Accepted
[Sun Oct  4 05:33:35 2026] 127.0.0.1:58188 Closing
[Sun Oct  4 05:33:35 2026] 127.0.0.1:40480 Accepted
[Sun Oct  4 05:33:35 2026] 127.0.0.1:40480 Closing
[Sun Oct  4 05:33:35 2026] 127.0.0.1:40484 Accepted
[Sun Oct  4 05:33:35 2026] 127.0.0.1:40484 Closing
[Sun Oct  4 05:33:35 2026] 127.0.0.1:40486 Accepted
[Sun Oct  4 05:33:35 2026] 127.0.0.1:40486 Closing
[Sun Oct  4 05:33:59 2026] 127.0.0.1:53828 Accepted
[Sun Oct  4 05:34:00 2026] 127.0.0.1:53828 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44418 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44418 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44434 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44434 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44440 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44440 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44444 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44444 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44454 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44454 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44464 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44464 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44468 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44468 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44472 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44472 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44486 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44486 Closing
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44496 Accepted
[Sun Oct  4 05:34:21 2026] 127.0.0.1:44496 Closing
[Sun Oct  4 05:34:24 2026] 127.0.0.1:44502 Accepted