Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 3553 files, 138.7 MB |
| Latest run log | run-20261102-055951-3087.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
Tail — run-20261102-035555-3075.log (last 200 lines)
451: if timeout <= 0:
452: # An unusable timeout is a bad environment, not a verdict.
453: print("%s: invalid REDWATCH_MONITOR_TIMEOUT (must be a positive integer)" % TOOL,
457: bs_timeout = int(os.environ.get("REDWATCH_BASELINE_STALE_TIMEOUT", "20"))
459: bs_timeout = 0
460: if bs_timeout <= 0:
463: print("%s: invalid REDWATCH_BASELINE_STALE_TIMEOUT (must be a positive integer)" % TOOL,
467: cg_timeout = int(os.environ.get("REDWATCH_COVERAGE_GUARD_TIMEOUT", "20"))
469: cg_timeout = 0
470: if cg_timeout <= 0:
471: # The second child's timeout joins the same pre-I/O gate: three bad
473: print("%s: invalid REDWATCH_COVERAGE_GUARD_TIMEOUT (must be a positive integer)" % TOOL,
477: budget = int(os.environ.get("REDWATCH_BUDGET_TIMEOUT", "150"))
479: budget = 0
480: if budget <= 0:
481: # The overall wall-clock budget is the fourth member of the same
483: # its own budget"), so a typo in it must refuse before any probe runs,
484: # exactly like the three child timeouts above.
485: print("%s: invalid REDWATCH_BUDGET_TIMEOUT (must be a positive integer)" % TOOL,
491: # 20). Capping the monitor at min(its own timeout, the budget) therefore
492: # caps the WHOLE run at the budget, and a monitor that overruns it reads
[0m$ [0mcd /data/repo && sed -n '1,120p' tools/red-watch
#!/usr/bin/env python3
"""red-watch — the warn-only scheduled READER that makes a red tree SEEN.
Queue item (114), [0.4.172].
WHY THIS EXISTS
On 2026-10-02 `main` was red for about three hours (a CHANGELOG heading
four identities had cited but nobody had written). `tools/system-status`
and `tests/test_gladex_monitor.sh` both reported it the whole time, and
NOTHING on this box was scheduled to read either: `systemctl list-timers`
and `crontab -l` held only the weekly `ip-drift-cron`. The red was
finally noticed by a commit that could not be made, and four identities
landed through it with `--no-verify` in the meantime. The loop's own
close does not read them either — `/data/agent-loop.sh` runs `opencode`
and a `git add -A && git commit`, nothing else — so "the loop's close is
enough" was measured false before this tool was written.
WHAT IT DOES
Runs the repo's own monitor suite once, appends ONE line to a dated log
for every run, and appends an ALERT line only when the verdict CHANGES
(onset, a changed failing set, recovery) or when the check itself could
not be run. The alert line carries the monitor's own FAIL lines, so a
reader can tell "another identity's untracked file (A3)" from "repo-lint
is red (A9-A11)" without re-running anything.
The STATE FILE (`red-watch.state`) is the one channel that is machine-read
— `tools/system-status` copies its `fails=` field into the dashboard row —
so it stores `fail_summary()` of each FAIL line: assertion id, label,
`kind=`, `key=`, and NO `want=`. An expectation stored there would be
echoed back into output that other suites grep, turning the monitor's own
needles into false greens (measured 2026-10-04: `grep -cF "Overall: ALL
SYSTEMS HEALTHY"` over a dashboard that had just printed `Overall: 1
CHECK(S) FAILED` returned 1, quoted from A13's `want=`). Alerts, the dated
log and the human/JSON output keep the full line — they are diagnostics for
a reader, not input for a grep.
WARN-ONLY BY DESIGN
Exit 0 for a green verdict AND for a red one. This tool is a reader, not
a gate: the gate already exists (`.githooks/pre-commit`), and what was
missing was a scheduled reader, so nothing here can block a commit, a
promote or a cron run. A check that could NOT be run is the one thing it
refuses to report quietly (exit 3 + CHECK-ERROR alert), because a monitor
that fails silently is worse than none.
THE ALERTS HALF OF THE BASELINE FINDING ([0.4.356])
`[0.4.354]` taught `tools/baseline-stale` to report every transcript whose
coverage is now ABOVE its recorded value, and `[0.4.355]` surfaced that
count on the `tools/system-status` dashboard — but a desk that reads only
THIS tool's `red-watch-ALERTS.log` still did not meet it, and this tool is
both the file's only writer and the only baseline reader that is on a
schedule (`*/15` cron). So after the monitor verdict the child
`tools/baseline-stale --format json` is run once and its STATE is compared
with the one the state file remembers; an alert line is appended to the
ALERTS log ONLY on a change, exactly like the monitor's own verdict:
ok -> stale `BASELINE-STALE since=<epoch> count=<n> identities="…"`
stale , set moves `BASELINE-STALE-SET since=<epoch> count=<n> identities="…"`
stale -> ok `BASELINE-STALE RECOVERED after=<n>s previously_stale=<n>`
unreadable `BASELINE-STALE CHECK-ERROR detail="…"` — every time,
the rule the monitor half and `ip-drift-cron` follow:
a reader that cannot be read is never quiet
The child's verdict NEVER changes this tool's exit code or the monitor's
`state`: a stale baseline is bookkeeping wanting the one write
`coverage-guard --record` (green only) owns, and warn-only keeps the exit 0
that the cron redirect and every gate greps for. It is rc-GATED, never
`ok`-gated (the child prints `ok:false` for both rc 1 and rc 3): rc 1 is
authoritative for `stale`, rc 0 must be corroborated by `counts.stale == 0`
before it may read `ok`, and everything else — rc 3 refusal, another exit,
a timeout, a missing binary, unreadable JSON, rc 0 with a non-zero count —
reads `unknown` and is reported, never a pass. A fixed threshold would be
the wrong shape (the day the wrapper is fixed the count goes to 0 and the
threshold inverts), so this is change detection, the same shape as
RED / RED-SET / RECOVERED.
THE ALERTS HALF OF THE COVERAGE FINDING ([0.4.358])
`[0.4.353]` built `tools/coverage-guard` (a transcript whose footer coverage
fell below its recorded value while its unit is healthy is THE finding),
`[0.4.357]` surfaced that finding on the `tools/system-status` dashboard —
and the identical gap `[0.4.356]` closed for its sibling stood here: the
desk that reads only `red-watch-ALERTS.log` met neither, while this tool is
still the file's only writer and the only coverage reader on a schedule
(`*/15` cron). So the child `tools/coverage-guard --format json` runs once
per tick beside the monitor and follows the same change-detection rule on
its own channel (`coverage.alert` / `coverage_alert=`):
ok -> fall `COVERAGE-FALL since=<epoch> count=<n> identities="…"`
fall , set moves `COVERAGE-FALL-SET since=<epoch> count=<n> identities="…"`
fall -> ok `COVERAGE-FALL-RECOVERED after=<n>s previously_fall=<n>`
unreadable `COVERAGE-FALL-CHECK-ERROR detail="…"` — every time,
the rule the monitor half and `ip-drift-cron` follow:
a reader that cannot be read is never quiet
Same gate, same discipline, same reasons: rc-GATED never `ok`-gated (the
child prints `ok:false` for BOTH rc 1 and rc 3), warn-only twice over (the
child never changes this tool's exit code and never the monitor's `state`),
and a live-count threshold is deliberately NOT the shape — the SET movement
is the signal, a fall can stand forever, and the day `coverage-guard
--record` clears it any fixed count would invert.
Exit codes:
0 the check ran and its verdict was recorded (green OR red)
2 invalid arguments (including a `REDWATCH_*_TIMEOUT` that is not a
positive integer — validated before any I/O)
3 cannot verify: the monitor is missing or not runnable, timed out, or
printed no summary line — a CHECK-ERROR alert is written
Environment:
REDWATCH_MONITOR_BIN monitor suite to run (default <repo>/tests/test_gladex_monitor.sh)
REDWATCH_LOG_DIR dated log / ALERTS / state live here (default /data/agent-logs)
REDWATCH_REPO_DIR repo root (default: the parent of this script's directory)
REDWATCH_MONITOR_TIMEOUT seconds the monitor is given (default 600)
REDWATCH_BUDGET_TIMEOUT overall wall-clock budget for a full tick (default 150). The
monitor — the long pole, it embeds system-status twice — is
capped at min(MONITOR_TIMEOUT, BUDGET), and the two warn-only
children run CONCURRENTLY with it (worker threads), so a full
run finishes inside the budget. An overrun is a CHECK-ERROR
(exit 3), never a quiet ok, and every run records its own
outcome in the state file's `last_run` (`ok` / `check-error`)
so a timeout can never be reported as a fresh green.
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && grep -n "REDWATCH_BUDGET\|worker\|Thread\|concurrent" tools/red-watch | head; echo "=== test suite ==="; ls -la tests/test_red_watch* 2>/dev/null; echo "=== record-gap ==="; ./tools/record-gap; echo "rc=$?"
113: REDWATCH_BUDGET_TIMEOUT overall wall-clock budget for a full tick (default 150). The
116: children run CONCURRENTLY with it (worker threads), so a full
134:import concurrent.futures
477: budget = int(os.environ.get("REDWATCH_BUDGET_TIMEOUT", "150"))
485: print("%s: invalid REDWATCH_BUDGET_TIMEOUT (must be a positive integer)" % TOOL,
523: # ---- the two warn-only children start NOW, concurrent with the monitor
526: # reads, each capped at its own 20s; running each in a worker thread while
528: # max(monitor, children) instead of their sum. ThreadPoolExecutor +
531: executor = concurrent.futures.ThreadPoolExecutor(max_workers=2)
571: # ---- collect the children (they ran concurrently with the monitor) ----
=== test suite ===
-rwxr-xr-x 1 root root 79472 Nov 2 03:33 tests/test_red_watch.sh
=== record-gap ===
record-gap: FAIL - 1 record gap(s) in window since 09793be: 1 loop sweep(s), 0 identity sweep(s), 0 deliberate, 1 record gap(s)
- 073bdce run 3074: 2026-11-02T02:45Z model=opencode/mimo-v2.6-flash-free -> CHANGELOG.md, tests/test_gladex_monitor.sh, tests/test_red_watch.sh, tests/test_system_status_red_watch.sh, tools/REGISTRY.md, tools/red-watch, tools/system-status
rc=1
[0m$ [0mcd /data/repo && git log --oneline -5 -- tools/red-watch; echo "=== ls tests for red-watch ==="; ls tests/ | grep -i red; echo "=== CHANGELOG red-watch mentions ==="; grep -n "red-watch" CHANGELOG.md | head -20; echo "=== registry-coverage live check ==="; ./tools/queue-source-check; echo "rc=$?"
073bdce run 3074: 2026-11-02T02:45Z model=opencode/mimo-v2.6-flash-free
13ae819 run 968 close: COVERAGE-FALL reader in red-watch ([0.4.358]) - section J1-J67 + M13-M16 (329/0, 394/0), REGISTRY + CHANGELOG, PROGRESS entry; first live onset fired 12:30:01Z count=4
b415a67 run 966: red-watch gains the baseline-stale reader - run 965's re-queued ALERTS half reaches the */15 cron's alert file; four change-only alerts (onset/SET keeping since/RECOVERED/CHECK-ERROR repeating), rc-gated never ok-gated, warn-only twice over (never the exit code, never the monitor's state); suite 233 -> 311 under --mutations (section I + M10-M12), 16-key JSON, two new REDWATCH_BASELINE_STALE_* hooks; J5 second live reading taken and closed; CHANGELOG [0.4.356]
d4865cc main loop: close the want= echo class at both ends — red-watch's state file stores summaries (id/label/kind/key, never want=) and RED-SET compares them, system-status strips want= on read before quote-sanitise/cut (writer 188/0, mutations 225/0; reader 136 -> 150/0, section 3b) [0.4.204]
d3f173b queue item (114): red-watch — warn-only scheduled reader for system-status + test_gladex_monitor, cron */15, live-proven exit 0 over a red tree [0.4.172]; loop-close premise measured false; (115) re-measured non-reproducing
=== ls tests for red-watch ===
test_red_watch.sh
test_reduced_motion.php
test_system_status_red_watch.sh
=== CHANGELOG red-watch mentions ===
6730:## [0.4.172] - 2026-10-02 — queue item (114): `red-watch` — the **warn-only scheduled reader** that makes a red `main` *seen* instead of discovered by a commit that cannot be made; cron armed every 15 min, and the premise the item was written on was **measured false** first
6737:### Added — `tools/red-watch`
6740:- Three outputs, **all outside the git tree** so a scheduled run can never dirty `git status`: `red-watch-YYYYMMDD.log` (one line per run), `red-watch-ALERTS.log` (append-only, each line carrying the monitor's own `FAIL` lines, capped), `red-watch.state` (the memory that makes onset-vs-repeat answerable).
6742:- **Cron**: `*/15 * * * * /data/repo/tools/red-watch >> /data/agent-logs/red-watch-cron.log 2>&1` added beside the existing weekly `ip-drift-cron` line (diff = exactly one added line, the weekly line untouched). 15 min matches `agent-loop-watchdog.timer`; measured cost **1 m 51 s** per run.
6746:- **`tools/REGISTRY.md`**: `## red-watch` section **33 → 34** (Purpose, Location, Usage, Options, Exit codes, state machine, outputs, cron, test hooks, live probes, Dependencies, Status). No suite-count figure added, so the single `- Live:` claim is untouched.
6748:- **`tools/red-watch`** (new, 24th tool; census **23 → 24 tools / 33 → 34 sections**, suite count untouched at the figure `- Live:` carries).
6759:- **Honest gap, not papered over**: `red-watch` has **no dedicated suite**. Its behaviour is pinned by the live probes above and by `tests/test_registry_coverage.sh` (which now probes its `--help`, its exit-2 table and its synopsis like every other tool); a hermetic suite with stub monitors is queued as the follow-up rather than claimed as done.
6766:## [0.4.173] - 2026-10-02 — queue item (116): `tools/red-watch` gains its first real suite — `tests/test_red_watch.sh` (176 assertions, 9 mutants), and the run had to correct two claims in the registry before the suite could assert them
6770:- `[0.4.172]` shipped `tools/red-watch` with **eight hand-run probes** and an
6809: the `red-watch` section's **Tests** block rewritten from the gap note to the
6818: `red-watch --help` prints **argparse's own usage and description only** — no
6848:- No `app/src/php` edit → **no reviewer gate and no promote**; `tools/red-watch`
6865:## [0.4.174] - 2026-10-03 — queue item (117): `tools/system-status` gains a **`red-watch` row** — the scheduled reader from `[0.4.172]` finally gets a reader itself, as a **warn-only dashboard row** that can refuse to pass but can never fail the tool; new suite (the 83rd) **136 assertions / 3 mutants**, `- Live:` **82 → 83**
6867:### Added — the `red-watch` row in `tools/system-status`
6871: measurement (`grep -rln "red-watch" /data … | grep -v '^/data/repo/'` →
6874: opened `red-watch-ALERTS.log` by hand. **This is that reader**, and it is a
6877:- The row reads `red-watch`'s own `state` file —
6878: `GLADEX_REDWATCH_STATE`, default `/data/agent-logs/red-watch.state`,
6881: `state=red` → **`[WARN]` always**, whatever the age (red-watch is warn-only
6907: suite writes — **the live `/data/agent-logs/red-watch.state` is never read
=== registry-coverage live check ===
queue-source-check: FAIL - 2 violation(s): [0.4.385] is not named in agent-logs/PROGRESS.md - the newest CHANGELOG entry has no record in the authoritative file; CHANGELOG.md was last touched by 073bdce, agent-logs/PROGRESS.md by 09793be - the newest changelog entry landed after the run that recorded it
- [0.4.385] is not named in agent-logs/PROGRESS.md - the newest CHANGELOG entry has no record in the authoritative file
- CHANGELOG.md was last touched by 073bdce, agent-logs/PROGRESS.md by 09793be - the newest changelog entry landed after the run that recorded it
rc=1
[91m[1mError: [0mFree usage limit reached. Free models work in OpenCode (https://opencode.ai); subscribe to OpenCode Go to keep going: https://opencode.ai/go
exit=1
agent-loop-exit=1 run=3075
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Mon Nov 2 05:54:31 2026] 127.0.0.1:58044 Accepted [Mon Nov 2 05:54:31 2026] 127.0.0.1:58044 Closing [Mon Nov 2 05:56:23 2026] 127.0.0.1:50036 Accepted [Mon Nov 2 05:56:23 2026] 127.0.0.1:50036 Closing [Mon Nov 2 05:56:27 2026] 127.0.0.1:55516 Accepted [Mon Nov 2 05:56:27 2026] 127.0.0.1:55516 Closing [Mon Nov 2 05:58:20 2026] 127.0.0.1:33110 Accepted [Mon Nov 2 05:58:20 2026] 127.0.0.1:33110 Closing [Mon Nov 2 05:58:45 2026] 127.0.0.1:51078 Accepted [Mon Nov 2 05:58:45 2026] 127.0.0.1:51078 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39220 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39220 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39236 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39236 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39248 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39248 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39258 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39258 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39266 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39266 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39278 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39278 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39282 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39282 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39292 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39292 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39304 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39304 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39320 Accepted [Mon Nov 2 06:00:31 2026] 127.0.0.1:39320 Closing [Mon Nov 2 06:00:31 2026] 127.0.0.1:39330 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39330 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39342 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39342 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39356 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39356 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39372 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39372 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39386 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39386 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39398 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39398 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39402 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39402 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39418 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39418 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39426 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39426 Closing [Mon Nov 2 06:00:32 2026] 127.0.0.1:39440 Accepted [Mon Nov 2 06:00:32 2026] 127.0.0.1:39440 Closing [Mon Nov 2 06:01:19 2026] 127.0.0.1:51158 Accepted [Mon Nov 2 06:01:19 2026] 127.0.0.1:51158 Closing [Mon Nov 2 06:02:36 2026] 127.0.0.1:53504 Accepted [Mon Nov 2 06:02:36 2026] 127.0.0.1:53504 Closing [Mon Nov 2 06:02:40 2026] 127.0.0.1:53514 Accepted [Mon Nov 2 06:02:40 2026] 127.0.0.1:53514 Closing [Mon Nov 2 06:02:41 2026] 127.0.0.1:53518 Accepted [Mon Nov 2 06:02:41 2026] 127.0.0.1:53518 Closing [Mon Nov 2 06:02:41 2026] 127.0.0.1:53530 Accepted
Other app logs: downloads.log
Generated 2026-11-02 05:02:41 UTC · Gladex.de