Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 2651 files, 133 MB |
| Latest run log | run-20261026-020948-2185.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
Tail — run-20261026-020948-2185.log (last 200 lines)
[0m$ [0mcd /data/repo && awk 'NR>=3225 && NR<=3236{printf "%d: %.130s\n", NR, $0}' tools/REGISTRY.md; echo "=== coverage-guard bullet end / hooks header"; awk 'NR>=3300 && NR<=3316{printf "%d: %.130s\n", NR, $0}' tools/REGISTRY.md
3225: `grep -cE '^ ok - M2[34]'` → **8** — reproducing
3226: `/tmp/opencode/probe764/teeth764at.txt` exactly (this run's artefact:
3227: `/tmp/opencode/run784teeth/teeth784au.txt`, both outside the repo, never
3228: committed). Family green after the edits: cron **340 / 0** (live-free
3229: **337 / 0**), healthcheck **122 / 0**, `registry_coverage` **464 / 0**,
3230: `queue_source` **278 / 0**, `repo_lint` suite **473 / 0**,
3231: `leak_figure_readers` **41 / 0**, `./tools/repo-lint` → **rc 0** (187 files).
3232:
3233: - Baseline staleness: `baseline-stale` — is `agent-logs/coverage-baseline.json`
3234: still where the tree's footer coverage stands? Asked through the sibling
3235: `tools/baseline-stale`'s own `--format json` (added `[0.4.355]`, run 965
3236: taking run 964's re-queued row: *"a `system-status`/monitor row that surfaces
=== coverage-guard bullet end / hooks header
3300: 600 chars), because it lands in hand-built JSON AND in the renderer's
3301: `grep -o '"detail":"[^"]*"'`. Guarded by
3302: `tests/test_system_status_coverage_guard.sh` (the 120th suite).
3303:
3304: **Unread-count contract (both DBs, since 2026-09-24)**: the investor primarily reads
3305: the **prod** thread (public `https://gladex.de` → Apache `:443` → prod `:8001`), while
3306: dev `:8000` carries a separate thread. The old read touched only
3307: `/opt/startup/dev/data/messages.db`, so an unread row on **prod** still reported
3308: `0 unread` / `ok` — a blind guard on the very check that protects STEP 0 (an
3309: unanswered investor = a failed run). Counts are summed; an unreadable/missing DB
3310: reports `?` and forces `warning`, so it can **never masquerade as `0`**.
3311: `agent_to_investor` rows are never counted — those are our own outgoing messages.
3312:
3313: **Test hooks (env)**:
3314: - `GLADEX_DEV_DB` / `GLADEX_PROD_DB` — point the unread check at fixture DBs.
3315: Defaults are the live `/opt/startup/{dev,prod}/data/messages.db`. Same pattern as
3316: `IDENTITY_REPO`/`IDENTITY_LOG` on `identity-run.sh`.
> build · mimo-v2.6-flash-free
[0m← [0mEdit repo/tools/REGISTRY.md
Index: repo/tools/REGISTRY.md
===================================================================
--- repo/tools/REGISTRY.md
+++ repo/tools/REGISTRY.md
@@ -3300,8 +3300,31 @@
600 chars), because it lands in hand-built JSON AND in the renderer's
`grep -o '"detail":"[^"]*"'`. Guarded by
`tests/test_system_status_coverage_guard.sh` (the 120th suite).
+- Loop outcomes: `loop-outcomes` — is the **agent loop itself** alive? Reads the
+ newest `GLADEX_RUNLOG_WINDOW` (default 40) `run-*.log` files and takes each
+ log's **last line** as its verdict: `exit=<n>` is a COMPLETED run, anything
+ else has no marker (counted as non-zero unless it is the **newest** log and
+ younger than `GLADEX_RUNLOG_STALE`, default 1 h — the in-flight run is
+ excluded, never counted). Streak = consecutive non-zero completed runs inside
+ the window: **`ok`** streak 0 (the detail still names the window's reds) ·
+ **`warning`** `streak=<k>` below `GLADEX_RUNLOG_STREAK` (default 5), a stale
+ green, or `cannot verify:` · **`error`** `streak=<k> …` with k ≥ the
+ threshold, **with `ERRORS++`**, so a dead loop reaches monitor A12 and
+ red-watch's ALERT without anyone reading a dashboard. The rate-limited count
+ in the detail matches **either** provider refusal through the single helper
+ `lo_rate_limited` — `Rate limit exceeded` (the 2026-10-11 wording this row was
+ built on) **or** `Free usage limit reached` (the wording measured 2026-10-26
+ on **39/39** logs of streak 2145..2183, **0/39** carrying the legacy needle:
+ the single-needle count printed `(0 rate-limited)` on a streak whose only
+ content was a provider limit — right about the streak, silent about the
+ cause, `[0.4.375]`). Pinned by `tests/test_system_status_failed_units.sh`
+ §16 (pins a–k, mutants M8–M10; **399 assertions / 0 failed** measured
+ 2026-10-26, 369 before §16k). Hooks: `GLADEX_RUNLOG_DIR` (an explicit value
+ IS the sandbox), `GLADEX_RUNLOG_WINDOW`, `GLADEX_RUNLOG_STREAK`,
+ `GLADEX_RUNLOG_STALE`.
+
**Unread-count contract (both DBs, since 2026-09-24)**: the investor primarily reads
the **prod** thread (public `https://gladex.de` → Apache `:443` → prod `:8001`), while
dev `:8000` carries a separate thread. The old read touched only
`/opt/startup/dev/data/messages.db`, so an unread row on **prod** still reported
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && tail -25 CHANGELOG.md
### Notes
- **Live controls, all measured 2026-10-25 against the real box** (the queue's own three): the **defect case** `dns-verify example.com A` → pre-fix **3** with `errors:[]`, post-fix **4** with `errors:["A: status: SERVFAIL — server did not answer the query"]` (and the default 7-type query → 4); **`--server 1.1.1.1 gladex.de A` → 0** with `{"value":"178.104.11.60"}` (control unmoved); **`--server 203.0.113.1 --timeout 1` → 4** (unreachable, control unmoved); plus the 3-vs-4 split on one name: the nonexistent `this-domain-should-be-available-12345.de` → **4** through this box's SERVFAILing resolver but **3** through 1.1.1.1, whose raw answer is `status: NXDOMAIN` — absence reported only where an answer actually said so. `--format human` quotes the status under `Network errors:` with no `Records:` section. The 2026-10-03 registry line's `this-domain… → exit 3` is kept as its dated measurement and superseded by the 2026-10-25 re-measurement.
- **No promote, and none exists to take**: `tools/dns-verify`, `tests/test_dns_verify.sh`, `tools/REGISTRY.md` and this file are repo scope, not one of `source-sync-check`'s env-synced trees — dev and prod are untouched and stay **0.4.28** (the investor's 2026-10-05 correction stands; `0.4.29` quoted nowhere). STEP 0 paid first and owed nothing: **0 unread** `investor_to_agent` in both DBs, `INBOX.md` 80/80/0, last reply dev 150 / prod 115 — no `agent_to_investor` row written, the apex incident still awaiting the investor's answer, the mia/aylin repin still folded into dev 150. No `systemctl` change, no `rm`, no DNS write, no mail, no `hiring/queue/` write, no paid API, no API key configured, **never `git stash`**, staging by explicit pathspec (never `git add -A` on this shared index). Spend **0.00**, free `*-free` models only.
### Queue
Authoritative next-candidate queue lives in `agent-logs/PROGRESS.md`, in the newest run entry's `Queue — next small step` bullet — it is not duplicated here, and has not been since `[0.4.67]`. To read the live queue, open `PROGRESS.md`; to know whether that is still true, run `tools/queue-source-check`.
## [0.4.374] - 2026-10-26 — `ip-drift-check` stops reading a SERVFAIL as "a query completed and the answer was empty": dig is now asked WITHOUT `+short` (the header is what carries the rcode), so `dns_lookup_failed` and `no_a_records` separate on `status: NOERROR|NXDOMAIN` instead of on rc 0 (suite 134 → 150 assertions on this box, registry figure 136 → 152, +2 mutants M7/M8)
### Fixed
- **The defect, queued by run 1007 from its own observation and measured here**: `tools/ip-drift-check` invoked `dig +short`, and `+short` prints **nothing at all** when the server fails — measured on this box: `dig +short gladex.de A` → empty stdout, **rc 0**, while the raw query says `;; ->>HEADER<<- … status: SERVFAIL, … ANSWER: 0`. `dig_query()` took `rc == 0` as *completed*, so this box's stub resolver was reported as `dns_status: "no_a_records"` = *"a query completed and the answer was empty"* for a query nobody had answered — the exact false-negative `dns-verify` closed the day before (`[0.4.373]`), one tool down the line, and the same class `domain-availability-check` was fixed for in `358558f`. Reproduced on the **live** resolver, both directions: `--dns-server 127.0.0.53` → **`no_a_records`** from the `HEAD` blob, **`dns_lookup_failed`** after the fix; `dig_query('gladex.de','A',None,10)` → pre-fix `([], True)` vs post-fix `([], False)`.
- **Why the queued question ("did the `ok` record come from a different resolver path or from headerless silence?") resolves to *different path*, with the blindness confined to the empty case**: `get_dns_a_record()` tries the authoritative NS first and then falls back to `@8.8.8.8`, and that fallback answers `status: NOERROR, ANSWER: 1` with the real `178.104.11.60` — so the verdict never rested on silence. What the silence *could* do is mislabel the **other** empty case: with every query through the local resolver counted as completed-but-empty, `completed_any` was true with zero answers and the tool would have said `no_a_records` wherever the fallback also returned nothing. Both readings are now recorded in `tools/REGISTRY.md`'s `## ip-drift-check` section (Live re-measurement, 2026-10-26).
- **Fix, in `tools/ip-drift-check`**: (1) dig is invoked **without `+short`** so the full output — header, sections, stats — is what the parser sees (`cmd = ["dig", domain, qtype, "+time=N", "+tries=1"]`, `@server` unchanged); (2) `dig_status()` + `COMPLETED_STATUSES = {NOERROR, NXDOMAIN}` decide `completed` — any other rcode returns `([], False)`, which flows into the **existing** `classify_dns()` and becomes `dns_lookup_failed`; (3) `answer_values()` walks only the `;; ANSWER SECTION:` block for full output (a `NXDOMAIN`'s AUTHORITY SOA and question/stats lines can never become a value), and keeps the bare-value path for a headerless answer, because that shape cannot express an rcode at all. Two invariants from the neighbours were preserved deliberately: the transport gate stays the literal **`if diagnostics:`** (exactly one occurrence — `test_dig_tools_transport.sh`'s mutant **M4** pins that literal) and now filters through `TRANSPORT_RE`, since full output carries structural `;;` comments on every healthy run; and the exit-code table is untouched (0/1/2/3, `drift: null` on every undetermined path) — only what `no_a_records` *means* narrowed.
### Tests
- **`bash tests/test_ip_drift_check.sh` → 150 passed / 0 failed on this box (was 134)**, registry figure **136 → 152** (the figure is the *network-present* count: §9's live arm prints 4 assertions when the live run exits 0 and 2 when the public-IP leg has no egress, which is this box — a 2-assertion gap, not a shortfall). The stub was rewritten to emit dig's **full output shape** — `;; ->>HEADER<<- … status: …`, QUESTION/ANSWER/AUTHORITY sections, footer — because a bare `+short` stub cannot express a SERVFAIL at all; it grew `servfail` and `refused` modes, `nodata` now answers `NXDOMAIN` **with an AUTHORITY SOA** (asserted to come back as `records: []`), and it honours `+short` exactly as real dig does (header suppressed, nothing printed on rc 0). New section **6b** pins the rcode contract: `SERVFAIL`/`REFUSED` → exit 3 with `error_code: dns_lookup_failed`, never `no_a_records`, `dns_records: []`, `drift: null`; dig asked **without** `+short` (asserted against the argv log); the completed NXDOMAIN's SOA never read as an answer.
- **Two new mutants, both precondition-counted surgical**: **M7** removes the rcode classifier (`if status is not None and status not in COMPLETED_STATUSES:` → `if False:`, exactly one occurrence) — `SERVFAIL`/`REFUSED` then read `no_a_records`, i.e. back to the pre-fix verdict, which is what makes M7 this fix's own regression guard; **M8** puts `+short` back on the invocation line (exactly one occurrence) — the stub goes silent exactly as real dig does and the mutant reports `no_a_records`, so a future re-introduction of the flag cannot pass the suite while the fixture stays faithful. M1–M6 and the transport suite's `if diagnostics:` pin all re-verified.
- **The shared transport suite was NOT modified and stays green**: `bash tests/test_dig_tools_transport.sh` → **80 passed / 0 failed** (its `ok` stub still prints bare values, so it exercises the headerless compatibility path end to end; M1–M4 all caught). `php tests/test_ip_drift_safety.php` → **15 passed / 0 failed**. **No new suite file, so the registry's `- Live:` figure stays `125`.**
- **Pre-fix replay of the `+short` blob against the new suite**: `IPDRIFT_CHECK=<that blob> bash tests/test_ip_drift_check.sh` → **139 passed / 6 failed**, the three substantive reds being exactly this target (`SERVFAIL … got 'no_a_records'`, `REFUSED … got 'no_a_records'`, `dig is asked WITHOUT +short`), the other three being replay artifacts (the blob's `--help` prints the blob's own filename; M7/M8's needles do not exist in it).
### Notes
- **Live controls, measured 2026-10-26**: default run → exit 3 `public_ip_unavailable` (this box has no egress to the public IP-echo services — the standing no-DNS-egress class) with the DNS leg unchanged, `dns_status: "ok"`, `dns_records ["178.104.11.60"]`; `--dns-server 127.0.0.53` → **`dns_lookup_failed`** (pre-fix `no_a_records`); `--dns-server 8.8.8.8` → `ok` with the same record; `--help` → 0. **No promote, and none exists to take**: `tools/ip-drift-check`, `tests/test_ip_drift_check.sh`, `tools/REGISTRY.md` and this file are repo scope, not one of `source-sync-check`'s env-synced trees — dev and prod are untouched and stay **0.4.28** (the investor's 2026-10-05 correction stands; `0.4.29` quoted nowhere). STEP 0 paid first and owed nothing: **0 unread** `investor_to_agent` in both DBs, `INBOX.md` 80/80/0, last reply dev 150 / prod 115 — no `agent_to_investor` row written, the apex incident still awaiting the investor's answer, the mia/aylin repin still folded into dev 150. No `systemctl` change, no `rm`, no DNS write of any kind, no mail, no `hiring/queue/` write, no paid API, no API key configured, **never `git stash`**, staging by explicit pathspec (never `git add -A` on this shared index). Spend **0.00**, free `*-free` models only.
### Queue
Authoritative next-candidate queue lives in `agent-logs/PROGRESS.md`, in the newest run entry's `Queue — next small step` bullet — it is not duplicated here, and has not been since `[0.4.67]`. To read the live queue, open `PROGRESS.md`; to know whether that is still true, run `tools/queue-source-check`.
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && cat >> CHANGELOG.md <<'EOF'
## [0.4.375] - 2026-10-26 — `loop-outcomes` stops hiding the CAUSE of a failure streak: the rate-limited count now matches the provider's CURRENT refusal (`Free usage limit reached`) as well as the 2026-10-11 wording (`Rate limit exceeded`), through one helper — measured on the 39-run streak 2145..2183, which was 39/39 the new wording and 0/39 the old (suite 369 → 399 assertions, +1 mutant M10)
### Fixed
- **The queue's own question ("why were 39 consecutive runs non-zero?") answered by measurement, then closed as a reporting defect**: run 1008 handed over `loop-outcomes` red with `streak=39 consecutive non-zero runs 2145..2183 (0 rate-limited), threshold=5`. Counted log by log over that exact range: **39/39 logs carry `Error: Free usage limit reached. Free models work in OpenCode …`, 0/39 carry `Rate limit exceeded`, 39/39 end `exit=1`** — one run per ~10 min from 2026-10-25 18:23:17+01:00 to 2026-10-26 00:50:51+01:00, every one of them `model=opencode/mimo-v2.6-flash-free`, and the loop recovered by itself at run 2184 (01:01:03, `exit=0`, same model). So the streak was a **provider-side free-usage limit, not a defect in this repo** — and the row's own `grep -qF "Rate limit exceeded"`, the needle from the 2026-10-11 incident it was built on, matched **none** of them: the detail printed `(0 rate-limited)`, i.e. right about the streak and silent about the one thing the next reader needs.
- **The fix is a two-sided needle behind ONE helper, not two greps**: `lo_rate_limited()` holds `grep -qF -e "Rate limit exceeded" -e "Free usage limit reached" -- "$1"` and both call sites (the completed-`exit=N` branch and the markerless branch) call it, so the list cannot drift between them. Verified live, same box, same window: **before** (the `HEAD` blob carries zero occurrences of the new wording) `loop-outcomes` → `ok … 1 ok / 38 red, 1 in-flight` with no rate-limited count at all; **after** → `ok … 1 ok / 38 red, 38 rate-limited, 1 in-flight`. The streak verdict logic is untouched — only the cause count moved — so no row's status, no `ERRORS++` and no monitor probe changed.
### Tests
- **`bash tests/test_system_status_failed_units.sh` → 399 passed / 0 failed (was 369)**. `plant_run` grew a third refusal mode printing the provider's **verbatim current wording** (copied out of run 2145's own log) beside the legacy one, and **§16k** pins it in three shapes: five free-usage reds → `error` with `5 rate-limited` and a separate assertion that the detail is **never** `(0 rate-limited)` when every red carried a refusal; the same five with a trailing `exit=0` → the `ok` line still names `5 red, 5 rate-limited` (the window count reads the same list); and a mixed window of both wordings → `5 rate-limited` (the needle is a list, not a string swap).
- **New mutant M10** replaces the helper's two `-e` needles with the legacy one alone (needle count asserted = 1): the mutant reproduces the pre-fix `(0 rate-limited)` on the §16k fixture while the real tool counts 5 on the same bytes, the streak verdict stays `error` in both (the needle changes the *cause* count, never the verdict), and the legacy fixture still counts 5 through the mutant — surgical on the old wording. M8/M9 re-verified.
- **No new suite file**, so the registry's `- Live:` figure stays **125**; no `tests/test_ip_drift_*` or transport suite touched.
### Notes
- **Second defect found while working this one, QUEUED NOT FIXED**: the row reads a log's **last line** as its verdict, so an **in-flight** run whose transcript happens to end with an `exit=N` line is counted as a COMPLETED run. Measured 2026-10-26 02:10 on the live box: the first `system-status` of this run reported `streak=1 consecutive non-zero runs 2185 (0 rate-limited)` — run 2185 is *this* run, then in flight, and the `exit=1` on its last line was the transcript echoing another log's content (`cat` of run 2145 prints its `exit=1`); minutes later the same read was `run 2184 exit=0, streak=0, … 1 in-flight`. A transient false streak that can reach red-watch's ALERT. Fix belongs in the loop's own marker shape (a token a transcript cannot forge, e.g. an `agent-loop-exit=` prefix accepted alongside the legacy `exit=` for the 2,652 historical logs) — **not** folded into this run's step, and `agent-loop.sh` is not to be edited blind.
- **No promote, and none exists to take**: `tools/system-status`, `tests/test_system_status_failed_units.sh`, `tools/REGISTRY.md` and this file are repo scope, not one of `source-sync-check`'s env-synced trees — dev and prod are untouched and stay **0.4.28** (the investor's 2026-10-05 correction stands; `0.4.29` quoted nowhere). STEP 0 paid first and owed nothing: **0 unread** `investor_to_agent` in both DBs, `INBOX.md` 80/80/0, last reply dev 150 / prod 115 — no `agent_to_investor` row written, the apex incident still awaiting the investor's answer, the mia/aylin repin still folded into dev 150. No `systemctl` change, no `rm`, no DNS write of any kind, no mail, no `hiring/queue/` write, no paid API, no API key configured, **never `git stash`**, staging by explicit pathspec (never `git add -A` on this shared index). Spend **0.00**, free `*-free` models only.
### Queue
Authoritative next-candidate queue lives in `agent-logs/PROGRESS.md`, in the newest run entry's `Queue — next small step` bullet — it is not duplicated here, and has not been since `[0.4.67]`. To read the live queue, open `PROGRESS.md`; to know whether that is still true, run `tools/queue-source-check`.
EOF
echo "appended"; tail -3 CHANGELOG.md | head -1; grep -c "^## \[0.4.375\]" CHANGELOG.md
appended
1
> build · mimo-v2.6-flash-free
Edits complete (tool + suite + registry + changelog). Running the affected suites in parallel:
[0m$ [0mcd /data/repo && for s in test_cross_reader_agreement.sh test_system_status_dns.sh test_system_status_go_tests.sh test_system_status_investor_duty.sh; do echo "== $s"; bash tests/$s 2>&1 | tail -3; done
== test_cross_reader_agreement.sh
ok - M6 red-watch is byte-identical after the battery
=== Results: 64 passed, 0 failed ===
== test_system_status_dns.sh
ok - unknown arg exits 2
=== Results: 47 passed, 2 failed ===
== test_system_status_go_tests.sh
ok - M6 caught: failing tests now exit 0 (the health signal goes dark)
=== Results: 91 passed, 1 failed, 0 skipped ===
== test_system_status_investor_duty.sh
ok - M4 child reads paths of its own (sources not forwarded) caught (status='warning' errors='1' overall='error', correct is 'error'/'1'/'error')
=== Results: 42 passed, 8 failed ===
[0m$ [0mcd /data/repo && for s in test_system_status_mx_soa.sh test_system_status_mx_soa_transport.sh test_system_status_tls_expiry.sh; do echo "== $s"; bash tests/$s 2>&1 | tail -3; done
== test_system_status_mx_soa.sh
ok - invalid arguments still exit 2 — distinct from unhealthy=1
=== Results: 28 passed, 3 failed ===
== test_system_status_mx_soa_transport.sh
ok - M4 caught: without classification the single-resolver warning cannot be decided (got 'error')
=== Results: 73 passed, 3 failed ===
== test_system_status_tls_expiry.sh
ok - the real tool was never edited by this suite (mutations ran on copies)
=== Results: 48 passed, 12 failed ===
[0m$ [0mcd /data/repo && for s in test_registry_coverage.sh test_registry_figure_readers.sh; do echo "== $s"; bash tests/$s 2>&1 | tail -3; done
== test_registry_coverage.sh
PASS: I6 accounting, last assertion in the file: the static sites below this header minus this one equal the assertions the section produced - so the total I2 predicted is the total this run prints
=== Results: 497 passed, 0 failed ===
== test_registry_figure_readers.sh
Command exceeded timeout of 900000 ms. Retry with a larger timeout if the command is expected to take longer.
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && echo "=== tls failures"; bash tests/test_system_status_tls_expiry.sh 2>&1 | grep -E "fail" | head -14; echo "=== go_tests failure"; bash tests/test_system_status_go_tests.sh 2>&1 | grep -E "fail" | head -6
=== tls failures
=== Results: 48 passed, 12 failed ===
=== go_tests failure
=== 2. go exits 1 with a FAIL line -> error 'failing: <pkg>', exit 1 ===
ok - test failure is an error
ok - detail starts with 'failing'
ok - detail names the failing package
ok - failing tests fail the tool (exit 1)
ok - a killed run fails the tool (exit 1)
[0m$ [0mcd /data/repo && echo "=== dns failures"; bash tests/test_system_status_dns.sh 2>&1 | grep -E "^( )?fail|FAIL" | head -10; echo "=== investor_duty failures"; bash tests/test_system_status_investor_duty.sh 2>&1 | grep -E "fail" | head -12
=== dns failures
FAIL - healthy run exits 0 (got 1)
FAIL - a single-resolver reading must not fail the tool (got exit 1)
=== investor_duty failures
=== 1. D1: an OPEN duty is an ERROR, and it fails the tool ===
FAIL - human renderer counts exactly 1 failed check
FAIL - M2 an open duty does not fail the tool NOT caught: status='error' errors='1' overall='error' (identical to correct)
=== Results: 42 passed, 8 failed ===
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && git status --porcelain; echo "=== other writers? recent commits"; git log --oneline -3; echo "=== concurrent changes to system-status?"; git diff --stat
M CHANGELOG.md
M tests/test_system_status_failed_units.sh
M tools/REGISTRY.md
M tools/system-status
=== other writers? recent commits
82ae58a docs: run 1008 close-correction — post-push readings on b9a9566
b9a9566 fix/tools: ip-drift-check asks dig WITHOUT +short and classifies on the header rcode — a SERVFAIL is dns_lookup_failed, never no_a_records [0.4.374]
fbfcdca identity sofia shift 2026-10-25T17:56Z model=opencode/nemotron-3-ultra-free
=== concurrent changes to system-status?
CHANGELOG.md | 18 ++++++
tests/test_system_status_failed_units.sh | 105 +++++++++++++++++++++++++++++--
tools/REGISTRY.md | 23 +++++++
tools/system-status | 42 ++++++++++---
4 files changed, 175 insertions(+), 13 deletions(-)
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Mon Oct 26 02:30:32 2026] 127.0.0.1:35382 Accepted [Mon Oct 26 02:30:32 2026] 127.0.0.1:35382 Closing [Mon Oct 26 02:30:32 2026] 127.0.0.1:35390 Accepted [Mon Oct 26 02:30:32 2026] 127.0.0.1:35390 Closing [Mon Oct 26 02:30:32 2026] 127.0.0.1:35400 Accepted [Mon Oct 26 02:30:32 2026] 127.0.0.1:35400 Closing [Mon Oct 26 02:30:32 2026] 127.0.0.1:35414 Accepted [Mon Oct 26 02:30:32 2026] 127.0.0.1:35414 Closing [Mon Oct 26 02:30:32 2026] 127.0.0.1:35418 Accepted [Mon Oct 26 02:30:32 2026] 127.0.0.1:35418 Closing [Mon Oct 26 02:30:32 2026] 127.0.0.1:35422 Accepted [Mon Oct 26 02:30:33 2026] 127.0.0.1:35422 Closing [Mon Oct 26 02:30:33 2026] 127.0.0.1:35438 Accepted [Mon Oct 26 02:30:33 2026] 127.0.0.1:35438 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36598 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36598 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36602 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36602 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36612 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36612 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36618 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36618 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36634 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36634 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36650 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36650 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36666 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36666 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36668 Accepted [Mon Oct 26 02:31:53 2026] 127.0.0.1:36668 Closing [Mon Oct 26 02:31:53 2026] 127.0.0.1:36684 Accepted [Mon Oct 26 02:31:54 2026] 127.0.0.1:36684 Closing [Mon Oct 26 02:31:54 2026] 127.0.0.1:36696 Accepted [Mon Oct 26 02:31:54 2026] 127.0.0.1:36696 Closing [Mon Oct 26 02:35:00 2026] 127.0.0.1:33858 Accepted [Mon Oct 26 02:35:00 2026] 127.0.0.1:33858 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46722 Accepted [Mon Oct 26 02:45:27 2026] 127.0.0.1:46722 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46738 Accepted [Mon Oct 26 02:45:27 2026] 127.0.0.1:46738 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46746 Accepted [Mon Oct 26 02:45:27 2026] 127.0.0.1:46746 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46754 Accepted [Mon Oct 26 02:45:27 2026] 127.0.0.1:46754 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46766 Accepted [Mon Oct 26 02:45:27 2026] 127.0.0.1:46766 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46782 Accepted [Mon Oct 26 02:45:27 2026] 127.0.0.1:46782 Closing [Mon Oct 26 02:45:27 2026] 127.0.0.1:46796 Accepted [Mon Oct 26 02:45:28 2026] 127.0.0.1:46796 Closing [Mon Oct 26 02:45:28 2026] 127.0.0.1:46798 Accepted [Mon Oct 26 02:45:28 2026] 127.0.0.1:46798 Closing [Mon Oct 26 02:45:28 2026] 127.0.0.1:46804 Accepted [Mon Oct 26 02:45:28 2026] 127.0.0.1:46804 Closing [Mon Oct 26 02:45:28 2026] 127.0.0.1:46814 Accepted [Mon Oct 26 02:45:28 2026] 127.0.0.1:46814 Closing [Mon Oct 26 02:46:36 2026] 127.0.0.1:56910 Accepted [Mon Oct 26 02:46:36 2026] 127.0.0.1:56910 Closing [Mon Oct 26 02:46:36 2026] 127.0.0.1:56918 Accepted
Other app logs: downloads.log
Generated 2026-10-26 01:46:36 UTC · Gladex.de