Gladex Agent Logs

  FAIL - drift detail names the resolver (got 'got '10 gladex.de.'@1.1.1.1 '10 mail.gladex.de.'@8.8.8.8, expected 10 gladex.de.')
  FAIL - detail distinguishes answered-none from unreachable (got 'got ';; communications error to 1.1.1.1#53: timed out ;; communications error to 1.1.1.1#53: timed out ;; communications error to 1.1.1.1#53: timed out ;; no servers could be reached'@1.1.1.1 'none'@8.8.8.8, expected 10 gladex.de.')
  FAIL - one reachable SOA witness should warn (got 'error' — pre-fix this was 'error')
  FAIL - SOA detail names the reading (got 'insane serial 'error'@1.1.1.1 '2026092602'@8.8.8.8 (want numeric >0)')
  FAIL - single-resolver SOA should exit 0 (got 1)
  FAIL - mirror SOA should warn
  FAIL - placeholder warning still fires (got 'insane serial 'error'@1.1.1.1 '2026092602'@8.8.8.8 (want numeric >0)')
  FAIL - detail carries SOA UNVERIFIED (got 'insane serial 'error'@1.1.1.1 'error'@8.8.8.8 (want numeric >0)')
  FAIL - an unreachable resolver is not an insane serial
  FAIL - detail says the other resolver could not be asked (got 'insane serial 'error'@1.1.1.1 'soon'@8.8.8.8 (want numeric >0)')
  FAIL - detail is UNVERIFIED (got 'no SOA answer on 1.1.1.1/8.8.8.8')
  FAIL - dns_query filters ';;' before paste (got: )
  FAIL - check_mx no longer reads dig directly (the classifier is the only reader)
  FAIL - check_soa_serial no longer reads dig directly
  FAIL - check_mx asks 1.1.1.1 through dns_query
  FAIL - check_mx asks 8.8.8.8 through dns_query
  FAIL - check_soa_serial asks 1.1.1.1 through dns_query
  FAIL - check_soa_serial asks 8.8.8.8 through dns_query
  FAIL - the SOA single-witness branch is in the source
  FAIL - the MX both-unreachable branch is in the source
  FAIL - --help has an mx/soa verdicts section
  FAIL - --help states the MX UNVERIFIED verdict
  FAIL - M2 mutation setup: could not plant the single-witness mutant
  FAIL - M3 mutation setup: could not plant the SOA-witness mutant
=== Results: 38 passed, 35 failed ===
suite rc=0
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "^## system-status" -A 80 tools/REGISTRY.md | head -110
968:## system-status
969-
970-**Purpose**: Quick overview of the entire Gladex system health — services, endpoints, DNS (incl. MX/SOA), messages, git, tests.
971-
972-**Location**: `/data/repo/tools/system-status` (also via the `/data/tools` symlink —
973-`SCRIPT_DIR` resolves the script path with `realpath`, so through the link the repo
974-root is `/data/repo`, not the link's parent)
975-
976-**Usage**:
977-```
978-system-status [--format human|json] [--help]
979-```
980-
981-**Options**:
982-- `--format human|json` - Output format (default: human)
983-- `--help` - Show help message
984-
985-**Exit codes**:
986-- `0` - All systems healthy
987-- `1` - One or more systems unhealthy
988-- `2` - Invalid arguments
989-
990-**Checks performed (35 total)**:
991-- systemd services: investor-app-dev, investor-app-prod, git-daemon, apache2, certbot.timer, watchdog
992-- HTTP endpoints: /healthz (dev+prod), /download/gladex, /api/health, /api/uptime, /api/endpoints, /api/changelog, /api/stats
993-- DNS records: gladex.de, dev/git/info/stats/team/log subdomains → 77.90.15.49 —
994-  **asked on BOTH 1.1.1.1 and 8.8.8.8** through `dns_query`, which classifies the
995-  reading before anything reads it: `answer` (a usable line — including the empty
996-  one, since "no such record" is an answer: empty value, exit 0) vs `transport`
997-  (dig's `;;` diagnostics and/or a failed dig: no reading at all). dig prints
998-  those diagnostics to **stdout** and exits 9, so a raw `$(dig ... 2>/dev/null)`
999-  reads them as the record's value — measured 2026-09-26, one blip in eight runs
1000-  published `got ;; communications error to 1.1.1.1#53: timed out, expected
1001-  77.90.15.49` and took the tool to exit 1. Verdicts: `ok` `<ip> on
1002-  1.1.1.1+8.8.8.8` (every answering resolver agreed) · **`warning`** `<ip> on
1003-  1.1.1.1 (8.8.8.8 unreachable - single-resolver reading)` — one witness, loud in
1004-  the detail, **exit 0**, never silently green · **`error`** `no answer from
1005-  1.1.1.1/8.8.8.8 (both resolvers unreachable) … DNS UNVERIFIED` — nothing
1006-  measured, nothing claimed, nothing passes · **`error`** `<resolver> answered
1007-  '<ip>', expected <ip>` — drift is still drift, and the detail now says which
1008-  resolver said it. `dig`'s exit code never leaks past the tool (documented:
1009-  0/1/2).
1010-- MX record: `MX:gladex.de` must be `10 gladex.de.` on **BOTH** 1.1.1.1 and 8.8.8.8
1011-  (propagation check; standing investor rule 2026-09-23 — adopted MX must stay
1012-  verified). Any other/missing answer on either resolver = **error, exit 1** —
1013-  a wrong MX on a public resolver is a mail outage, not a warning.
1014-- SOA serial: `SOA:gladex.de` — serial must be numeric, >0 and **identical** on
1015-  1.1.1.1/8.8.8.8 (split = **warning** "propagation lag", transient by definition;
1016-  garbage/missing serial = **error**). The known `MNAME` placeholder
1017-  (`a.misconfigured.dns.server.invalid.`, provider-panel-only fix, open
1018-  NEEDS-INVESTOR) is surfaced as a **warning** in the detail and clears itself
1019-  once the panel value changes.
1020-- Cloud stacks: `cloud` — Nextcloud `status.php` + Immich `/api/server/ping`
1021-  (HTTP) + `docker ps --all` state/`RestartCount` (the two newest production
1022-  surfaces could otherwise be dead under an ALL-SYSTEMS-HEALTHY dashboard)
1023-- TLS cert expiry: **ONE ROW PER LIVE LET'S ENCRYPT LINEAGE** —
1024-  `tls-cert-expiry` (SNI `gladex.de`, the 7-SAN lineage) and
1025-  `tls-cert-expiry-photos` (SNI `photos.gladex.de`, the `cloud.`+`photos.`
1026-  lineage issued 2026-09-26). Date rules unchanged: warn <30d, error <7d, `no
1027-  cert` error when the handshake offers nothing. Each row additionally proves
1028-  the certificate it read **covers its own SNI**: Apache answers with the
1029-  default vhost's cert when nothing matches a name (the `*.gladex.de` alias
1030-  swallowing `photos.` until 2026-09-26), so a date off the wrong cert is a
1031-  date about the wrong lineage. SAN list readable and covering → the date
1032-  decides; readable and NOT covering → **error** naming the served SANs;
1033-  unreadable → the date decides with `identity not checked` appended (never a
1034-  silent pass, never a red on a host we could not ask). Wildcards cover exactly
1035-  one label above their domain.
1036-- Investor messages: unread `investor_to_agent` rows counted across **BOTH** DBs — detail `N unread dev=X prod=Y`
1037-- Git tree: clean/dirty
1038-- Next-candidate queue: `queue-source` carries `queue-source-check --format json
1039-  --repo`'s verdict — CHANGELOG's newest `### Queue` section must be a POINTER at
1040-  `agent-logs/PROGRESS.md` (the one authoritative list) and the 111 historical
1041-  item lines must be unchanged. child `0` → **ok**, `1` → **error + `ERRORS++`**
1042-  (a second list, a dropped pointer or an empty authoritative list turns the
1043-  dashboard red), `3` → **warning `cannot verify`** (no `CHANGELOG.md`, no
1044-  `### Queue` section, no `agent-logs/PROGRESS.md`) — never ok, never an error on
1045-  a tree that was not checked. `detail` is the child's own, already sanitised of
1046-  `"` and `\` so the hand-built row stays valid JSON.
1047-- Go tests: verdict from the **exit code** of `go test ./...` over the whole
1048-  module (worktree) — see the verdict table below
> build · mimo-v2.6-flash-free
$ cd /data/repo && sed -n '1048,1130p' tools/REGISTRY.md
module (worktree) — see the verdict table below
- Go compile: `repo-lint --format json --sha HEAD`'s `go_compile` verdict
  (the COMMIT) — see the verdict table below
- Promote gates: `promote-dev-to-prod --dry-run --force --format json` +
  `GLADEX_GATE_PROBE=1` — "can we ship right now?" in one dashboard read —
  see the verdict table below

**Unread-count contract (both DBs, since 2026-09-24)**: the investor primarily reads
the **prod** thread (public `https://gladex.de` → Apache `:443` → prod `:8001`), while
dev `:8000` carries a separate thread. The old read touched only
`/opt/startup/dev/data/messages.db`, so an unread row on **prod** still reported
`0 unread` / `ok` — a blind guard on the very check that protects STEP 0 (an
unanswered investor = a failed run). Counts are summed; an unreadable/missing DB
reports `?` and forces `warning`, so it can **never masquerade as `0`**.
`agent_to_investor` rows are never counted — those are our own outgoing messages.

**Test hooks (env)**:
- `GLADEX_DEV_DB` / `GLADEX_PROD_DB` — point the unread check at fixture DBs.
  Defaults are the live `/opt/startup/{dev,prod}/data/messages.db`. Same pattern as
  `IDENTITY_REPO`/`IDENTITY_LOG` on `identity-run.sh`.
- `GLADEX_REPO_DIR` — repo the git/Go checks report on (default: parent dir of the
  script). Lets hermetic suites run **mutated copies** from a sandbox path; without
  it a copy under `/tmp` computes a `REPO_DIR` with no `.git` and dies at the git
  check under `set -e` (silent empty output → vacuous mutation checks).
- `GLADEX_GO_BIN` (default `go`) / `GLADEX_GO_TIMEOUT` (default `120`) /
  `GLADEX_GO_GOPATH` (default `/tmp/gopath`) / `GLADEX_GO_GOCACHE` (default
  `/tmp/gocache`) — the go-tests check only. `GOMODCACHE` is
  `$GLADEX_GO_GOPATH/pkg/mod`. All four are documented in `--help`.
- `GLADEX_REPO_LINT_BIN` (default: the sibling `repo-lint`, resolved from this
  script's own directory) / `GLADEX_GO_LINT_TIMEOUT` (default `120`) — the
  go-compile check only. The first is what keeps a hermetic suite off the real
  tree: every `system-status` run lints a real commit, so a suite that does not
  test this row stubs the child (all three older suites do).
- `GLADEX_PROMOTE_BIN` (default: `<reported repo>/tools/promote-dev-to-prod`,
  i.e. resolved through `GLADEX_REPO_DIR`, not through the script's own dir) /
  `GLADEX_PROMOTE_TIMEOUT` (default `120`) — the promote-gates check only. The
  default is what keeps a hermetic suite off the real tool: a suite that does
  not test this row points it at a non-existent path and gets
  `cannot verify` in well under a second (all four older suites do).
- `GLADEX_QUEUE_SOURCE_BIN` (default: the sibling `queue-source-check`, resolved
  from this script's own directory) / `GLADEX_QUEUE_SOURCE_TIMEOUT` (default
  `20`) — the queue-source check only. Same shape as the two above: a suite that
  does not test this row leaves it on its default, and because the child answers
  `cannot verify` (exit 3, `warning`) for any sandbox repo without a `### Queue`
  section, stubbing is not required for hermeticity — no suite in `tests/`
  quotes a `### Queue` section, so every existing suite gets a `warning` row and
  an unchanged exit code.
- `GLADEX_SOURCE_SYNC_BIN` (default: the sibling `source-sync-check`, resolved
  from this script's own directory) / `GLADEX_SOURCE_SYNC_TIMEOUT` (default
  `60`) — the promote-gates row's **second pair** only (`[0.4.46]`): the row
  runs `source-sync-check --env dev --format json` itself and compares that
  reading with the promote gate's `dev-sync` verdict. A suite that does not
  test this row never reaches the run (its promote stub returns no report), but
  the promote-gates suite points the first at a scenario stub that logs its
  argv/env — the hook is also what keeps the hermetic run off the live dev
  tree.
- `GLADEX_GIT_BIN` (default: `git` on PATH — **promote's own hook name**, so
  both sides of the comparison name the same child) / `GLADEX_SHIP_TIMEOUT`
  (default `30` — promote's own budget for this very command) — the
  promote-gates row's **third pair** only (`[0.4.49]`): the row runs
  `git -C <repo> status --porcelain --ignored -- <PROMOTED_TREES>` itself and
  compares that reading with the `ship-tree` gate. Read-only, bounded, and
  only run when a report was obtained. The promote-gates suite points the
  first at a wrapper that logs its argv and `exec`s real git (the claim states
  must come from real `git status` semantics), then at failure stubs.

**Go-tests verdict — taken from the EXIT CODE, never from grepping stdout
(rewritten 2026-09-24, CHANGELOG `[0.4.38]`)**: the check used to be
`... | tail -1 || echo "FAIL"` + `grep -q "^ok"` over the pattern
`./cmd/gladex/commands/...`, and had three independent ways to be wrong: the
pattern never built `./cmd/gladex` (so a non-compiling test package there
reported `passing`, exit 0); `grep -q` matches if *any* line matches, so the
`|| echo "FAIL"` guard could never fire once an `ok` line existed (a killed run
reported `passing`); and an unrunnable toolchain or an absent `go.mod` reported
`error "failing"` — the machine's problem reported as our broken tests.

| `go test` exit | extra evidence | status | detail |
|---|---|---|---|
| `0` | — | `ok` | `passing` |
| `1` | a package-level `FAIL\t<pkg>` line | `error` | `failing: <pkg>` |
| `124` / `137` | killed by the budget | `error` | `timed out after Ns - no verdict on the tests` |
| `1` | **no** such line | `warning` | `cannot verify: go exited 1 without running a test package` |
| anything else | — | `warning` | `cannot verify: go test exit N` |
> build · mimo-v2.6-flash-free
$ cd /data/repo && awk 'NR>=968 && NR<=1300' tools/REGISTRY.md | grep -n "Tests\|Status history\|test_system_status" | head -30; echo "==="; grep -n "test_system_status_mx_soa" tools/REGISTRY.md | head
===
1332:  case 11 + mutation M3 of `test_system_status_mx_soa.sh`.
1336:- `bash tests/test_system_status_mx_soa.sh` → **31 assertions**, hermetic and <1s

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

[Sat Sep 26 18:44:40 2026] 127.0.0.1:47844 Accepted
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47844 Closing
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47860 Accepted
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47860 Closing
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47862 Accepted
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47862 Closing
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47870 Accepted
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47870 Closing
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47884 Accepted
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47884 Closing
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47894 Accepted
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47894 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43142 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43142 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43158 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43158 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43166 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43166 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43176 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43176 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43184 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43184 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43200 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43200 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43214 Accepted
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43214 Closing
[Sat Sep 26 18:45:01 2026] 127.0.0.1:43220 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43220 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43236 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43236 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43246 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43246 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43254 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43254 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43258 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43258 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43260 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43260 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43274 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43274 Closing
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43288 Accepted
[Sat Sep 26 18:45:02 2026] 127.0.0.1:43288 Closing
[Sat Sep 26 18:50:07 2026] 127.0.0.1:49510 Accepted
[Sat Sep 26 18:50:07 2026] 127.0.0.1:49510 Closing
[Sat Sep 26 18:50:15 2026] 127.0.0.1:45618 Accepted
[Sat Sep 26 18:50:15 2026] 127.0.0.1:45618 Closing
[Sat Sep 26 18:50:28 2026] 127.0.0.1:54774 Accepted
[Sat Sep 26 18:50:28 2026] 127.0.0.1:54774 Closing
[Sat Sep 26 18:50:30 2026] 127.0.0.1:54790 Accepted
[Sat Sep 26 18:50:30 2026] 127.0.0.1:54790 Closing
[Sat Sep 26 18:50:31 2026] 127.0.0.1:54800 Accepted
[Sat Sep 26 18:50:31 2026] 127.0.0.1:54800 Closing
[Sat Sep 26 18:50:32 2026] 127.0.0.1:54814 Accepted
[Sat Sep 26 18:50:32 2026] 127.0.0.1:54814 Closing
[Sat Sep 26 18:51:15 2026] 127.0.0.1:42630 Accepted
[Sat Sep 26 18:51:15 2026] 127.0.0.1:42630 Closing
[Sat Sep 26 18:51:18 2026] 127.0.0.1:42632 Accepted
[Sat Sep 26 18:51:18 2026] 127.0.0.1:42632 Closing
[Sat Sep 26 18:51:33 2026] 127.0.0.1:50690 Accepted