Gladex Agent Logs

$ cd /data/repo && grep -rn "33 checks\|32 checks\|34 checks\|checks, errors" tools/ tests/ *.md 2>/dev/null | head -20; echo "=== suite list pins ==="; grep -rn "test_system_status" tools/REGISTRY.md CHANGELOG.md agent-logs/PROGRESS.md | head -10
=== suite list pins ===
tools/REGISTRY.md:1280:- `bash tests/test_system_status_unread.sh` → **24 assertions**, hermetic and <1s
tools/REGISTRY.md:1301:  case 11 + mutation M3 of `test_system_status_mx_soa.sh`.
tools/REGISTRY.md:1305:- `bash tests/test_system_status_mx_soa.sh` → **31 assertions**, hermetic and <1s
tools/REGISTRY.md:1317:- `bash tests/test_system_status_go_tests.sh` → **66 assertions, 6 mutations**,
tools/REGISTRY.md:1335:  tests/test_system_status_go_tests.sh` → **15 passed / 39 failed / 1 skipped**.
tools/REGISTRY.md:1339:- `bash tests/test_system_status_go_compile.sh` → **82 assertions, 6 mutations**,
tools/REGISTRY.md:1357:  8894894f178db8ed97691f6b132855bd> bash tests/test_system_status_go_compile.sh`
tools/REGISTRY.md:1370:- `bash tests/test_system_status_promote_gates.sh` → **295 assertions, 39 mutations**,
tools/REGISTRY.md:1561:  — `bash tests/test_system_status_promote_gates.sh`, logged at
tools/REGISTRY.md:1575:  tests/test_system_status_promote_gates.sh`) → **234 passed / 9 failed / 0 skipped**:
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "^## " tools/REGISTRY.md | sed -n '1,80p'
3:## domain-availability-check
60:## budget-show
110:## dns-verify
171:## healthcheck
217:## promote-dev-to-prod
693:## ip-drift-check
758:## pdns-api.py
817:## ip-drift-cron
926:## verify-landing
968:## system-status
1704:## version-check
1742:## tls-check
1780:## smtp-relay-probe
1820:## source-sync-check
1868:## queue-source-check
1963:## test_identity_wrapper.sh
1990:## test_app_contrast_meta.php
2015:## repo-lint
2071:## regression-run
2129:## test_regression_run.sh
2158:## test_queue_source.sh
> build · mimo-v2.6-flash-free
$ cd /data/repo && sed -n '968,1060p' tools/REGISTRY.md
## system-status

**Purpose**: Quick overview of the entire Gladex system health — services, endpoints, DNS (incl. MX/SOA), messages, git, tests.

**Location**: `/data/repo/tools/system-status` (also via the `/data/tools` symlink —
`SCRIPT_DIR` resolves the script path with `realpath`, so through the link the repo
root is `/data/repo`, not the link's parent)

**Usage**:
```
system-status [--format human|json] [--help]
```

**Options**:
- `--format human|json` - Output format (default: human)
- `--help` - Show help message

**Exit codes**:
- `0` - All systems healthy
- `1` - One or more systems unhealthy
- `2` - Invalid arguments

**Checks performed (30 total)**:
- systemd services: investor-app-dev, investor-app-prod, git-daemon, apache2, certbot.timer, watchdog
- HTTP endpoints: /healthz (dev+prod), /download/gladex, /api/health, /api/uptime, /api/endpoints, /api/changelog, /api/stats
- DNS records: gladex.de, dev/git/info/stats/team/log subdomains → 77.90.15.49
- MX record: `MX:gladex.de` must be `10 gladex.de.` on **BOTH** 1.1.1.1 and 8.8.8.8
  (propagation check; standing investor rule 2026-09-23 — adopted MX must stay
  verified). Any other/missing answer on either resolver = **error, exit 1** —
  a wrong MX on a public resolver is a mail outage, not a warning.
- SOA serial: `SOA:gladex.de` — serial must be numeric, >0 and **identical** on
  1.1.1.1/8.8.8.8 (split = **warning** "propagation lag", transient by definition;
  garbage/missing serial = **error**). The known `MNAME` placeholder
  (`a.misconfigured.dns.server.invalid.`, provider-panel-only fix, open
  NEEDS-INVESTOR) is surfaced as a **warning** in the detail and clears itself
  once the panel value changes.
- TLS cert expiry: openssl check, warn <30d, error <7d
- Investor messages: unread `investor_to_agent` rows counted across **BOTH** DBs — detail `N unread dev=X prod=Y`
- Git tree: clean/dirty
- Next-candidate queue: `queue-source` carries `queue-source-check --format json
  --repo`'s verdict — CHANGELOG's newest `### Queue` section must be a POINTER at
  `agent-logs/PROGRESS.md` (the one authoritative list) and the 111 historical
  item lines must be unchanged. child `0` → **ok**, `1` → **error + `ERRORS++`**
  (a second list, a dropped pointer or an empty authoritative list turns the
  dashboard red), `3` → **warning `cannot verify`** (no `CHANGELOG.md`, no
  `### Queue` section, no `agent-logs/PROGRESS.md`) — never ok, never an error on
  a tree that was not checked. `detail` is the child's own, already sanitised of
  `"` and `\` so the hand-built row stays valid JSON.
- Go tests: verdict from the **exit code** of `go test ./...` over the whole
  module (worktree) — see the verdict table below
- Go compile: `repo-lint --format json --sha HEAD`'s `go_compile` verdict
  (the COMMIT) — see the verdict table below
- Promote gates: `promote-dev-to-prod --dry-run --force --format json` +
  `GLADEX_GATE_PROBE=1` — "can we ship right now?" in one dashboard read —
  see the verdict table below

**Unread-count contract (both DBs, since 2026-09-24)**: the investor primarily reads
the **prod** thread (public `https://gladex.de` → Apache `:443` → prod `:8001`), while
dev `:8000` carries a separate thread. The old read touched only
`/opt/startup/dev/data/messages.db`, so an unread row on **prod** still reported
`0 unread` / `ok` — a blind guard on the very check that protects STEP 0 (an
unanswered investor = a failed run). Counts are summed; an unreadable/missing DB
reports `?` and forces `warning`, so it can **never masquerade as `0`**.
`agent_to_investor` rows are never counted — those are our own outgoing messages.

**Test hooks (env)**:
- `GLADEX_DEV_DB` / `GLADEX_PROD_DB` — point the unread check at fixture DBs.
  Defaults are the live `/opt/startup/{dev,prod}/data/messages.db`. Same pattern as
  `IDENTITY_REPO`/`IDENTITY_LOG` on `identity-run.sh`.
- `GLADEX_REPO_DIR` — repo the git/Go checks report on (default: parent dir of the
  script). Lets hermetic suites run **mutated copies** from a sandbox path; without
  it a copy under `/tmp` computes a `REPO_DIR` with no `.git` and dies at the git
  check under `set -e` (silent empty output → vacuous mutation checks).
- `GLADEX_GO_BIN` (default `go`) / `GLADEX_GO_TIMEOUT` (default `120`) /
  `GLADEX_GO_GOPATH` (default `/tmp/gopath`) / `GLADEX_GO_GOCACHE` (default
  `/tmp/gocache`) — the go-tests check only. `GOMODCACHE` is
  `$GLADEX_GO_GOPATH/pkg/mod`. All four are documented in `--help`.
- `GLADEX_REPO_LINT_BIN` (default: the sibling `repo-lint`, resolved from this
  script's own directory) / `GLADEX_GO_LINT_TIMEOUT` (default `120`) — the
  go-compile check only. The first is what keeps a hermetic suite off the real
  tree: every `system-status` run lints a real commit, so a suite that does not
  test this row stubs the child (all three older suites do).
- `GLADEX_PROMOTE_BIN` (default: `<reported repo>/tools/promote-dev-to-prod`,
  i.e. resolved through `GLADEX_REPO_DIR`, not through the script's own dir) /
  `GLADEX_PROMOTE_TIMEOUT` (default `120`) — the promote-gates check only. The
  default is what keeps a hermetic suite off the real tool: a suite that does
  not test this row points it at a non-existent path and gets
  `cannot verify` in well under a second (all four older suites do).
- `GLADEX_QUEUE_SOURCE_BIN` (default: the sibling `queue-source-check`, resolved
  from this script's own directory) / `GLADEX_QUEUE_SOURCE_TIMEOUT` (default
  `20`) — the queue-source check only. Same shape as the two above: a suite that
  does not test this row leaves it on its default, and because the child answers
  `cannot verify` (exit 3, `warning`) for any sandbox repo without a `### Queue`
$ cd /data/repo && grep -n -i "tls\|checks:\|33 \|32 checks" tools/REGISTRY.md | awk -F: '$1>968 && $1<1710' | head -40
1004:- TLS cert expiry: openssl check, warn <30d, error <7d
1548:  M33 the hand-off dropped (`SS_SHIP_OUT=""` → our half of the comparison is blind)
1653:28 checks: the COMMIT is now a row too — `go-compile` carries `repo-lint`'s
1656:30 checks: the next-candidate queue is a row too — `queue-source` carries
1661:29 checks: the SHIP decision is a row too — `promote-gates` carries
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -rn "tls" tests/test_system_status_cloud.sh | head; echo "--- how cloud suite is structured (head) ---"; sed -n '1,60p' tests/test_system_status_cloud.sh
--- how cloud suite is structured (head) ---
#!/usr/bin/env bash
# Regression test for the `cloud` row in tools/system-status (Nextcloud + Immich).
#
# Motivated by queue item (10): system-status had 33 rows and not one of them knew
# either cloud stack existed, so Nextcloud (:8080) and Immich (:2283) — the two
# newest production surfaces — could BOTH be dead while the dashboard still printed
# ALL SYSTEMS HEALTHY. The row closes that with three independent witnesses (HTTP
# status.php, HTTP /api/server/ping, docker container state + RestartCount) and it
# is worth pinning every branch, because two of them are judgements rather than
# measurements:
#   * a 200 whose BODY cannot be read must be `warning`, never `ok` (an unreadable
#     body is not a healthy service) and never `error` (it is not a measured
#     failure either) — the row must be willing to say it did not measure;
#   * a docker binary that will not answer is `cannot verify`, not "the stacks are
#     down" — a verdict about a host we could not ask is not a verdict.
#
# Hermetic by construction:
#   * systemctl / curl / dig / openssl / go / repo-lint are stubbed on PATH — the
#     curl stub is scenario-driven (it answers the cloud probes with a body AND a
#     code, and the plain endpoint probes with a bare code), so no network is
#     touched and the ~20s `go test` is skipped;
#   * GLADEX_CLOUD_DOCKER_BIN points the container witness at a scenario stub, so
#     this suite never reads (and can never be wrong about) the host's live
#     containers;
#   * GLADEX_DEV_DB / GLADEX_PROD_DB point at absent fixture DBs, so the LIVE
#     message DBs (which guard STEP 0) are never read by this suite;
#   * SYSTEM_STATUS_BIN points the mutation checks at a mutated copy, so the real
#     tool is never edited by this test.
#
# Run: bash tests/test_system_status_cloud.sh
# Exit codes: 0 = all pass, 1 = at least one failure.
set -u

REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
REAL_TOOL="${SYSTEM_STATUS_BIN:-$REPO/tools/system-status}"
TOOL="$REAL_TOOL"

PASS=0
FAIL=0
ok()  { PASS=$((PASS+1)); echo "  ok   - $1"; }
bad() { FAIL=$((FAIL+1)); echo "  FAIL - $1"; }

[ -f "$TOOL" ] || { echo "SKIP: tool not found: $TOOL"; exit 0; }

SANDBOX="$(mktemp -d /tmp/scd-test.XXXXXX)" || exit 1
trap 'rm -rf "$SANDBOX"' EXIT INT TERM

# ---------------------------------------------------------------- stubs -----
STUB="$SANDBOX/bin"
mkdir -p "$STUB"
printf '#!/bin/sh\necho active\n'  > "$STUB/systemctl"
printf '#!/bin/sh\necho "notAfter=Dec 22 12:00:00 2026 GMT"\n' > "$STUB/openssl"
printf '#!/bin/sh\necho "ok  github.com/gladex/gladex/cmd/gladex/commands 0.1s"\n' > "$STUB/go"

# Scenario curl: the tool asks the cloud probes for a BODY plus a code
# (`-w $'\n%{http_code}'`, no -o) and the plain endpoint probes for a bare code
# (`-o /dev/null -w %{http_code}`). Answering both from one stub is what lets a
# single PATH entry drive every HTTP witness in the tool.
cat > "$STUB/curl" <<'STUBCURL'
#!/bin/sh

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

[Sat Sep 26 15:08:45 2026] 127.0.0.1:47802 Accepted
[Sat Sep 26 15:08:45 2026] 127.0.0.1:47802 Closing
[Sat Sep 26 15:08:45 2026] 127.0.0.1:47812 Accepted
[Sat Sep 26 15:08:45 2026] 127.0.0.1:47812 Closing
[Sat Sep 26 15:10:54 2026] 127.0.0.1:34982 Accepted
[Sat Sep 26 15:10:54 2026] 127.0.0.1:34982 Closing
[Sat Sep 26 15:11:16 2026] 127.0.0.1:56780 Accepted
[Sat Sep 26 15:11:16 2026] 127.0.0.1:56780 Closing
[Sat Sep 26 15:25:07 2026] 127.0.0.1:44666 Accepted
[Sat Sep 26 15:25:07 2026] 127.0.0.1:44666 Closing
[Sat Sep 26 15:26:11 2026] 127.0.0.1:32954 Accepted
[Sat Sep 26 15:26:11 2026] 127.0.0.1:32954 Closing
[Sat Sep 26 15:26:11 2026] 127.0.0.1:32970 Accepted
[Sat Sep 26 15:26:11 2026] 127.0.0.1:32970 Closing
[Sat Sep 26 15:26:16 2026] 127.0.0.1:38468 Accepted
[Sat Sep 26 15:26:16 2026] 127.0.0.1:38468 Closing
[Sat Sep 26 15:38:46 2026] 127.0.0.1:39898 Accepted
[Sat Sep 26 15:38:46 2026] 127.0.0.1:39898 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45468 Accepted
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45468 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45478 Accepted
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45478 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45492 Accepted
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45492 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45508 Accepted
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45508 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45524 Accepted
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45524 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45532 Accepted
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45532 Closing
[Sat Sep 26 15:51:12 2026] 127.0.0.1:45548 Accepted
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45548 Closing
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45550 Accepted
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45550 Closing
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45556 Accepted
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45556 Closing
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45568 Accepted
[Sat Sep 26 15:51:13 2026] 127.0.0.1:45568 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48584 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48584 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48594 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48594 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48604 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48604 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48608 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48608 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48612 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48612 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48624 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48624 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48628 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48628 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48640 Accepted
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48640 Closing
[Sat Sep 26 15:51:46 2026] 127.0.0.1:48644 Accepted
[Sat Sep 26 15:51:47 2026] 127.0.0.1:48644 Closing
[Sat Sep 26 15:51:47 2026] 127.0.0.1:48646 Accepted
[Sat Sep 26 15:51:47 2026] 127.0.0.1:48646 Closing
[Sat Sep 26 15:51:53 2026] 127.0.0.1:48660 Accepted