Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 569 files, 20 MB |
| Latest run log | run-20260926-180537-167.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
| File | Size | Modified (UTC) |
|---|---|---|
| run-20260926-180537-167.log | 184 KB | 2026-09-26 16:51:27 |
| run-20260926-170523-166.log | 164 KB | 2026-09-26 15:55:37 |
| run-20260926-162230-165.log | 178 KB | 2026-09-26 14:55:23 |
| run-20260926-154050-164.log | 198 KB | 2026-09-26 14:12:30 |
| run-20260926-153049-163.log | 153 B | 2026-09-26 13:30:50 |
| run-20260926-152049-162.log | 153 B | 2026-09-26 13:20:49 |
| run-20260926-151048-161.log | 153 B | 2026-09-26 13:10:49 |
| run-20260926-150047-160.log | 153 B | 2026-09-26 13:00:48 |
| run-20260926-145046-159.log | 153 B | 2026-09-26 12:50:47 |
| run-20260926-144046-158.log | 153 B | 2026-09-26 12:40:46 |
| run-20260926-143045-157.log | 153 B | 2026-09-26 12:30:46 |
| run-20260926-142044-156.log | 153 B | 2026-09-26 12:20:45 |
| run-20260926-141044-155.log | 153 B | 2026-09-26 12:10:44 |
| run-20260926-140043-154.log | 153 B | 2026-09-26 12:00:44 |
| run-20260926-135042-153.log | 190 B | 2026-09-26 11:50:43 |
| run-20260926-134042-152.log | 153 B | 2026-09-26 11:40:42 |
| run-20260926-133041-151.log | 153 B | 2026-09-26 11:30:42 |
| run-20260926-132040-150.log | 190 B | 2026-09-26 11:20:41 |
| run-20260926-131039-149.log | 153 B | 2026-09-26 11:10:40 |
| run-20260926-130039-148.log | 153 B | 2026-09-26 11:00:39 |
| run-20260926-125038-147.log | 190 B | 2026-09-26 10:50:39 |
| run-20260926-124037-146.log | 153 B | 2026-09-26 10:40:38 |
| run-20260926-123037-145.log | 153 B | 2026-09-26 10:30:37 |
| run-20260926-122036-144.log | 190 B | 2026-09-26 10:20:37 |
| run-20260926-121035-143.log | 190 B | 2026-09-26 10:10:36 |
| run-20260926-120035-142.log | 153 B | 2026-09-26 10:00:35 |
| run-20260926-115034-141.log | 153 B | 2026-09-26 09:50:34 |
| run-20260926-114033-140.log | 153 B | 2026-09-26 09:40:34 |
| run-20260926-113032-139.log | 153 B | 2026-09-26 09:30:33 |
| run-20260926-112032-138.log | 153 B | 2026-09-26 09:20:32 |
| run-20260926-111031-137.log | 153 B | 2026-09-26 09:10:32 |
| run-20260926-110026-136.log | 153 B | 2026-09-26 09:00:31 |
| run-20260926-105025-135.log | 153 B | 2026-09-26 08:50:26 |
| run-20260926-104024-134.log | 190 B | 2026-09-26 08:40:25 |
| run-20260926-103023-133.log | 153 B | 2026-09-26 08:30:24 |
| run-20260926-102023-132.log | 153 B | 2026-09-26 08:20:23 |
| run-20260926-101022-131.log | 190 B | 2026-09-26 08:10:23 |
| run-20260926-100021-130.log | 153 B | 2026-09-26 08:00:22 |
| run-20260926-095021-129.log | 153 B | 2026-09-26 07:50:21 |
| run-20260926-090029-128.log | 230 KB | 2026-09-26 07:40:21 |
| run-20260926-081623-127.log | 209 KB | 2026-09-26 06:50:29 |
| run-20260926-073109-126.log | 146 KB | 2026-09-26 06:06:23 |
| run-20260926-061035-125.log | 341 KB | 2026-09-26 05:21:09 |
| run-20260926-052113-124.log | 352 KB | 2026-09-26 04:00:35 |
| run-20260926-043030-123.log | 311 KB | 2026-09-26 03:11:13 |
| run-20260926-032802-122.log | 338 KB | 2026-09-26 02:20:30 |
| run-20260926-024118-121.log | 334 KB | 2026-09-26 01:18:02 |
| run-20260926-020038-120.log | 273 KB | 2026-09-26 00:31:18 |
| run-20260926-015037-119.log | 153 B | 2026-09-25 23:50:38 |
| run-20260926-014036-118.log | 153 B | 2026-09-25 23:40:37 |
Tail — run-20260926-180537-167.log (last 200 lines)
FAIL - drift detail names the resolver (got 'got '10 gladex.de.'@1.1.1.1 '10 mail.gladex.de.'@8.8.8.8, expected 10 gladex.de.')
FAIL - detail distinguishes answered-none from unreachable (got 'got ';; communications error to 1.1.1.1#53: timed out ;; communications error to 1.1.1.1#53: timed out ;; communications error to 1.1.1.1#53: timed out ;; no servers could be reached'@1.1.1.1 'none'@8.8.8.8, expected 10 gladex.de.')
FAIL - one reachable SOA witness should warn (got 'error' — pre-fix this was 'error')
FAIL - SOA detail names the reading (got 'insane serial 'error'@1.1.1.1 '2026092602'@8.8.8.8 (want numeric >0)')
FAIL - single-resolver SOA should exit 0 (got 1)
FAIL - mirror SOA should warn
FAIL - placeholder warning still fires (got 'insane serial 'error'@1.1.1.1 '2026092602'@8.8.8.8 (want numeric >0)')
FAIL - detail carries SOA UNVERIFIED (got 'insane serial 'error'@1.1.1.1 'error'@8.8.8.8 (want numeric >0)')
FAIL - an unreachable resolver is not an insane serial
FAIL - detail says the other resolver could not be asked (got 'insane serial 'error'@1.1.1.1 'soon'@8.8.8.8 (want numeric >0)')
FAIL - detail is UNVERIFIED (got 'no SOA answer on 1.1.1.1/8.8.8.8')
FAIL - dns_query filters ';;' before paste (got: )
FAIL - check_mx no longer reads dig directly (the classifier is the only reader)
FAIL - check_soa_serial no longer reads dig directly
FAIL - check_mx asks 1.1.1.1 through dns_query
FAIL - check_mx asks 8.8.8.8 through dns_query
FAIL - check_soa_serial asks 1.1.1.1 through dns_query
FAIL - check_soa_serial asks 8.8.8.8 through dns_query
FAIL - the SOA single-witness branch is in the source
FAIL - the MX both-unreachable branch is in the source
FAIL - --help has an mx/soa verdicts section
FAIL - --help states the MX UNVERIFIED verdict
FAIL - M2 mutation setup: could not plant the single-witness mutant
FAIL - M3 mutation setup: could not plant the SOA-witness mutant
=== Results: 38 passed, 35 failed ===
suite rc=0
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && grep -n "^## system-status" -A 80 tools/REGISTRY.md | head -110
968:## system-status
969-
970-**Purpose**: Quick overview of the entire Gladex system health — services, endpoints, DNS (incl. MX/SOA), messages, git, tests.
971-
972-**Location**: `/data/repo/tools/system-status` (also via the `/data/tools` symlink —
973-`SCRIPT_DIR` resolves the script path with `realpath`, so through the link the repo
974-root is `/data/repo`, not the link's parent)
975-
976-**Usage**:
977-```
978-system-status [--format human|json] [--help]
979-```
980-
981-**Options**:
982-- `--format human|json` - Output format (default: human)
983-- `--help` - Show help message
984-
985-**Exit codes**:
986-- `0` - All systems healthy
987-- `1` - One or more systems unhealthy
988-- `2` - Invalid arguments
989-
990-**Checks performed (35 total)**:
991-- systemd services: investor-app-dev, investor-app-prod, git-daemon, apache2, certbot.timer, watchdog
992-- HTTP endpoints: /healthz (dev+prod), /download/gladex, /api/health, /api/uptime, /api/endpoints, /api/changelog, /api/stats
993-- DNS records: gladex.de, dev/git/info/stats/team/log subdomains → 77.90.15.49 —
994- **asked on BOTH 1.1.1.1 and 8.8.8.8** through `dns_query`, which classifies the
995- reading before anything reads it: `answer` (a usable line — including the empty
996- one, since "no such record" is an answer: empty value, exit 0) vs `transport`
997- (dig's `;;` diagnostics and/or a failed dig: no reading at all). dig prints
998- those diagnostics to **stdout** and exits 9, so a raw `$(dig ... 2>/dev/null)`
999- reads them as the record's value — measured 2026-09-26, one blip in eight runs
1000- published `got ;; communications error to 1.1.1.1#53: timed out, expected
1001- 77.90.15.49` and took the tool to exit 1. Verdicts: `ok` `<ip> on
1002- 1.1.1.1+8.8.8.8` (every answering resolver agreed) · **`warning`** `<ip> on
1003- 1.1.1.1 (8.8.8.8 unreachable - single-resolver reading)` — one witness, loud in
1004- the detail, **exit 0**, never silently green · **`error`** `no answer from
1005- 1.1.1.1/8.8.8.8 (both resolvers unreachable) … DNS UNVERIFIED` — nothing
1006- measured, nothing claimed, nothing passes · **`error`** `<resolver> answered
1007- '<ip>', expected <ip>` — drift is still drift, and the detail now says which
1008- resolver said it. `dig`'s exit code never leaks past the tool (documented:
1009- 0/1/2).
1010-- MX record: `MX:gladex.de` must be `10 gladex.de.` on **BOTH** 1.1.1.1 and 8.8.8.8
1011- (propagation check; standing investor rule 2026-09-23 — adopted MX must stay
1012- verified). Any other/missing answer on either resolver = **error, exit 1** —
1013- a wrong MX on a public resolver is a mail outage, not a warning.
1014-- SOA serial: `SOA:gladex.de` — serial must be numeric, >0 and **identical** on
1015- 1.1.1.1/8.8.8.8 (split = **warning** "propagation lag", transient by definition;
1016- garbage/missing serial = **error**). The known `MNAME` placeholder
1017- (`a.misconfigured.dns.server.invalid.`, provider-panel-only fix, open
1018- NEEDS-INVESTOR) is surfaced as a **warning** in the detail and clears itself
1019- once the panel value changes.
1020-- Cloud stacks: `cloud` — Nextcloud `status.php` + Immich `/api/server/ping`
1021- (HTTP) + `docker ps --all` state/`RestartCount` (the two newest production
1022- surfaces could otherwise be dead under an ALL-SYSTEMS-HEALTHY dashboard)
1023-- TLS cert expiry: **ONE ROW PER LIVE LET'S ENCRYPT LINEAGE** —
1024- `tls-cert-expiry` (SNI `gladex.de`, the 7-SAN lineage) and
1025- `tls-cert-expiry-photos` (SNI `photos.gladex.de`, the `cloud.`+`photos.`
1026- lineage issued 2026-09-26). Date rules unchanged: warn <30d, error <7d, `no
1027- cert` error when the handshake offers nothing. Each row additionally proves
1028- the certificate it read **covers its own SNI**: Apache answers with the
1029- default vhost's cert when nothing matches a name (the `*.gladex.de` alias
1030- swallowing `photos.` until 2026-09-26), so a date off the wrong cert is a
1031- date about the wrong lineage. SAN list readable and covering → the date
1032- decides; readable and NOT covering → **error** naming the served SANs;
1033- unreadable → the date decides with `identity not checked` appended (never a
1034- silent pass, never a red on a host we could not ask). Wildcards cover exactly
1035- one label above their domain.
1036-- Investor messages: unread `investor_to_agent` rows counted across **BOTH** DBs — detail `N unread dev=X prod=Y`
1037-- Git tree: clean/dirty
1038-- Next-candidate queue: `queue-source` carries `queue-source-check --format json
1039- --repo`'s verdict — CHANGELOG's newest `### Queue` section must be a POINTER at
1040- `agent-logs/PROGRESS.md` (the one authoritative list) and the 111 historical
1041- item lines must be unchanged. child `0` → **ok**, `1` → **error + `ERRORS++`**
1042- (a second list, a dropped pointer or an empty authoritative list turns the
1043- dashboard red), `3` → **warning `cannot verify`** (no `CHANGELOG.md`, no
1044- `### Queue` section, no `agent-logs/PROGRESS.md`) — never ok, never an error on
1045- a tree that was not checked. `detail` is the child's own, already sanitised of
1046- `"` and `\` so the hand-built row stays valid JSON.
1047-- Go tests: verdict from the **exit code** of `go test ./...` over the whole
1048- module (worktree) — see the verdict table below
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && sed -n '1048,1130p' tools/REGISTRY.md
module (worktree) — see the verdict table below
- Go compile: `repo-lint --format json --sha HEAD`'s `go_compile` verdict
(the COMMIT) — see the verdict table below
- Promote gates: `promote-dev-to-prod --dry-run --force --format json` +
`GLADEX_GATE_PROBE=1` — "can we ship right now?" in one dashboard read —
see the verdict table below
**Unread-count contract (both DBs, since 2026-09-24)**: the investor primarily reads
the **prod** thread (public `https://gladex.de` → Apache `:443` → prod `:8001`), while
dev `:8000` carries a separate thread. The old read touched only
`/opt/startup/dev/data/messages.db`, so an unread row on **prod** still reported
`0 unread` / `ok` — a blind guard on the very check that protects STEP 0 (an
unanswered investor = a failed run). Counts are summed; an unreadable/missing DB
reports `?` and forces `warning`, so it can **never masquerade as `0`**.
`agent_to_investor` rows are never counted — those are our own outgoing messages.
**Test hooks (env)**:
- `GLADEX_DEV_DB` / `GLADEX_PROD_DB` — point the unread check at fixture DBs.
Defaults are the live `/opt/startup/{dev,prod}/data/messages.db`. Same pattern as
`IDENTITY_REPO`/`IDENTITY_LOG` on `identity-run.sh`.
- `GLADEX_REPO_DIR` — repo the git/Go checks report on (default: parent dir of the
script). Lets hermetic suites run **mutated copies** from a sandbox path; without
it a copy under `/tmp` computes a `REPO_DIR` with no `.git` and dies at the git
check under `set -e` (silent empty output → vacuous mutation checks).
- `GLADEX_GO_BIN` (default `go`) / `GLADEX_GO_TIMEOUT` (default `120`) /
`GLADEX_GO_GOPATH` (default `/tmp/gopath`) / `GLADEX_GO_GOCACHE` (default
`/tmp/gocache`) — the go-tests check only. `GOMODCACHE` is
`$GLADEX_GO_GOPATH/pkg/mod`. All four are documented in `--help`.
- `GLADEX_REPO_LINT_BIN` (default: the sibling `repo-lint`, resolved from this
script's own directory) / `GLADEX_GO_LINT_TIMEOUT` (default `120`) — the
go-compile check only. The first is what keeps a hermetic suite off the real
tree: every `system-status` run lints a real commit, so a suite that does not
test this row stubs the child (all three older suites do).
- `GLADEX_PROMOTE_BIN` (default: `<reported repo>/tools/promote-dev-to-prod`,
i.e. resolved through `GLADEX_REPO_DIR`, not through the script's own dir) /
`GLADEX_PROMOTE_TIMEOUT` (default `120`) — the promote-gates check only. The
default is what keeps a hermetic suite off the real tool: a suite that does
not test this row points it at a non-existent path and gets
`cannot verify` in well under a second (all four older suites do).
- `GLADEX_QUEUE_SOURCE_BIN` (default: the sibling `queue-source-check`, resolved
from this script's own directory) / `GLADEX_QUEUE_SOURCE_TIMEOUT` (default
`20`) — the queue-source check only. Same shape as the two above: a suite that
does not test this row leaves it on its default, and because the child answers
`cannot verify` (exit 3, `warning`) for any sandbox repo without a `### Queue`
section, stubbing is not required for hermeticity — no suite in `tests/`
quotes a `### Queue` section, so every existing suite gets a `warning` row and
an unchanged exit code.
- `GLADEX_SOURCE_SYNC_BIN` (default: the sibling `source-sync-check`, resolved
from this script's own directory) / `GLADEX_SOURCE_SYNC_TIMEOUT` (default
`60`) — the promote-gates row's **second pair** only (`[0.4.46]`): the row
runs `source-sync-check --env dev --format json` itself and compares that
reading with the promote gate's `dev-sync` verdict. A suite that does not
test this row never reaches the run (its promote stub returns no report), but
the promote-gates suite points the first at a scenario stub that logs its
argv/env — the hook is also what keeps the hermetic run off the live dev
tree.
- `GLADEX_GIT_BIN` (default: `git` on PATH — **promote's own hook name**, so
both sides of the comparison name the same child) / `GLADEX_SHIP_TIMEOUT`
(default `30` — promote's own budget for this very command) — the
promote-gates row's **third pair** only (`[0.4.49]`): the row runs
`git -C <repo> status --porcelain --ignored -- <PROMOTED_TREES>` itself and
compares that reading with the `ship-tree` gate. Read-only, bounded, and
only run when a report was obtained. The promote-gates suite points the
first at a wrapper that logs its argv and `exec`s real git (the claim states
must come from real `git status` semantics), then at failure stubs.
**Go-tests verdict — taken from the EXIT CODE, never from grepping stdout
(rewritten 2026-09-24, CHANGELOG `[0.4.38]`)**: the check used to be
`... | tail -1 || echo "FAIL"` + `grep -q "^ok"` over the pattern
`./cmd/gladex/commands/...`, and had three independent ways to be wrong: the
pattern never built `./cmd/gladex` (so a non-compiling test package there
reported `passing`, exit 0); `grep -q` matches if *any* line matches, so the
`|| echo "FAIL"` guard could never fire once an `ok` line existed (a killed run
reported `passing`); and an unrunnable toolchain or an absent `go.mod` reported
`error "failing"` — the machine's problem reported as our broken tests.
| `go test` exit | extra evidence | status | detail |
|---|---|---|---|
| `0` | — | `ok` | `passing` |
| `1` | a package-level `FAIL\t<pkg>` line | `error` | `failing: <pkg>` |
| `124` / `137` | killed by the budget | `error` | `timed out after Ns - no verdict on the tests` |
| `1` | **no** such line | `warning` | `cannot verify: go exited 1 without running a test package` |
| anything else | — | `warning` | `cannot verify: go test exit N` |
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && awk 'NR>=968 && NR<=1300' tools/REGISTRY.md | grep -n "Tests\|Status history\|test_system_status" | head -30; echo "==="; grep -n "test_system_status_mx_soa" tools/REGISTRY.md | head
===
1332: case 11 + mutation M3 of `test_system_status_mx_soa.sh`.
1336:- `bash tests/test_system_status_mx_soa.sh` → **31 assertions**, hermetic and <1s
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Sat Sep 26 18:44:40 2026] 127.0.0.1:47844 Accepted [Sat Sep 26 18:44:40 2026] 127.0.0.1:47844 Closing [Sat Sep 26 18:44:40 2026] 127.0.0.1:47860 Accepted [Sat Sep 26 18:44:40 2026] 127.0.0.1:47860 Closing [Sat Sep 26 18:44:40 2026] 127.0.0.1:47862 Accepted [Sat Sep 26 18:44:40 2026] 127.0.0.1:47862 Closing [Sat Sep 26 18:44:40 2026] 127.0.0.1:47870 Accepted [Sat Sep 26 18:44:40 2026] 127.0.0.1:47870 Closing [Sat Sep 26 18:44:40 2026] 127.0.0.1:47884 Accepted [Sat Sep 26 18:44:40 2026] 127.0.0.1:47884 Closing [Sat Sep 26 18:44:40 2026] 127.0.0.1:47894 Accepted [Sat Sep 26 18:44:40 2026] 127.0.0.1:47894 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43142 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43142 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43158 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43158 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43166 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43166 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43176 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43176 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43184 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43184 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43200 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43200 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43214 Accepted [Sat Sep 26 18:45:01 2026] 127.0.0.1:43214 Closing [Sat Sep 26 18:45:01 2026] 127.0.0.1:43220 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43220 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43236 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43236 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43246 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43246 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43254 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43254 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43258 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43258 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43260 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43260 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43274 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43274 Closing [Sat Sep 26 18:45:02 2026] 127.0.0.1:43288 Accepted [Sat Sep 26 18:45:02 2026] 127.0.0.1:43288 Closing [Sat Sep 26 18:50:07 2026] 127.0.0.1:49510 Accepted [Sat Sep 26 18:50:07 2026] 127.0.0.1:49510 Closing [Sat Sep 26 18:50:15 2026] 127.0.0.1:45618 Accepted [Sat Sep 26 18:50:15 2026] 127.0.0.1:45618 Closing [Sat Sep 26 18:50:28 2026] 127.0.0.1:54774 Accepted [Sat Sep 26 18:50:28 2026] 127.0.0.1:54774 Closing [Sat Sep 26 18:50:30 2026] 127.0.0.1:54790 Accepted [Sat Sep 26 18:50:30 2026] 127.0.0.1:54790 Closing [Sat Sep 26 18:50:31 2026] 127.0.0.1:54800 Accepted [Sat Sep 26 18:50:31 2026] 127.0.0.1:54800 Closing [Sat Sep 26 18:50:32 2026] 127.0.0.1:54814 Accepted [Sat Sep 26 18:50:32 2026] 127.0.0.1:54814 Closing [Sat Sep 26 18:51:15 2026] 127.0.0.1:42630 Accepted [Sat Sep 26 18:51:15 2026] 127.0.0.1:42630 Closing [Sat Sep 26 18:51:18 2026] 127.0.0.1:42632 Accepted [Sat Sep 26 18:51:18 2026] 127.0.0.1:42632 Closing [Sat Sep 26 18:51:33 2026] 127.0.0.1:50690 Accepted
Generated 2026-09-26 16:51:33 UTC · Gladex.de