Gladex Agent Logs

Agent run logs & app logs · env: prod · LAN-only investor surface

Overview
Run logs710 files, 28.7 MB
Latest run logrun-20260928-161526-308.log
Log directory/data/agent-logs
App log directory/opt/startup/prod/logs
Run logs (newest first, last 50)
FileSizeModified (UTC)
run-20260928-161526-308.log 115 KB 2026-09-28 14:27:14
run-20260928-160525-307.log 153 B 2026-09-28 14:05:26
run-20260928-155524-306.log 153 B 2026-09-28 13:55:25
run-20260928-154524-305.log 153 B 2026-09-28 13:45:24
run-20260928-153523-304.log 153 B 2026-09-28 13:35:24
run-20260928-152522-303.log 153 B 2026-09-28 13:25:23
run-20260928-151521-302.log 153 B 2026-09-28 13:15:22
run-20260928-150521-301.log 153 B 2026-09-28 13:05:21
run-20260928-145520-300.log 153 B 2026-09-28 12:55:21
run-20260928-144519-299.log 153 B 2026-09-28 12:45:20
run-20260928-143519-298.log 153 B 2026-09-28 12:35:19
run-20260928-142518-297.log 153 B 2026-09-28 12:25:19
run-20260928-141517-296.log 153 B 2026-09-28 12:15:18
run-20260928-140517-295.log 153 B 2026-09-28 12:05:17
run-20260928-135516-294.log 153 B 2026-09-28 11:55:17
run-20260928-134515-293.log 153 B 2026-09-28 11:45:16
run-20260928-133515-292.log 153 B 2026-09-28 11:35:15
run-20260928-132514-291.log 153 B 2026-09-28 11:25:15
run-20260928-131513-290.log 153 B 2026-09-28 11:15:14
run-20260928-130513-289.log 153 B 2026-09-28 11:05:13
run-20260928-125512-288.log 153 B 2026-09-28 10:55:13
run-20260928-124511-287.log 153 B 2026-09-28 10:45:12
run-20260928-123511-286.log 153 B 2026-09-28 10:35:11
run-20260928-122510-285.log 153 B 2026-09-28 10:25:11
run-20260928-121509-284.log 153 B 2026-09-28 10:15:10
run-20260928-120509-283.log 153 B 2026-09-28 10:05:09
run-20260928-115508-282.log 153 B 2026-09-28 09:55:09
run-20260928-114507-281.log 153 B 2026-09-28 09:45:08
run-20260928-113507-280.log 153 B 2026-09-28 09:35:07
run-20260928-112506-279.log 153 B 2026-09-28 09:25:07
run-20260928-111505-278.log 153 B 2026-09-28 09:15:06
run-20260928-110505-277.log 153 B 2026-09-28 09:05:05
run-20260928-101134-276.log 189 KB 2026-09-28 08:55:05
run-20260928-084440-275.log 249 KB 2026-09-28 08:01:34
run-20260928-065526-274.log 230 KB 2026-09-28 06:34:40
run-20260928-051427-273.log 475 KB 2026-09-28 04:45:26
run-20260928-034708-272.log 266 KB 2026-09-28 03:04:27
run-20260928-024242-271.log 330 KB 2026-09-28 01:37:08
run-20260928-020359-270.log 288 KB 2026-09-28 00:32:42
run-20260928-015358-269.log 153 B 2026-09-27 23:53:59
run-20260928-014358-268.log 153 B 2026-09-27 23:43:58
run-20260928-013357-267.log 153 B 2026-09-27 23:33:58
run-20260928-012356-266.log 190 B 2026-09-27 23:23:57
run-20260928-011356-265.log 190 B 2026-09-27 23:13:56
run-20260928-010355-264.log 153 B 2026-09-27 23:03:56
run-20260928-005354-263.log 153 B 2026-09-27 22:53:55
run-20260928-004354-262.log 153 B 2026-09-27 22:43:54
run-20260928-003353-261.log 153 B 2026-09-27 22:33:54
run-20260928-002352-260.log 153 B 2026-09-27 22:23:53
run-20260928-001352-259.log 153 B 2026-09-27 22:13:52
Tail — run-20260928-161526-308.log (last 200 lines)
      pointer at agent-logs/PROGRESS.md, PROGRESS carries the live list, and the 111 historical
      item lines are unchanged
  1 - a violation: a second list exists, the pointer was dropped, the
      authoritative list is empty, one entry owns two `### Queue` sections, or
      a `### Queue` heading was not read as a section (see the violations[]
      array - every cause is reported, not just the first)
  2 - invalid arguments
  3 - cannot verify: CHANGELOG.md, agent-logs/PROGRESS.md or a ### Queue section is absent -
      NOT a pass, never reported as one
$ cd /data/repo && grep -rn "\b57 suite\|57 s\b\|suites discovered\|result: 57" tests/ tools/ app/ 2>/dev/null | head -20; echo "=== monitor suites ==="; grep -n "suites" tests/test_gladex_monitor.sh | head -20
tests/test_regression_run.sh:244:assert_contains "B28 empty dir explains itself" "no test suites discovered" "$OUT"
tools/REGISTRY.md:2641:- `5` - no test suites discovered
tools/regression-run:905:        payload["error"] = f"no test suites discovered under {tests_dir}"
tools/regression-run:911:            print(f"  no test suites discovered under {tests_dir}")
=== monitor suites ===
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "progress_bullets\|PROGRESS carries\|live list\|QUEUE_ITEM\|item_line" tools/queue-source-check | head -30
22:    historical sections are FROZEN - they are history, not a live list, so a
61:What R5 COUNTS (it is reported as `progress_bullets`, and until [0.4.80] that
182:      pointer at {prog}, PROGRESS carries the live list, and the 111 historical
292:            "progress_bullets": None,
384:    report["rules"]["progress_bullets"] = bullets
485:            "progress_bullets": None,
507:            "agent-logs/PROGRESS.md carries the live list and CHANGELOG.md's "
> build · mimo-v2.6-flash-free
$ d=$(mktemp -d /tmp/opencode/covprobe.XXXXXX); for f in /data/repo/tools/*; do [ -f "$f" ] && cp "$f" "$d/"; done; rcsum=0; for f in "$d"/*; do b=$(basename "$f"); [ "$b" = REGISTRY.md ] && continue; out=$(timeout 20 "$f" --help 2>&1); rc=$?; n=$(printf '%s' "$out" | wc -c); if [ "$rc" -ne 0 ] || [ "$n" -eq 0 ]; then echo "PROBLEM $b rc=$rc bytes=$n"; rcsum=1; fi; done; echo "all-ok=$([ $rcsum -eq 0 ] && echo YES || echo NO)"; rm -rf "$d"
all-ok=YES
> build · mimo-v2.6-flash-free
All 20 tools respond from a sandbox copy. Writing the suite:
← Write repo/tests/test_registry_coverage.sh
Created file successfully: repo/tests/test_registry_coverage.sh
> build · mimo-v2.6-flash-free
$ cd /data/repo && sed -n '121,190p' tools/REGISTRY.md; echo "=================== healthcheck"; sed -n '189,240p' tools/REGISTRY.md; echo "=================== tls-check"; sed -n '2142,2182p' tools/REGISTRY.md
## dns-verify

**Purpose**: Verify DNS records via dig for a given domain. Outputs machine-readable JSON Lines.

**Location**: `/data/tools/dns-verify`

**Usage**:
```
dns-verify [--help] [--format jsonl|human] [--server SERVER] [--timeout SECONDS] <domain> [record-type]
```

**Options**:
- `--format jsonl|human` - Output format (default: jsonl)
- `--server SERVER` - DNS server to query (e.g., 8.8.8.8, 1.1.1.1)
- `--timeout SECONDS` - Timeout per query in seconds (default: 10). **Enforced twice**: passed to dig as `+time=N +tries=1` AND as a subprocess deadline (`N+2s`), so a hung dig cannot hang the caller. `< 1` → exit 2.
- `--help` - Show help message

**Exit codes**:
- `0` - Success (records found)
- `1` - General error
- `2` - Invalid arguments / usage error: bad domain, **unknown record type**, bad `--format`, `--timeout < 1`
- `3` - Query **completed** but no records exist for domain/type (a real negative answer)
- `4` - Network/timeout error: the query could **not** be completed (unreachable server, `dig` missing, deadline exceeded). Never folded into `3` — "asked and got nothing" ≠ "never got an answer".

**Record-type validation**: the `[record-type]` argument is checked against a
fixed allowlist (`A AAAA NS MX CNAME TXT SOA PTR SRV CAA DS DNSKEY NSEC NSEC3
NSEC3PARAM RRSIG TLSA SVCB HTTPS SPF DNAME HINFO LOC APL ANY`) plus RFC 3597
`TYPE<n>`; anything else → exit 2 before `dig` is invoked. This matters because
**`dig` does not reject unknown types — it silently reinterprets the token as
the query NAME** (`dig BOGUSTYPE gladex.de` asks for name `BOGUSTYPE`, type A),
which used to make the tool emit a fabricated record
`{"type":"BOGUSTYPE","value":"10.2.3.8"}` and exit 0.

**Output (JSONL)**: One JSON object with structure:
```json
{
  "domain": "example.de",
  "timestamp": "2026-09-20T07:30:00Z",
  "records": [{"type": "NS", "value": "ns1.example.de"}, {"type": "A", "value": "1.2.3.4"}],
  "record_types_queried": ["A", "AAAA", "NS", "MX", "CNAME", "TXT", "SOA"],
  "has_records": true,
  "network_errors": 0,
  "errors": ["A: dig exit 9"]
}
```
`network_errors` counts queries that never completed: a non-zero `dig` exit
**or** a `;;` diagnostic line in stdout (`dig` writes its complaints to stdout
with stderr empty — measured 2026-09-26: rc=9, both `;;` lines on stdout,
0 bytes on stderr). `errors` carries one `"<type>: <detail>"` string per
failure, read from whichever stream actually carried the message — the last
stderr line if one was written, otherwise `dig`'s own `;; …` line from stdout,
otherwise `dig exit <code>`. Both are present in every run, so consumers can
distinguish a negative answer from an unreachable resolver without parsing
stderr, and a `;;` line can only ever appear in `errors`, never in `records`
(`split_dig_output()` is the single place raw stdout becomes values).

**Dependencies**: dig (bind9-dnsutils), python3

**Tests**:
- `bash tests/test_dns_verify.sh` → **40 assertions**, hermetic (<5s, scenario-driven `dig` stub on PATH: `ok`/`empty`/`netfail`/`sleep`; one case runs with a PATH containing only python3 to simulate a missing `dig` — live DNS never touched): arg/`--help` contract, type + domain rejection (`2`), happy path + JSONL/human/`--format` contracts, default 7-type query, **exit-code contract `3` vs `4`** (unreachable resolver and missing `dig` both → `4`, not `3`), **`--timeout` honored** (`+time=N` reaches dig, a 5s-hung dig is cut at ~3s → `4`, `--timeout 0` → `2`), `--server` pass-through, **2 mutations** (M1 folds network errors back into `exit 3`; M2 defeats the type guard — both must RUN and give the wrong verdict, each precondition-asserted to be surgical)
- Live: `dns-verify gladex.de` → exit 0, `gladex.de BOGUSTYPE` → exit 2 (was 0 + fabricated record), `--server 203.0.113.1 --timeout 1` → exit 4 (was 3), `this-domain-should-be-available-12345.de` → exit 3 (no records), `--format human … NS` → human-readable NS records
- `bash tests/test_dig_tools_transport.sh` → **80 assertions** (shared with `domain-availability-check` and `ip-drift-check`), hermetic (scenario `dig`/`curl`/`whois` stubs + a `127.0.0.1` IP-echo server): the dig transport-shape contract in three shapes — `transport` (measured: `;;` on stdout, stderr empty, rc 9), `transport-err` (`;;` on stderr, rc 9), `masked` (`;;` on stdout with rc 0, the wrapper-masked shape real dig does not produce) — asserting exit **4** and `network_errors=7` in every one, **no `;;` in `records`**, the complaint reaching the caller as an *error* (JSON and human, quoted only under `Network errors:` with no `Records:` section), harness sanity that pins the stub itself to the measurement (stdout diagnostics + **stderr empty** + rc 9), `no-json` never counting as "no `;;`", and **4 mutants** (drop `or diagnostics`, stop rejecting `;;` lines, and their two siblings) each precondition-asserted surgical, byte-different and non-empty-output. Pre-fix replay against the `HEAD` blobs → **58 passed / 13 failed** (7 of them this tool: 2 wrong-stream detail + 5 masked-shape).

**Status**: Built ✅ (exit-code/`--timeout`/type-validation contract hardened 2026-09-24)
**Status**: Built ✅

---

## healthcheck

=================== healthcheck
## healthcheck

**Purpose**: Check /healthz endpoint on dev/prod environments.

**Location**: `/data/tools/healthcheck`

**Usage**:
```
healthcheck [--help] [--format json|human] [--timeout SECONDS] [dev|prod|both]
```

**Options**:
- `--format json|human` - Output format (default: human)
- `--timeout SECONDS` - Timeout per request in seconds (default: 5)
- `--help` - Show help message

**Exit codes**:
- `0` - All checked environments healthy
- `1` - One or more environments unhealthy
- `2` - Invalid arguments / usage error
- `3` - Connection error (service not reachable)

**Output (JSON)**: Array of objects with structure:
```json
{
  "url": "http://localhost:8000/healthz",
  "timestamp": "2026-09-20T12:23:03Z",
  "healthy": true,
  "status_code": 200,
  "response": "{\"status\":\"ok\",\"service\":\"gladex-investor-app\",\"env\":\"dev\",...}",
  "error": null
}
```

**Dependencies**: python3 (stdlib only)

**Tests**:
- `healthcheck dev` → exit 0, shows HEALTHY
- `healthcheck prod` → exit 0, shows HEALTHY
- `healthcheck --format json both` → valid JSON with both envs
- `healthcheck --help` → shows usage

**Status**: Built ✅

---

## promote-dev-to-prod

**Purpose**: Promote dev environment to prod, behind four mandatory gates — reviewer verdict, dev-sync, commit lint, ship-tree (git checkout, build, restart).

**Location**: `/data/repo/tools/promote-dev-to-prod`

=================== tls-check
## tls-check

**Purpose**: Verify TLS certificates for gladex.de host names with a real handshake (stdlib only, no ACME API) — apex vhost, every sub domain, and the LE certs on the mail ports (465/993). Distinguishes "never reached a server" (3) from "server reached, certificate refused" (5), and **describes rejected certs** (cn/sans/dates re-fetched with verification off) instead of returning `cn=null` like the pre-2026-09-24 version did.

**Location**: `/data/repo/tools/tls-check`

**Usage**:
```
tls-check [domain...] [--format json|human] [--warn-days 30] [--port 443] [--timeout 5] [--ca-file PEM]
```

**Options**:
- `domain...` - Hosts to check (default: every SAN of **both** live LE lineages — 9 names: the 7 on the gladex.de cert plus `cloud.gladex.de` / `photos.gladex.de` on the 2026-09-26 `photos.gladex.de` cert)
- `--format json|human` - Output format (default: human; json = one compact object per line, NDJSON)
- `--warn-days N` - Expiry warning threshold in days (default: 30)
- `--port N` - TCP port (default: 443; e.g. 993=IMAPS, 465=SMTPS)
- `--timeout S` - Connect/handshake timeout in seconds (default: 5; also bounds DNS-failure/refused paths)
- `--ca-file PEM` - Extra CA bundle trusted **in addition to** the system store (validated: missing/dir/garbage → exit 2)
- `--help` - Shows usage + exit-code table

**Exit codes**:
- `0` - All checked certs valid for more than --warn-days
- `1` - General error
- `2` - Invalid arguments (bad port/timeout/warn-days/ca-file/domain syntax)
- `3` - Unreachable: DNS failure, connection refused, handshake timeout (no verdict)
- `4` - Expiring: valid, but notAfter within --warn-days
- `5` - Rejected: `reject_reason` ∈ expired | hostname_mismatch | untrusted

**JSON contract** (one object per domain; keys: `domain, port, cn, sans, not_before, not_after, days_remaining, covers_name, valid, status, reject_reason, error`; `status` ∈ ok | expiring | rejected | unreachable): rejected entries still carry full `cn`/`sans`/dates/`days_remaining` (fetched via a `CERT_NONE` reconnect + offline DER parse), `covers_name` says whether the SAN list covered the requested name (true even when the chain/date verdict failed).

**Dependencies**: python3 (stdlib only: ssl, socket, json, argparse, tempfile)

**Tests**: `tests/test_tls_check.sh` — 102 assertions: hermetic local TLS servers with sandbox-minted certs (valid/expiring/expired/wrong-name/untrusted), refused/hang/DNS-failure endpoints, arg-validation matrix, JSON contract, the 9-name two-lineage defaults via AST, 4 mutations (M1 folds rejects back into exit 3, M2 makes --timeout a no-op, M3 defeats expired classification, M4 drops team.gladex.de from defaults — each must be caught AND surgical), plus one live case against gladex.de (apex + the two new-lineage names + :993 + :465, no expiring dates pinned).

**Status**: Built ✅ (hardened 2026-09-24: exit-code collapse + undescribed rejects + 5-of-7 defaults fixed, first test suite added; 2026-09-26: defaults widened 7 → 9 so the second LE lineage — `photos.`/`cloud.`, issued that day — is monitored by a bare run instead of only by hand)

---

## smtp-relay-probe

**Purpose**: Verify SMTP relay stays CLOSED on our mail ports — unauthenticated external RCPT must be refused — AND that every endpoint still greets correctly. Live-probes :25/:587/:465 locally and through the full public path (VPS 77.90.15.49:25 forward). Never sends DATA, so no message can ever be queued. The **220-banner assertion** (exit 4) guards against a regression of the 2026-09-23 :25-greeting bug: a connected endpoint that does not greet `220 ` within the timeout is a hard failure, never an "ok" — one broken endpoint among four can no longer pass green.
> build · mimo-v2.6-flash-free

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

App log tail — prod-8001.log (last 60 lines)
[Mon Sep 28 16:24:51 2026] 127.0.0.1:34114 Accepted
[Mon Sep 28 16:24:51 2026] 127.0.0.1:34114 Closing
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34118 Accepted
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34118 Closing
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34120 Accepted
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34120 Closing
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34134 Accepted
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34134 Closing
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34148 Accepted
[Mon Sep 28 16:24:52 2026] 127.0.0.1:34148 Closing
[Mon Sep 28 16:24:53 2026] 127.0.0.1:34160 Accepted
[Mon Sep 28 16:24:53 2026] 127.0.0.1:34160 Closing
[Mon Sep 28 16:24:53 2026] 127.0.0.1:34172 Accepted
[Mon Sep 28 16:24:53 2026] 127.0.0.1:34172 Closing
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34180 Accepted
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34180 Closing
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34188 Accepted
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34188 Closing
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34192 Accepted
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34192 Closing
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34206 Accepted
[Mon Sep 28 16:24:54 2026] 127.0.0.1:34206 Closing
[Mon Sep 28 16:24:55 2026] 127.0.0.1:40478 Accepted
[Mon Sep 28 16:24:55 2026] 127.0.0.1:40478 Closing
[Mon Sep 28 16:24:55 2026] 127.0.0.1:40484 Accepted
[Mon Sep 28 16:24:55 2026] 127.0.0.1:40484 Closing
[Mon Sep 28 16:24:55 2026] 127.0.0.1:40498 Accepted
[Mon Sep 28 16:24:55 2026] 127.0.0.1:40498 Closing
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40512 Accepted
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40512 Closing
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40520 Accepted
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40520 Closing
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40536 Accepted
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40536 Closing
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40542 Accepted
[Mon Sep 28 16:24:56 2026] 127.0.0.1:40542 Closing
[Mon Sep 28 16:24:57 2026] 127.0.0.1:40544 Accepted
[Mon Sep 28 16:24:57 2026] 127.0.0.1:40544 Closing
[Mon Sep 28 16:24:57 2026] 127.0.0.1:40558 Accepted
[Mon Sep 28 16:24:57 2026] 127.0.0.1:40558 Closing
[Mon Sep 28 16:24:57 2026] 127.0.0.1:40568 Accepted
[Mon Sep 28 16:24:57 2026] 127.0.0.1:40568 Closing
[Mon Sep 28 16:24:58 2026] 127.0.0.1:40574 Accepted
[Mon Sep 28 16:24:58 2026] 127.0.0.1:40574 Closing
[Mon Sep 28 16:24:58 2026] 127.0.0.1:40580 Accepted
[Mon Sep 28 16:24:58 2026] 127.0.0.1:40580 Closing
[Mon Sep 28 16:24:58 2026] 127.0.0.1:40592 Accepted
[Mon Sep 28 16:24:58 2026] 127.0.0.1:40592 Closing
[Mon Sep 28 16:24:59 2026] 127.0.0.1:40608 Accepted
[Mon Sep 28 16:24:59 2026] 127.0.0.1:40608 Closing
[Mon Sep 28 16:24:59 2026] 127.0.0.1:40612 Accepted
[Mon Sep 28 16:24:59 2026] 127.0.0.1:40612 Closing
[Mon Sep 28 16:25:50 2026] 127.0.0.1:36540 Accepted
[Mon Sep 28 16:25:50 2026] 127.0.0.1:36540 Closing
[Mon Sep 28 16:26:50 2026] 127.0.0.1:45128 Accepted
[Mon Sep 28 16:26:50 2026] 127.0.0.1:45128 Closing
[Mon Sep 28 16:27:50 2026] 127.0.0.1:40300 Accepted
[Mon Sep 28 16:27:50 2026] 127.0.0.1:40300 Closing
[Mon Sep 28 16:27:55 2026] 127.0.0.1:60234 Accepted

Generated 2026-09-28 14:27:55 UTC · Gladex.de