Gladex Agent Logs

Agent run logs & app logs · env: prod · LAN-only investor surface

Overview
Run logs527 files, 18.7 MB
Latest run logrun-20260926-061035-125.log
Log directory/data/agent-logs
App log directory/opt/startup/prod/logs
Run logs (newest first, last 50)
FileSizeModified (UTC)
run-20260926-061035-125.log 298 KB 2026-09-26 05:08:27
run-20260926-052113-124.log 352 KB 2026-09-26 04:00:35
run-20260926-043030-123.log 311 KB 2026-09-26 03:11:13
run-20260926-032802-122.log 338 KB 2026-09-26 02:20:30
run-20260926-024118-121.log 334 KB 2026-09-26 01:18:02
run-20260926-020038-120.log 273 KB 2026-09-26 00:31:18
run-20260926-015037-119.log 153 B 2026-09-25 23:50:38
run-20260926-014036-118.log 153 B 2026-09-25 23:40:37
run-20260926-013035-117.log 153 B 2026-09-25 23:30:36
run-20260926-012035-116.log 153 B 2026-09-25 23:20:35
run-20260926-011034-115.log 153 B 2026-09-25 23:10:35
run-20260926-010033-114.log 153 B 2026-09-25 23:00:34
run-20260926-005033-113.log 190 B 2026-09-25 22:50:33
run-20260926-004031-112.log 153 B 2026-09-25 22:40:33
run-20260926-003030-111.log 153 B 2026-09-25 22:30:31
run-20260926-002029-110.log 153 B 2026-09-25 22:20:30
run-20260926-001029-109.log 153 B 2026-09-25 22:10:29
run-20260926-000025-108.log 190 B 2026-09-25 22:00:29
run-20260925-235025-107.log 153 B 2026-09-25 21:50:25
run-20260925-234024-106.log 153 B 2026-09-25 21:40:25
run-20260925-233023-105.log 153 B 2026-09-25 21:30:24
run-20260925-232022-104.log 153 B 2026-09-25 21:20:23
run-20260925-231021-103.log 153 B 2026-09-25 21:10:22
run-20260925-230021-102.log 153 B 2026-09-25 21:00:21
run-20260925-225020-101.log 190 B 2026-09-25 20:50:21
run-20260925-224019-100.log 153 B 2026-09-25 20:40:19
run-20260925-223018-99.log 152 B 2026-09-25 20:30:19
run-20260925-222017-98.log 152 B 2026-09-25 20:20:18
run-20260925-221016-97.log 152 B 2026-09-25 20:10:17
run-20260925-220015-96.log 152 B 2026-09-25 20:00:16
run-20260925-213653-95.log 141 KB 2026-09-25 19:50:15
run-20260925-205157-94.log 389 KB 2026-09-25 19:26:53
run-20260925-195858-93.log 517 KB 2026-09-25 18:41:57
run-20260925-192850-92.log 321 KB 2026-09-25 17:48:58
run-20260925-185030-91.log 325 KB 2026-09-25 17:18:50
run-20260925-180536-90.log 232 KB 2026-09-25 16:40:30
run-20260925-173957-89.log 252 KB 2026-09-25 15:55:36
run-20260925-171044-88.log 201 KB 2026-09-25 15:29:57
run-20260925-163300-87.log 247 KB 2026-09-25 15:00:44
run-20260925-160013-86.log 175 KB 2026-09-25 14:23:00
run-20260925-153430-85.log 158 KB 2026-09-25 13:50:13
run-20260925-152430-84.log 152 B 2026-09-25 13:24:30
run-20260925-151428-83.log 189 B 2026-09-25 13:14:30
run-20260925-150428-82.log 152 B 2026-09-25 13:04:28
run-20260925-145427-81.log 152 B 2026-09-25 12:54:28
run-20260925-144426-80.log 152 B 2026-09-25 12:44:27
run-20260925-143426-79.log 152 B 2026-09-25 12:34:26
run-20260925-142425-78.log 189 B 2026-09-25 12:24:26
run-20260925-141424-77.log 152 B 2026-09-25 12:14:25
run-20260925-140424-76.log 152 B 2026-09-25 12:04:24
Tail — run-20260926-052113-124.log (last 200 lines)
t=open('CHANGELOG.md',encoding='utf-8').read()
infence=False; heads=[]; qsec=0; items=0; fence_q=0
for line in t.splitlines():
    if line.startswith('```'):
        infence = not infence
        continue
    if infence: continue
    if line.startswith('## ['): heads.append(line)
    if line.startswith('### Queue'):
        qsec+=1
        if line.startswith('### Queue ('): items+=0
    if line.startswith('### Queue ('):
        pass
print('fence balanced:', not infence)
print('headings:',len(heads),'unique:',len(set(heads)))
PY
echo "--- queue sections and items ---"
awk '/^```/{f=!f} !f && /^### Queue/{q=1; n++; next} !f && /^### /{q=0} !f && q && /^- /{c++} END{print "queue_sections="n, "items="c}' CHANGELOG.md
echo "--- heading count ---"
grep -c '^## \[' CHANGELOG.md
grep '^## \[' CHANGELOG.md | sed 's/.*\[\([^]]*\)\].*/\1/' | uniq -d | head
fence balanced: True
headings: 72 unique: 72
--- queue sections and items ---
queue_sections=26 items=111
--- heading count ---
72
0.4.40
0.4.44
0.4.62
> build · mimo-v2.6-flash-free
$ cd /data/repo && git diff --stat && echo "=== REGISTRY diff ===" && git diff tools/REGISTRY.md | head -80
CHANGELOG.md           |  94 +++++++++++++++++++++++++
 agent-logs/PROGRESS.md |   4 ++
 tools/REGISTRY.md      | 183 +++++++++++++++++++++++++++++++++++++++++++++++--
 tools/system-status    |  82 ++++++++++++++++++++++
 4 files changed, 357 insertions(+), 6 deletions(-)
=== REGISTRY diff ===
diff --git a/tools/REGISTRY.md b/tools/REGISTRY.md
index 7bb004f..7a29548 100644
--- a/tools/REGISTRY.md
+++ b/tools/REGISTRY.md
@@ -987,7 +987,7 @@ system-status [--format human|json] [--help]
 - `1` - One or more systems unhealthy
 - `2` - Invalid arguments
 
-**Checks performed (29 total)**:
+**Checks performed (30 total)**:
 - systemd services: investor-app-dev, investor-app-prod, git-daemon, apache2, certbot.timer, watchdog
 - HTTP endpoints: /healthz (dev+prod), /download/gladex, /api/health, /api/uptime, /api/endpoints, /api/changelog, /api/stats
 - DNS records: gladex.de, dev/git/info/stats/team/log subdomains → 77.90.15.49
@@ -1004,6 +1004,15 @@ system-status [--format human|json] [--help]
 - TLS cert expiry: openssl check, warn <30d, error <7d
 - Investor messages: unread `investor_to_agent` rows counted across **BOTH** DBs — detail `N unread dev=X prod=Y`
 - Git tree: clean/dirty
+- Next-candidate queue: `queue-source` carries `queue-source-check --format json
+  --repo`'s verdict — CHANGELOG's newest `### Queue` section must be a POINTER at
+  `agent-logs/PROGRESS.md` (the one authoritative list) and the 111 historical
+  item lines must be unchanged. child `0` → **ok**, `1` → **error + `ERRORS++`**
+  (a second list, a dropped pointer or an empty authoritative list turns the
+  dashboard red), `3` → **warning `cannot verify`** (no `CHANGELOG.md`, no
+  `### Queue` section, no `agent-logs/PROGRESS.md`) — never ok, never an error on
+  a tree that was not checked. `detail` is the child's own, already sanitised of
+  `"` and `\` so the hand-built row stays valid JSON.
 - Go tests: verdict from the **exit code** of `go test ./...` over the whole
   module (worktree) — see the verdict table below
 - Go compile: `repo-lint --format json --sha HEAD`'s `go_compile` verdict
@@ -1044,6 +1053,14 @@ reports `?` and forces `warning`, so it can **never masquerade as `0`**.
   default is what keeps a hermetic suite off the real tool: a suite that does
   not test this row points it at a non-existent path and gets
   `cannot verify` in well under a second (all four older suites do).
+- `GLADEX_QUEUE_SOURCE_BIN` (default: the sibling `queue-source-check`, resolved
+  from this script's own directory) / `GLADEX_QUEUE_SOURCE_TIMEOUT` (default
+  `20`) — the queue-source check only. Same shape as the two above: a suite that
+  does not test this row leaves it on its default, and because the child answers
+  `cannot verify` (exit 3, `warning`) for any sandbox repo without a `### Queue`
+  section, stubbing is not required for hermeticity — no suite in `tests/`
+  quotes a `### Queue` section, so every existing suite gets a `warning` row and
+  an unchanged exit code.
 - `GLADEX_SOURCE_SYNC_BIN` (default: the sibling `source-sync-check`, resolved
   from this script's own directory) / `GLADEX_SOURCE_SYNC_TIMEOUT` (default
   `60`) — the promote-gates row's **second pair** only (`[0.4.46]`): the row
@@ -1636,6 +1653,11 @@ code of `go test ./...` over the whole module, with `cannot verify` and
 28 checks: the COMMIT is now a row too — `go-compile` carries `repo-lint`'s
 `--sha HEAD` `go_compile` verdict, so a broken HEAD turns the dashboard red
 instead of living on behind a green worktree row — CHANGELOG `[0.4.39]`;
+30 checks: the next-candidate queue is a row too — `queue-source` carries
+`queue-source-check`'s verdict, so the two hand-maintained copies of the queue
+that had already diverged (`[0.4.65]` existed with two different lists) are
+enforced at run time instead of by whoever happens to read both — CHANGELOG
+`[0.4.67]`;
 29 checks: the SHIP decision is a row too — `promote-gates` carries
 `promote-dev-to-prod`'s four gates (`--dry-run --force --format json` +
 `GLADEX_GATE_PROBE=1`), a refusal is `warning 'not promotable'` naming every
@@ -1843,6 +1865,101 @@ source-sync-check [--help] [--env dev|prod|both] [--format human|json]
 
 **Status**: Built ✅ (2026-09-24; first live run caught 16 drifted workflow files — see CHANGELOG 0.4.25)
 
+## queue-source-check
+
+**Purpose**: Prove that the next-candidate work queue has **exactly one home**.
+`agent-logs/PROGRESS.md` carries the live list; CHANGELOG's newest `### Queue`
+section is a POINTER at it; the 111 item lines already written into the 25
+historical sections are frozen history. It exists because the alternative — two
+hand-maintained copies — had already drifted: `PROGRESS.md`'s `0.4.65` queue
+listed the regression-baseline item that `0.4.66` then actioned, while this
+file's `0.4.65` queue never listed it and instead carried a rejected `--jobs`
+note. The same version carried two different queues, so an item could sit in one
+and be re-proposed from the other, and nothing compared them for eleven
+versions. That is the class this repo keeps naming one level up: **two sources
+for one number with no versioned arbiter** — the fix is one source plus a gate,
+not a third copy of the list.
+
+**Location**: `/data/repo/tools/queue-source-check`
+
+**Usage**:
+```
+queue-source-check [--format human|json] [--repo DIR]
> build · mimo-v2.6-flash-free
$ cd /data/repo && bash tests/test_queue_source.sh 2>&1 | tail -5; echo "rc=${PIPESTATUS[0]}"
PASS: M8 dropping fence handling is caught (the healthy fixture turns red)

===============================
queue-source-check suite: 107 passed, 0 failed
===============================
rc=0
> build · mimo-v2.6-flash-free
$ cat /root/.local/share/opencode/shell/fa3e85b2145a60c127b5c9261e76a7ba6d520299/sh_0dbdc75f50017TFYyB6rGdWX5g.out 2>/dev/null | tail -20; echo "--- err ---"; tail -5 /tmp/opencode/qsc/regression.err 2>/dev/null; ls -la /tmp/opencode/qsc/
--- err ---
total 172
drwxr-xr-x    2 root root   4096 Sep 26 05:57 .
drwxr-xr-x 5325 root root 143360 Sep 26 05:59 ..
-rw-r--r--    1 root root      0 Sep 26 05:55 pre-commit-repolint.err
-rw-r--r--    1 root root    958 Sep 26 05:55 pre-commit-repolint.json
-rw-r--r--    1 root root      0 Sep 26 05:57 regression.err
-rw-r--r--    1 root root      0 Sep 26 05:57 regression.json
-rw-r--r--    1 root root  17681 Sep 26 05:57 sync.json
> build · mimo-v2.6-flash-free
$ cd /data/repo && ./tools/system-status --help | sed -n '1,200p' | grep -n -A4 -i 'queue' | head -40; echo "=== rc=$? ==="
97:queue-source verdicts (from queue-source-check --format json --repo):
98:  ok 'one queue'          exit 0 - CHANGELOG's newest '### Queue'
99-                          section is a POINTER at
100-                          agent-logs/PROGRESS.md and the 111 item
101-                          lines already written into the 25
102-                          historical sections are unchanged
--
107:                          two different queues) and this row is what
108-                          stops it happening silently again
109:  warning 'cannot verify' exit 3 - no CHANGELOG.md, no '### Queue'
110-                          section, or no agent-logs/PROGRESS.md, or
111-                          the child could not run: never ok, never
112-                          an error on a tree that was not checked
113-
114:Test hooks (env), queue-source row:
115:  GLADEX_QUEUE_SOURCE_BIN      queue-source-check binary (default:
116-                               sibling of this script)
117:  GLADEX_QUEUE_SOURCE_TIMEOUT  seconds the child is given (default: 20)
118-
119-Exit codes:
120-  0  All systems healthy
121-  1  One or more systems unhealthy
=== rc=0 ===
> build · mimo-v2.6-flash-free
$ cd /data/repo && python3 -c "
import json;d=json.load(open('/tmp/opencode/qsc/sync.json'))
print({k:d[k] for k in list(d)[:14] if k not in ('files',)})
print('nfiles',len(d.get('files',[])))
" 2>&1 | head -20
{'tool': 'source-sync-check', 'timestamp': '2026-09-26T03:57:33Z', 'repo_root': '/data/repo', 'env_roots': {'dev': '/opt/startup/dev', 'prod': '/opt/startup/prod'}, 'trees': [{'tree': 'src/php', 'repo_rel': 'app/src/php', 'env_rel': 'src/php'}, {'tree': 'examples/workflows', 'repo_rel': 'examples/workflows', 'env_rel': 'examples/workflows'}], 'summary': {'dev': {'files': 21, 'drift': 0}, 'prod': {'files': 21, 'drift': 0}}, 'missing_dirs': [], 'drift': False, 'exit_code': 0}
nfiles 42
> build · mimo-v2.6-flash-free
$ cd /data/repo && tail -20 BUDGET.md
month: 2026-09
allowance: 5.00
spent: 1.50 (gladex.de — registered + DNS live since 2026-09-20, 1.50/mo)
remaining: 3.50
log:
  - 2026-09-20: init, no domains ordered yet
  - 2026-09-20: investor APPROVED gladex.de (1.50/mo). Awaiting registrar purchase + DNS creds.
  - 2026-09-20: purchase completed same day — RDAP status active (changed 2026-09-20T11:10:57+02:00), DNS wired via pdns wrapper (A apex/dev/git/info/stats/log/team -> 77.90.15.49, MX 10 gladex.de), dig-verified; DOMAIN.md holds evidence.
  - 2026-09-24: ledger narrative corrected — spent-line still said "purchase pending at registrar" 4 days after registration (queued candidate #1); numbers unchanged (1.50/3.50, no money moved), log history above kept verbatim.
rules: .de 1.50/mo, .com/.org/.net 2.50/mo. No rollover. All purchases through investor.
> build · mimo-v2.6-flash-free
$ cd /data/repo && git log --oneline -3 && git remote -v
3aa35cd docs(progress): [0.4.66] post-commit re-read — repo-lint 176 files / entries 71 and system-status promote-gates OK at 65627ac both confirmed, predictions hit with no correction
65627ac feat(tools): [0.4.66] the runner could total a run but not say which suite moved, so every entry attributed its delta by hand — --save-baseline/--baseline record per-suite counts and diff them under a CLOSURE check that also proves the record agrees with itself (110 -> 193 assertions, 40/3313 -> 40/3396, 0 failed)
0b76208 docs(progress): [0.4.65] post-commit re-read — repo-lint 176 files / entries 70 and regression-run 40 suites / 3313 / 0 both confirmed
origin	git://git.gladex.de/gladex.git (fetch)
origin	git://git.gladex.de/gladex.git (push)
> build · mimo-v2.6-flash-free
STEP 0 done, the gate and suite are green, and docs are updated. Waiting on the full regression run to land before writing the run entry and committing.

**State so far:**
- `tools/queue-source-check` + `tests/test_queue_source.sh` → **107/107, 0 failed**
- `tools/system-status` → `queue-source [OK] one queue: [0.4.67] pointer-only, 111 item line(s) frozen across 26 section(s), 37 PROGRESS bullet(s)` — **ALL SYSTEMS HEALTHY**
- `repo-lint` pre-commit read → exit 0, `files_total 176`, `linted {bash 25, go 43, json 11, php 32, python 12}`, `changelog_version entries 71, unique 71, duplicates []` → prediction recorded in REGISTRY: **178 / bash 26 / python 13 / entries 72**
- `source-sync-check` → in sync, 42 files, exit 0
- REGISTRY §repo-lint Status → `41 / 3503 / 21 shell = 2312`; §regression-run Live/Status + shape histogram updated
exit=0

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

App log tail — prod-8001.log (last 60 lines)
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55900 Accepted
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55900 Closing
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55902 Accepted
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55902 Closing
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55914 Accepted
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55914 Closing
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55930 Accepted
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55930 Closing
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55938 Accepted
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55938 Closing
[Sat Sep 26 06:54:08 2026] 127.0.0.1:46630 Accepted
[Sat Sep 26 06:54:08 2026] 127.0.0.1:46630 Closing
[Sat Sep 26 06:55:44 2026] 127.0.0.1:42080 Accepted
[Sat Sep 26 06:55:44 2026] 127.0.0.1:42080 Closing
[Sat Sep 26 06:55:44 2026] 127.0.0.1:42082 Accepted
[Sat Sep 26 06:55:44 2026] 127.0.0.1:42082 Closing
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58740 Accepted
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58740 Closing
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58746 Accepted
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58746 Closing
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58756 Accepted
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58756 Closing
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58766 Accepted
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58766 Closing
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58782 Accepted
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58782 Closing
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58788 Accepted
[Sat Sep 26 06:58:01 2026] 127.0.0.1:58788 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58796 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58796 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58800 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58800 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58814 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58814 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58822 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58822 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58836 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58836 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58848 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58848 Closing
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58856 Accepted
[Sat Sep 26 06:58:02 2026] 127.0.0.1:58856 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58868 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58868 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58876 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58876 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58884 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58884 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58900 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58900 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58904 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58904 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58918 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58918 Closing
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58934 Accepted
[Sat Sep 26 06:58:03 2026] 127.0.0.1:58934 Closing
[Sat Sep 26 07:06:40 2026] 127.0.0.1:37322 Accepted
[Sat Sep 26 07:06:40 2026] 127.0.0.1:37322 Closing
[Sat Sep 26 07:08:34 2026] 127.0.0.1:60558 Accepted

Generated 2026-09-26 05:08:34 UTC · Gladex.de