Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 527 files, 18.7 MB |
| Latest run log | run-20260926-061035-125.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
| File | Size | Modified (UTC) |
|---|---|---|
| run-20260926-061035-125.log | 298 KB | 2026-09-26 05:08:27 |
| run-20260926-052113-124.log | 352 KB | 2026-09-26 04:00:35 |
| run-20260926-043030-123.log | 311 KB | 2026-09-26 03:11:13 |
| run-20260926-032802-122.log | 338 KB | 2026-09-26 02:20:30 |
| run-20260926-024118-121.log | 334 KB | 2026-09-26 01:18:02 |
| run-20260926-020038-120.log | 273 KB | 2026-09-26 00:31:18 |
| run-20260926-015037-119.log | 153 B | 2026-09-25 23:50:38 |
| run-20260926-014036-118.log | 153 B | 2026-09-25 23:40:37 |
| run-20260926-013035-117.log | 153 B | 2026-09-25 23:30:36 |
| run-20260926-012035-116.log | 153 B | 2026-09-25 23:20:35 |
| run-20260926-011034-115.log | 153 B | 2026-09-25 23:10:35 |
| run-20260926-010033-114.log | 153 B | 2026-09-25 23:00:34 |
| run-20260926-005033-113.log | 190 B | 2026-09-25 22:50:33 |
| run-20260926-004031-112.log | 153 B | 2026-09-25 22:40:33 |
| run-20260926-003030-111.log | 153 B | 2026-09-25 22:30:31 |
| run-20260926-002029-110.log | 153 B | 2026-09-25 22:20:30 |
| run-20260926-001029-109.log | 153 B | 2026-09-25 22:10:29 |
| run-20260926-000025-108.log | 190 B | 2026-09-25 22:00:29 |
| run-20260925-235025-107.log | 153 B | 2026-09-25 21:50:25 |
| run-20260925-234024-106.log | 153 B | 2026-09-25 21:40:25 |
| run-20260925-233023-105.log | 153 B | 2026-09-25 21:30:24 |
| run-20260925-232022-104.log | 153 B | 2026-09-25 21:20:23 |
| run-20260925-231021-103.log | 153 B | 2026-09-25 21:10:22 |
| run-20260925-230021-102.log | 153 B | 2026-09-25 21:00:21 |
| run-20260925-225020-101.log | 190 B | 2026-09-25 20:50:21 |
| run-20260925-224019-100.log | 153 B | 2026-09-25 20:40:19 |
| run-20260925-223018-99.log | 152 B | 2026-09-25 20:30:19 |
| run-20260925-222017-98.log | 152 B | 2026-09-25 20:20:18 |
| run-20260925-221016-97.log | 152 B | 2026-09-25 20:10:17 |
| run-20260925-220015-96.log | 152 B | 2026-09-25 20:00:16 |
| run-20260925-213653-95.log | 141 KB | 2026-09-25 19:50:15 |
| run-20260925-205157-94.log | 389 KB | 2026-09-25 19:26:53 |
| run-20260925-195858-93.log | 517 KB | 2026-09-25 18:41:57 |
| run-20260925-192850-92.log | 321 KB | 2026-09-25 17:48:58 |
| run-20260925-185030-91.log | 325 KB | 2026-09-25 17:18:50 |
| run-20260925-180536-90.log | 232 KB | 2026-09-25 16:40:30 |
| run-20260925-173957-89.log | 252 KB | 2026-09-25 15:55:36 |
| run-20260925-171044-88.log | 201 KB | 2026-09-25 15:29:57 |
| run-20260925-163300-87.log | 247 KB | 2026-09-25 15:00:44 |
| run-20260925-160013-86.log | 175 KB | 2026-09-25 14:23:00 |
| run-20260925-153430-85.log | 158 KB | 2026-09-25 13:50:13 |
| run-20260925-152430-84.log | 152 B | 2026-09-25 13:24:30 |
| run-20260925-151428-83.log | 189 B | 2026-09-25 13:14:30 |
| run-20260925-150428-82.log | 152 B | 2026-09-25 13:04:28 |
| run-20260925-145427-81.log | 152 B | 2026-09-25 12:54:28 |
| run-20260925-144426-80.log | 152 B | 2026-09-25 12:44:27 |
| run-20260925-143426-79.log | 152 B | 2026-09-25 12:34:26 |
| run-20260925-142425-78.log | 189 B | 2026-09-25 12:24:26 |
| run-20260925-141424-77.log | 152 B | 2026-09-25 12:14:25 |
| run-20260925-140424-76.log | 152 B | 2026-09-25 12:04:24 |
Tail — run-20260926-052113-124.log (last 200 lines)
t=open('CHANGELOG.md',encoding='utf-8').read()
infence=False; heads=[]; qsec=0; items=0; fence_q=0
for line in t.splitlines():
if line.startswith('```'):
infence = not infence
continue
if infence: continue
if line.startswith('## ['): heads.append(line)
if line.startswith('### Queue'):
qsec+=1
if line.startswith('### Queue ('): items+=0
if line.startswith('### Queue ('):
pass
print('fence balanced:', not infence)
print('headings:',len(heads),'unique:',len(set(heads)))
PY
echo "--- queue sections and items ---"
awk '/^```/{f=!f} !f && /^### Queue/{q=1; n++; next} !f && /^### /{q=0} !f && q && /^- /{c++} END{print "queue_sections="n, "items="c}' CHANGELOG.md
echo "--- heading count ---"
grep -c '^## \[' CHANGELOG.md
grep '^## \[' CHANGELOG.md | sed 's/.*\[\([^]]*\)\].*/\1/' | uniq -d | head
fence balanced: True
headings: 72 unique: 72
--- queue sections and items ---
queue_sections=26 items=111
--- heading count ---
72
0.4.40
0.4.44
0.4.62
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && git diff --stat && echo "=== REGISTRY diff ===" && git diff tools/REGISTRY.md | head -80
CHANGELOG.md | 94 +++++++++++++++++++++++++
agent-logs/PROGRESS.md | 4 ++
tools/REGISTRY.md | 183 +++++++++++++++++++++++++++++++++++++++++++++++--
tools/system-status | 82 ++++++++++++++++++++++
4 files changed, 357 insertions(+), 6 deletions(-)
=== REGISTRY diff ===
diff --git a/tools/REGISTRY.md b/tools/REGISTRY.md
index 7bb004f..7a29548 100644
--- a/tools/REGISTRY.md
+++ b/tools/REGISTRY.md
@@ -987,7 +987,7 @@ system-status [--format human|json] [--help]
- `1` - One or more systems unhealthy
- `2` - Invalid arguments
-**Checks performed (29 total)**:
+**Checks performed (30 total)**:
- systemd services: investor-app-dev, investor-app-prod, git-daemon, apache2, certbot.timer, watchdog
- HTTP endpoints: /healthz (dev+prod), /download/gladex, /api/health, /api/uptime, /api/endpoints, /api/changelog, /api/stats
- DNS records: gladex.de, dev/git/info/stats/team/log subdomains → 77.90.15.49
@@ -1004,6 +1004,15 @@ system-status [--format human|json] [--help]
- TLS cert expiry: openssl check, warn <30d, error <7d
- Investor messages: unread `investor_to_agent` rows counted across **BOTH** DBs — detail `N unread dev=X prod=Y`
- Git tree: clean/dirty
+- Next-candidate queue: `queue-source` carries `queue-source-check --format json
+ --repo`'s verdict — CHANGELOG's newest `### Queue` section must be a POINTER at
+ `agent-logs/PROGRESS.md` (the one authoritative list) and the 111 historical
+ item lines must be unchanged. child `0` → **ok**, `1` → **error + `ERRORS++`**
+ (a second list, a dropped pointer or an empty authoritative list turns the
+ dashboard red), `3` → **warning `cannot verify`** (no `CHANGELOG.md`, no
+ `### Queue` section, no `agent-logs/PROGRESS.md`) — never ok, never an error on
+ a tree that was not checked. `detail` is the child's own, already sanitised of
+ `"` and `\` so the hand-built row stays valid JSON.
- Go tests: verdict from the **exit code** of `go test ./...` over the whole
module (worktree) — see the verdict table below
- Go compile: `repo-lint --format json --sha HEAD`'s `go_compile` verdict
@@ -1044,6 +1053,14 @@ reports `?` and forces `warning`, so it can **never masquerade as `0`**.
default is what keeps a hermetic suite off the real tool: a suite that does
not test this row points it at a non-existent path and gets
`cannot verify` in well under a second (all four older suites do).
+- `GLADEX_QUEUE_SOURCE_BIN` (default: the sibling `queue-source-check`, resolved
+ from this script's own directory) / `GLADEX_QUEUE_SOURCE_TIMEOUT` (default
+ `20`) — the queue-source check only. Same shape as the two above: a suite that
+ does not test this row leaves it on its default, and because the child answers
+ `cannot verify` (exit 3, `warning`) for any sandbox repo without a `### Queue`
+ section, stubbing is not required for hermeticity — no suite in `tests/`
+ quotes a `### Queue` section, so every existing suite gets a `warning` row and
+ an unchanged exit code.
- `GLADEX_SOURCE_SYNC_BIN` (default: the sibling `source-sync-check`, resolved
from this script's own directory) / `GLADEX_SOURCE_SYNC_TIMEOUT` (default
`60`) — the promote-gates row's **second pair** only (`[0.4.46]`): the row
@@ -1636,6 +1653,11 @@ code of `go test ./...` over the whole module, with `cannot verify` and
28 checks: the COMMIT is now a row too — `go-compile` carries `repo-lint`'s
`--sha HEAD` `go_compile` verdict, so a broken HEAD turns the dashboard red
instead of living on behind a green worktree row — CHANGELOG `[0.4.39]`;
+30 checks: the next-candidate queue is a row too — `queue-source` carries
+`queue-source-check`'s verdict, so the two hand-maintained copies of the queue
+that had already diverged (`[0.4.65]` existed with two different lists) are
+enforced at run time instead of by whoever happens to read both — CHANGELOG
+`[0.4.67]`;
29 checks: the SHIP decision is a row too — `promote-gates` carries
`promote-dev-to-prod`'s four gates (`--dry-run --force --format json` +
`GLADEX_GATE_PROBE=1`), a refusal is `warning 'not promotable'` naming every
@@ -1843,6 +1865,101 @@ source-sync-check [--help] [--env dev|prod|both] [--format human|json]
**Status**: Built ✅ (2026-09-24; first live run caught 16 drifted workflow files — see CHANGELOG 0.4.25)
+## queue-source-check
+
+**Purpose**: Prove that the next-candidate work queue has **exactly one home**.
+`agent-logs/PROGRESS.md` carries the live list; CHANGELOG's newest `### Queue`
+section is a POINTER at it; the 111 item lines already written into the 25
+historical sections are frozen history. It exists because the alternative — two
+hand-maintained copies — had already drifted: `PROGRESS.md`'s `0.4.65` queue
+listed the regression-baseline item that `0.4.66` then actioned, while this
+file's `0.4.65` queue never listed it and instead carried a rejected `--jobs`
+note. The same version carried two different queues, so an item could sit in one
+and be re-proposed from the other, and nothing compared them for eleven
+versions. That is the class this repo keeps naming one level up: **two sources
+for one number with no versioned arbiter** — the fix is one source plus a gate,
+not a third copy of the list.
+
+**Location**: `/data/repo/tools/queue-source-check`
+
+**Usage**:
+```
+queue-source-check [--format human|json] [--repo DIR]
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && bash tests/test_queue_source.sh 2>&1 | tail -5; echo "rc=${PIPESTATUS[0]}"
PASS: M8 dropping fence handling is caught (the healthy fixture turns red)
===============================
queue-source-check suite: 107 passed, 0 failed
===============================
rc=0
> build · mimo-v2.6-flash-free
[0m$ [0mcat /root/.local/share/opencode/shell/fa3e85b2145a60c127b5c9261e76a7ba6d520299/sh_0dbdc75f50017TFYyB6rGdWX5g.out 2>/dev/null | tail -20; echo "--- err ---"; tail -5 /tmp/opencode/qsc/regression.err 2>/dev/null; ls -la /tmp/opencode/qsc/
--- err ---
total 172
drwxr-xr-x 2 root root 4096 Sep 26 05:57 .
drwxr-xr-x 5325 root root 143360 Sep 26 05:59 ..
-rw-r--r-- 1 root root 0 Sep 26 05:55 pre-commit-repolint.err
-rw-r--r-- 1 root root 958 Sep 26 05:55 pre-commit-repolint.json
-rw-r--r-- 1 root root 0 Sep 26 05:57 regression.err
-rw-r--r-- 1 root root 0 Sep 26 05:57 regression.json
-rw-r--r-- 1 root root 17681 Sep 26 05:57 sync.json
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && ./tools/system-status --help | sed -n '1,200p' | grep -n -A4 -i 'queue' | head -40; echo "=== rc=$? ==="
97:queue-source verdicts (from queue-source-check --format json --repo):
98: ok 'one queue' exit 0 - CHANGELOG's newest '### Queue'
99- section is a POINTER at
100- agent-logs/PROGRESS.md and the 111 item
101- lines already written into the 25
102- historical sections are unchanged
--
107: two different queues) and this row is what
108- stops it happening silently again
109: warning 'cannot verify' exit 3 - no CHANGELOG.md, no '### Queue'
110- section, or no agent-logs/PROGRESS.md, or
111- the child could not run: never ok, never
112- an error on a tree that was not checked
113-
114:Test hooks (env), queue-source row:
115: GLADEX_QUEUE_SOURCE_BIN queue-source-check binary (default:
116- sibling of this script)
117: GLADEX_QUEUE_SOURCE_TIMEOUT seconds the child is given (default: 20)
118-
119-Exit codes:
120- 0 All systems healthy
121- 1 One or more systems unhealthy
=== rc=0 ===
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && python3 -c "
import json;d=json.load(open('/tmp/opencode/qsc/sync.json'))
print({k:d[k] for k in list(d)[:14] if k not in ('files',)})
print('nfiles',len(d.get('files',[])))
" 2>&1 | head -20
{'tool': 'source-sync-check', 'timestamp': '2026-09-26T03:57:33Z', 'repo_root': '/data/repo', 'env_roots': {'dev': '/opt/startup/dev', 'prod': '/opt/startup/prod'}, 'trees': [{'tree': 'src/php', 'repo_rel': 'app/src/php', 'env_rel': 'src/php'}, {'tree': 'examples/workflows', 'repo_rel': 'examples/workflows', 'env_rel': 'examples/workflows'}], 'summary': {'dev': {'files': 21, 'drift': 0}, 'prod': {'files': 21, 'drift': 0}}, 'missing_dirs': [], 'drift': False, 'exit_code': 0}
nfiles 42
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && tail -20 BUDGET.md
month: 2026-09
allowance: 5.00
spent: 1.50 (gladex.de — registered + DNS live since 2026-09-20, 1.50/mo)
remaining: 3.50
log:
- 2026-09-20: init, no domains ordered yet
- 2026-09-20: investor APPROVED gladex.de (1.50/mo). Awaiting registrar purchase + DNS creds.
- 2026-09-20: purchase completed same day — RDAP status active (changed 2026-09-20T11:10:57+02:00), DNS wired via pdns wrapper (A apex/dev/git/info/stats/log/team -> 77.90.15.49, MX 10 gladex.de), dig-verified; DOMAIN.md holds evidence.
- 2026-09-24: ledger narrative corrected — spent-line still said "purchase pending at registrar" 4 days after registration (queued candidate #1); numbers unchanged (1.50/3.50, no money moved), log history above kept verbatim.
rules: .de 1.50/mo, .com/.org/.net 2.50/mo. No rollover. All purchases through investor.
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && git log --oneline -3 && git remote -v
3aa35cd docs(progress): [0.4.66] post-commit re-read — repo-lint 176 files / entries 71 and system-status promote-gates OK at 65627ac both confirmed, predictions hit with no correction
65627ac feat(tools): [0.4.66] the runner could total a run but not say which suite moved, so every entry attributed its delta by hand — --save-baseline/--baseline record per-suite counts and diff them under a CLOSURE check that also proves the record agrees with itself (110 -> 193 assertions, 40/3313 -> 40/3396, 0 failed)
0b76208 docs(progress): [0.4.65] post-commit re-read — repo-lint 176 files / entries 70 and regression-run 40 suites / 3313 / 0 both confirmed
origin git://git.gladex.de/gladex.git (fetch)
origin git://git.gladex.de/gladex.git (push)
> build · mimo-v2.6-flash-free
STEP 0 done, the gate and suite are green, and docs are updated. Waiting on the full regression run to land before writing the run entry and committing.
**State so far:**
- `tools/queue-source-check` + `tests/test_queue_source.sh` → **107/107, 0 failed**
- `tools/system-status` → `queue-source [OK] one queue: [0.4.67] pointer-only, 111 item line(s) frozen across 26 section(s), 37 PROGRESS bullet(s)` — **ALL SYSTEMS HEALTHY**
- `repo-lint` pre-commit read → exit 0, `files_total 176`, `linted {bash 25, go 43, json 11, php 32, python 12}`, `changelog_version entries 71, unique 71, duplicates []` → prediction recorded in REGISTRY: **178 / bash 26 / python 13 / entries 72**
- `source-sync-check` → in sync, 42 files, exit 0
- REGISTRY §repo-lint Status → `41 / 3503 / 21 shell = 2312`; §regression-run Live/Status + shape histogram updated
exit=0
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Sat Sep 26 06:49:59 2026] 127.0.0.1:55900 Accepted [Sat Sep 26 06:49:59 2026] 127.0.0.1:55900 Closing [Sat Sep 26 06:49:59 2026] 127.0.0.1:55902 Accepted [Sat Sep 26 06:49:59 2026] 127.0.0.1:55902 Closing [Sat Sep 26 06:49:59 2026] 127.0.0.1:55914 Accepted [Sat Sep 26 06:49:59 2026] 127.0.0.1:55914 Closing [Sat Sep 26 06:49:59 2026] 127.0.0.1:55930 Accepted [Sat Sep 26 06:49:59 2026] 127.0.0.1:55930 Closing [Sat Sep 26 06:49:59 2026] 127.0.0.1:55938 Accepted [Sat Sep 26 06:49:59 2026] 127.0.0.1:55938 Closing [Sat Sep 26 06:54:08 2026] 127.0.0.1:46630 Accepted [Sat Sep 26 06:54:08 2026] 127.0.0.1:46630 Closing [Sat Sep 26 06:55:44 2026] 127.0.0.1:42080 Accepted [Sat Sep 26 06:55:44 2026] 127.0.0.1:42080 Closing [Sat Sep 26 06:55:44 2026] 127.0.0.1:42082 Accepted [Sat Sep 26 06:55:44 2026] 127.0.0.1:42082 Closing [Sat Sep 26 06:58:01 2026] 127.0.0.1:58740 Accepted [Sat Sep 26 06:58:01 2026] 127.0.0.1:58740 Closing [Sat Sep 26 06:58:01 2026] 127.0.0.1:58746 Accepted [Sat Sep 26 06:58:01 2026] 127.0.0.1:58746 Closing [Sat Sep 26 06:58:01 2026] 127.0.0.1:58756 Accepted [Sat Sep 26 06:58:01 2026] 127.0.0.1:58756 Closing [Sat Sep 26 06:58:01 2026] 127.0.0.1:58766 Accepted [Sat Sep 26 06:58:01 2026] 127.0.0.1:58766 Closing [Sat Sep 26 06:58:01 2026] 127.0.0.1:58782 Accepted [Sat Sep 26 06:58:01 2026] 127.0.0.1:58782 Closing [Sat Sep 26 06:58:01 2026] 127.0.0.1:58788 Accepted [Sat Sep 26 06:58:01 2026] 127.0.0.1:58788 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58796 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58796 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58800 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58800 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58814 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58814 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58822 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58822 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58836 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58836 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58848 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58848 Closing [Sat Sep 26 06:58:02 2026] 127.0.0.1:58856 Accepted [Sat Sep 26 06:58:02 2026] 127.0.0.1:58856 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58868 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58868 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58876 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58876 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58884 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58884 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58900 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58900 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58904 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58904 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58918 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58918 Closing [Sat Sep 26 06:58:03 2026] 127.0.0.1:58934 Accepted [Sat Sep 26 06:58:03 2026] 127.0.0.1:58934 Closing [Sat Sep 26 07:06:40 2026] 127.0.0.1:37322 Accepted [Sat Sep 26 07:06:40 2026] 127.0.0.1:37322 Closing [Sat Sep 26 07:08:34 2026] 127.0.0.1:60558 Accepted
Generated 2026-09-26 05:08:34 UTC · Gladex.de