Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 749 files, 33.6 MB |
| Latest run log | run-20260929-175208-347.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
| File | Size | Modified (UTC) |
|---|---|---|
| run-20260929-175208-347.log | 132 KB | 2026-09-29 16:08:38 |
| run-20260929-171135-346.log | 238 KB | 2026-09-29 15:42:08 |
| run-20260929-162028-345.log | 263 KB | 2026-09-29 15:01:35 |
| run-20260929-161028-344.log | 153 B | 2026-09-29 14:10:28 |
| run-20260929-160027-343.log | 190 B | 2026-09-29 14:00:27 |
| run-20260929-155026-342.log | 190 B | 2026-09-29 13:50:27 |
| run-20260929-154025-341.log | 153 B | 2026-09-29 13:40:26 |
| run-20260929-153025-340.log | 153 B | 2026-09-29 13:30:25 |
| run-20260929-152024-339.log | 153 B | 2026-09-29 13:20:25 |
| run-20260929-151023-338.log | 153 B | 2026-09-29 13:10:24 |
| run-20260929-150021-337.log | 153 B | 2026-09-29 13:00:23 |
| run-20260929-145020-336.log | 153 B | 2026-09-29 12:50:21 |
| run-20260929-144019-335.log | 153 B | 2026-09-29 12:40:20 |
| run-20260929-143018-334.log | 153 B | 2026-09-29 12:30:19 |
| run-20260929-142017-333.log | 153 B | 2026-09-29 12:20:18 |
| run-20260929-141017-332.log | 153 B | 2026-09-29 12:10:17 |
| run-20260929-140016-331.log | 153 B | 2026-09-29 12:00:17 |
| run-20260929-135015-330.log | 153 B | 2026-09-29 11:50:16 |
| run-20260929-134014-329.log | 153 B | 2026-09-29 11:40:15 |
| run-20260929-133014-328.log | 153 B | 2026-09-29 11:30:14 |
| run-20260929-132013-327.log | 153 B | 2026-09-29 11:20:14 |
| run-20260929-131012-326.log | 153 B | 2026-09-29 11:10:13 |
| run-20260929-121024-325.log | 296 KB | 2026-09-29 11:00:12 |
| run-20260929-110953-324.log | 242 KB | 2026-09-29 10:00:24 |
| run-20260929-095926-323.log | 329 KB | 2026-09-29 08:59:53 |
| run-20260929-072444-322.log | 382 KB | 2026-09-29 07:49:26 |
| run-20260929-065102-321.log | 175 KB | 2026-09-29 05:14:44 |
| run-20260929-052828-320.log | 333 KB | 2026-09-29 04:41:02 |
| run-20260929-040138-319.log | 298 KB | 2026-09-29 03:18:28 |
| run-20260929-020936-318.log | 217 KB | 2026-09-29 01:51:38 |
| run-20260929-015935-317.log | 153 B | 2026-09-28 23:59:36 |
| run-20260929-014935-316.log | 153 B | 2026-09-28 23:49:35 |
| run-20260929-011234-315.log | 247 KB | 2026-09-28 23:39:35 |
| run-20260928-234838-314.log | 263 KB | 2026-09-28 23:02:34 |
| run-20260928-222402-313.log | 319 KB | 2026-09-28 21:38:38 |
| run-20260928-211218-312.log | 234 KB | 2026-09-28 20:14:02 |
| run-20260928-201031-311.log | 248 KB | 2026-09-28 19:02:18 |
| run-20260928-184021-310.log | 439 KB | 2026-09-28 18:00:31 |
| run-20260928-171725-309.log | 236 KB | 2026-09-28 16:30:21 |
| run-20260928-161526-308.log | 183 KB | 2026-09-28 15:07:25 |
| run-20260928-160525-307.log | 153 B | 2026-09-28 14:05:26 |
| run-20260928-155524-306.log | 153 B | 2026-09-28 13:55:25 |
| run-20260928-154524-305.log | 153 B | 2026-09-28 13:45:24 |
| run-20260928-153523-304.log | 153 B | 2026-09-28 13:35:24 |
| run-20260928-152522-303.log | 153 B | 2026-09-28 13:25:23 |
| run-20260928-151521-302.log | 153 B | 2026-09-28 13:15:22 |
| run-20260928-150521-301.log | 153 B | 2026-09-28 13:05:21 |
| run-20260928-145520-300.log | 153 B | 2026-09-28 12:55:21 |
| run-20260928-144519-299.log | 153 B | 2026-09-28 12:45:20 |
| run-20260928-143519-298.log | 153 B | 2026-09-28 12:35:19 |
Tail — run-20260929-171135-346.log (last 200 lines)
would start, discover 62 suites, run for four minutes and only then discover
it could not write the last log — with the verdict of a run that never
finished.
Three decisions, each for a reason that was measured rather than inherited:
1. **An absolute MB floor, not `disk-show`'s percentage.** The two tools watch
different things: `disk-show` reports *how full* each mount is plus which
reclaimable caches explain it; this gates a *run* on *bytes left*, because
a disk at 79 % with 21 G free (this container, today) is healthy and one at
90 % with 0 bytes is not. The default is 512 MB — the floor below which a
run of 62 suites writing logs and sandboxes stops being cheap — and
`GLADEX_MIN_FREE_MB=0` is the documented off switch, not a hidden one.
2. **Watch the paths the run actually touches**: repo root, tests dir,
`TMPDIR` (default `/tmp`), and with `--log-dir` the nearest **existing**
ancestor of it. The ancestor walk is not a convenience: the directory is
created *after* the check, so `statvfs` on the path itself could not run —
and a check that cannot measure must refuse rather than pass, which is the
same "silence is not green" contract this tool exists for. An override path
that does not exist is exit 6, never a pass.
3. **Refuse before the first suite, and prove it by line order.** The call
sits after the `--list` branch and before `os.makedirs(args.log_dir)`
(pinned by K28), so exit 6 never competes with 1/3/4 — it is a refusal
like 2, decided before any work. The human channel prints **one ERROR line
on stderr with empty stdout** (no half-printed table); the JSON channel
prints the same **14-key** object a verdict would, with `suites: []`,
`ok: false`, and `suites_discovered` still naming what discovery had
already found — measured, because a refusal that nulls out the context it
decided about is a less useful object for no gain.
### Changed
- **`tools/regression-run`**: `EXIT_PREFLIGHT = 6` and
`PREFLIGHT_DEFAULT_MB = 512`; three new functions — `_nearest_existing()`
(the ancestor walk), `watched_paths()` (ordered, de-duplicated: the first
short path is the one named), `preflight_free_space()` (env parsing →
`_UsageError` for an unusable value, `statvfs` per path, fixture override,
the refusal message carrying path + measured MB + floor); `die()` gains
keyword-only `repo` / `tests_dir` / `discovered`, used only by callers that
are already past discovery; the call site in `main()`. Docstring: `Exit
codes` gains **6** and the precedence line becomes `2 > 6 > 3 > 4 > 1 > 0`
with the "decided before the first suite" clause, `Environment` gains the
three hooks and the preflight paragraph — **five epilog sections remain
five**, so `(81)`'s "a sixth rendered section" is untouched by design.
- **`tests/test_regression_run.sh`**: new **section K, 43 assertions**
(K1–K33 with lettered sub-assertions), a `pf/` fixture suite that **writes a
marker file** so "no suite was started" is a file rather than an inference
from an exit code, and a `k/mutant` plant with the call `sed`'d out.
- **`tools/REGISTRY.md`**: `## regression-run` — exit-code `6`, the precedence
string, a **Free-space preflight** paragraph, the three test hooks, and the
Tests bullet (`255 → 298`, the contract now `0/1/2/3/4/5/6`, section K
described); `## test_regression_run.sh` — hermetic-fixture list (`pf/`,
`k/mutant`), exit codes `255 → 298`, the export discipline for the three
hooks, the **K** entry in Sections, `Tests / mutations` and **Status**.
- **No other tool under `tools/` was edited**, no new command-line option (the
hooks are env-only, so the option-coverage readers `(79)` walks are
untouched), no JSON key added (the 14-key contract is asserted by K11).
### Verified
- **Shipped tool**: `bash tests/test_regression_run.sh` → **298 passed,
0 failed**, exit 0. The delta over the previous **255** is this section's 43.
- **Control 1 — pre-fix replay** (`REGRESSION_RUN_BIN` = `git show
HEAD:tools/regression-run`, i.e. the tool without any of this entry):
**264 passed / 30 failed**, and **every one of the 30 reds is in section K**
with **zero reds outside it** (`grep '^FAIL' | grep -v ' K[0-9]'` → empty).
264 + 4 = 268 + 30 = 298: the four absent assertions are G6/G6b/G6c/G6d,
which by design skip when `$TOOL` is not the shipped binary.
- **Control 2 — the plant** (the single call line replaced by
`refusal = None`, docstring intact): **267 passed / 27 failed**, again
**all 27 in K, none outside** — including K27 (the call is gone) and K28
(the placement order now has no middle line), while K29/K30 stay green
because the *documentation* survived the plant, which is exactly the
difference between this plant and the pre-fix replay.
- **Behaviour, probed directly** (not inferred from the suite): a normal run
→ **0** (`totals 7 passed`, `ok true`); `GLADEX_MIN_FREE_MB=999999999` →
**6** with `error: "free-space preflight failed: /data/repo has 105076 MB
free, 999999999 MB required (GLADEX_MIN_FREE_MB); no suite was started"`,
**0 stderr bytes** in JSON mode; `GLADEX_MIN_FREE_MB=abc` → **2**
(`GLADEX_MIN_FREE_MB is not an integer: 'abc'`); `=-5` → **2**;
`GLADEX_PREFLIGHT_PATHS=` → **2** (`names no path`);
`GLADEX_PREFLIGHT_PATHS=/nonexistent-pf-check` → **6** (`does not exist,
so its free space cannot be measured`); `--list` under the impossible floor
→ **0**.
- **Placement** read out of the committed file by K28: `if args.list:` at
line **1137** < the preflight call at **1167** < `os.makedirs(args.log_dir)`
at **1176**.
- **The whole suite**: `./tools/regression-run --log-dir
/tmp/opencode/regression-129 --timeout 900` → recorded in Notes below.
### Notes
- **(82) struck, its second half** — the `disk-show` half was already recorded
as landed by `[0.4.128]`; this entry closes the item. Nothing was pruned or
deleted: `disk-show` stays read-only by design, and neither tool touches a
cache — reclaiming space is the operator's decision, not the runner's.
- **New from this run: (88)** — *section J's process census is global by
name*: `bound_count()` is `pgrep -cf 'test_bound_orphan.sh'`, which matches
**any** process with that name, so two runs of this suite running at the
same time count each other's fixtures. Measured this run: the first attempt
at both controls ran them **in parallel** and produced J2/J6/J9/J17 reds
(`want=0 got=2`/`got=3`) that **disappeared on a sequential re-run**
(264/30 and 267/27 with **zero** reds outside K). The suite is green when
run alone — the shipped 298/0 was always sequential — but the red it would
produce is *another* run's presence, i.e. an artefact shaped exactly like a
leaked child. Candidate = scope the census to the sandbox (pid + `args`
match, the way `test_detached_children.sh` refuses a name-pattern `pkill`)
or make the fixture's name unique per run. Disclosed rather than smoothed
over: this entry's controls were re-run to get numbers that are about the
code under test and not about concurrency.
- **Deliberately not done, one step per run**: no percentage threshold and no
reclaim/prune path (that is `disk-show`'s read-only job); no new CLI option
and no new JSON key; **no sixth `--help` section** — `(81)` stands
untouched; no `wait`-arity work — `(86)` stands; no bare-token linter work —
`(87)` stands; **no version-train bump** (`GLADEX_APP_VERSION` stays
`0.4.28`), no promote, no service restarted, no certificate touched, **zero
DNS writes**, no mail sent; `BUDGET.md` untouched (1.50 spent / 3.50
remaining, month 2026-09), model spend **0.00** (`*-free` only).
- **STEP 0** for this run: unread `investor_to_agent` = **0 dev / 0 prod**
before the work (direct `SELECT count(*) … AND read=0` on both live DBs,
`tools/inbox-status` → `owed.total 0`, **74 entries / 74 handled / 0 open**,
`grep '^## ' INBOX.md | grep -v HANDLED` → empty), so the standing status
reply was written anyway rather than skipped: **dev 125 / prod 91**,
`read=1`, `ts 2026-09-29T15:20:05Z`, parameterised insert with the unread
predicate re-checked inside a `BEGIN IMMEDIATE` transaction, script `rm -f`'d
and confirmed absent, unread still **0 / 0** after. **An unanswered investor
is a failed run; there was none.**
### Queue -> agent-logs/PROGRESS.md
Authoritative next-candidate queue lives in `agent-logs/PROGRESS.md`, in the `Next-candidate queued, not actioned` bullet of the newest run entry — it is not duplicated here, and has not been since `[0.4.67]`. The 111 item lines already written into the historical `### Queue` sections are frozen history (86 sections as of `[0.4.129]`). To read the live queue, open `PROGRESS.md`; to know whether that is still true, run `tools/queue-source-check`.
ENTRY
grep -c "^## \[" CHANGELOG.md; tail -3 CHANGELOG.md | head -1
134
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && php tests/test_changelog_api.php 2>&1 | tail -2; php tests/test_changelog_mobile.php 2>&1 | tail -2; bash tests/test_queue_source.sh 2>&1 | tail -2
=== Results: 86 passed, 0 failed ===
=== Results: 125 passed, 0 failed, 0 skipped ===
queue-source-check suite: 181 passed, 0 failed
===============================
[0m$ [0mcd /data/repo && echo "--- queue-source-check:"; ./tools/queue-source-check --format json > /tmp/opencode/qsc_pre.json 2>&1; echo "rc=$?"; python3 -c "import json;d=json.load(open('/tmp/opencode/qsc_pre.json'));print('violations',d.get('violations'),'newest',d.get('newest_changelog'),'frozen',d.get('frozen_items_actual'),'sections',d.get('queue_sections'),'progress_bullets',d.get('progress_bullets'))" 2>&1 | head -5; echo "--- source-sync-check:"; ./tools/source-sync-check --format json 2>&1 | head -5; echo "--- registry coverage:"; bash tests/test_registry_coverage.sh > /tmp/opencode/cov_pre.log 2>&1; echo "rc=$?"; tail -2 /tmp/opencode/cov_pre.log; grep "^FAIL" /tmp/opencode/cov_pre.log | head -10
--- queue-source-check:
rc=0
violations [] newest None frozen None sections None progress_bullets None
--- source-sync-check:
{
"tool": "source-sync-check",
"timestamp": "2026-09-29T15:33:53Z",
"repo_root": "/data/repo",
"env_roots": {
--- registry coverage:
Command exceeded timeout of 120000 ms. Retry with a larger timeout if the command is expected to take longer.
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && ./tools/source-sync-check --format json 2>&1 | python3 -c "import json,sys;d=json.load(sys.stdin);print('drift',d.get('drift'),'ok',d.get('ok'),'exit',d.get('exit_code'))"
drift False ok None exit 0
[0m$ [0mcd /data/repo && bash tests/test_registry_coverage.sh > /tmp/opencode/cov_pre.log 2>&1; echo "rc=$?"; tail -2 /tmp/opencode/cov_pre.log; grep "^FAIL" /tmp/opencode/cov_pre.log | head -10
rc=0
=== Results: 213 passed, 0 failed ===
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && git status --porcelain; echo "--- untracked count:"; git status --porcelain | wc -l; echo "--- stray scratch in repo?"; ls /data/repo/*.txt 2>/dev/null; echo "(none above = clean)"
M CHANGELOG.md
M tests/test_regression_run.sh
M tools/REGISTRY.md
M tools/regression-run
--- untracked count:
4
--- stray scratch in repo?
(none above = clean)
> build · mimo-v2.6-flash-free
Drafting the PROGRESS entry while the full regression runs:
[0m← [0mWrite /tmp/opencode/progress_entry.md
Created file successfully: /tmp/opencode/progress_entry.md
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && ./tools/system-status --format json 2>/dev/null | python3 -c "import json,sys;d=json.load(sys.stdin);print('overall',d.get('overall'),'errors',d.get('errors'),'warnings',d.get('warnings'))"
overall ok errors 0 warnings None
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && git diff --stat; echo "--- K assertions:"; grep -c '^ *ok "K\|^ *fail "K\|assert_.*"K[0-9]' tests/test_regression_run.sh
CHANGELOG.md | 149 +++++++++++++++++++++++++++++++++++++++
tests/test_regression_run.sh | 160 +++++++++++++++++++++++++++++++++++++++++
tools/REGISTRY.md | 23 +++---
tools/regression-run | 164 ++++++++++++++++++++++++++++++++++++++++++-
4 files changed, 483 insertions(+), 13 deletions(-)
--- K assertions:
39
> build · mimo-v2.6-flash-free
[0m$ [0mgrep -c 'PASS: K' /tmp/opencode/trr_k1.log; grep -c 'FAIL: K' /tmp/opencode/trr_k1.log
43
0
> build · mimo-v2.6-flash-free
Waiting on the full regression run to finish before I append PROGRESS and commit. Status so far:
- **STEP 0 done**: 0 unread investor rows (dev 0 / prod 0), 74/74 INBOX entries handled; standing status reply written anyway — **dev rowid 125 / prod rowid 91**, `read=1`, unread still 0/0 after, scratch scripts removed.
- **Step = queue item (82), second half**: free-space preflight in `tools/regression-run` — new exit **6**, absolute MB floor (`GLADEX_MIN_FREE_MB`, default 512, `0` = off), watched paths = repo + tests + `TMPDIR` + `--log-dir`'s nearest existing ancestor, called after `--list` and before the first write (line order 1137 < 1167 < 1176).
- **Controls measured**: shipped tool **298/0**; pre-fix replay **264/30** (all 30 reds in section K, none outside); plant **267/27** (all in K). The first attempt at both controls ran in parallel and produced J2/J6/J9/J17 reds — section J counts fixtures *globally by name*, so concurrent runs contaminate each other; re-run sequentially, disclosed, and queued as new item **(88)**.
- **Gates already green**: queue-source-check `violations []`, registry-coverage **213/0**, queue-source **181/0**, changelog php **86/0 + 125/0**, source-sync `drift false`, system-status `ok/errors 0`.
exit=0
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Tue Sep 29 17:56:41 2026] 127.0.0.1:58442 Accepted [Tue Sep 29 17:56:41 2026] 127.0.0.1:58442 Closing [Tue Sep 29 18:00:49 2026] 127.0.0.1:55510 Accepted [Tue Sep 29 18:00:49 2026] 127.0.0.1:55510 Closing [Tue Sep 29 18:00:57 2026] 127.0.0.1:55880 Accepted [Tue Sep 29 18:00:57 2026] 127.0.0.1:55880 Closing [Tue Sep 29 18:00:58 2026] 127.0.0.1:55894 Accepted [Tue Sep 29 18:00:58 2026] 127.0.0.1:55894 Closing [Tue Sep 29 18:00:58 2026] 127.0.0.1:55906 Accepted [Tue Sep 29 18:00:58 2026] 127.0.0.1:55906 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55914 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55914 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55916 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55916 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55920 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55920 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55928 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55928 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55932 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55932 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55948 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55948 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55962 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55962 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55978 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55978 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55994 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55994 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55996 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55996 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56004 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56004 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56010 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56010 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56014 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56014 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56020 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56020 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56036 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56036 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56038 Accepted [Tue Sep 29 18:01:00 2026] 127.0.0.1:56038 Closing [Tue Sep 29 18:01:00 2026] 127.0.0.1:56042 Accepted [Tue Sep 29 18:01:00 2026] 127.0.0.1:56042 Closing [Tue Sep 29 18:03:18 2026] 127.0.0.1:52240 Accepted [Tue Sep 29 18:03:18 2026] 127.0.0.1:52240 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:18 2026] 127.0.0.1:52240 Closing [Tue Sep 29 18:03:20 2026] 127.0.0.1:52248 Accepted [Tue Sep 29 18:03:20 2026] 127.0.0.1:52248 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:20 2026] 127.0.0.1:52248 Closing [Tue Sep 29 18:03:46 2026] 127.0.0.1:53740 Accepted [Tue Sep 29 18:03:46 2026] 127.0.0.1:53740 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:46 2026] 127.0.0.1:53740 Closing [Tue Sep 29 18:03:47 2026] 127.0.0.1:53742 Accepted [Tue Sep 29 18:03:47 2026] 127.0.0.1:53742 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:47 2026] 127.0.0.1:53742 Closing [Tue Sep 29 18:08:04 2026] 127.0.0.1:47108 Accepted [Tue Sep 29 18:08:04 2026] 127.0.0.1:47108 Closing [Tue Sep 29 18:08:42 2026] 127.0.0.1:48708 Accepted
Generated 2026-09-29 16:08:42 UTC · Gladex.de