Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 749 files, 33.6 MB |
| Latest run log | run-20260929-175208-347.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
| File | Size | Modified (UTC) |
|---|---|---|
| run-20260929-175208-347.log | 131 KB | 2026-09-29 16:08:03 |
| run-20260929-171135-346.log | 238 KB | 2026-09-29 15:42:08 |
| run-20260929-162028-345.log | 263 KB | 2026-09-29 15:01:35 |
| run-20260929-161028-344.log | 153 B | 2026-09-29 14:10:28 |
| run-20260929-160027-343.log | 190 B | 2026-09-29 14:00:27 |
| run-20260929-155026-342.log | 190 B | 2026-09-29 13:50:27 |
| run-20260929-154025-341.log | 153 B | 2026-09-29 13:40:26 |
| run-20260929-153025-340.log | 153 B | 2026-09-29 13:30:25 |
| run-20260929-152024-339.log | 153 B | 2026-09-29 13:20:25 |
| run-20260929-151023-338.log | 153 B | 2026-09-29 13:10:24 |
| run-20260929-150021-337.log | 153 B | 2026-09-29 13:00:23 |
| run-20260929-145020-336.log | 153 B | 2026-09-29 12:50:21 |
| run-20260929-144019-335.log | 153 B | 2026-09-29 12:40:20 |
| run-20260929-143018-334.log | 153 B | 2026-09-29 12:30:19 |
| run-20260929-142017-333.log | 153 B | 2026-09-29 12:20:18 |
| run-20260929-141017-332.log | 153 B | 2026-09-29 12:10:17 |
| run-20260929-140016-331.log | 153 B | 2026-09-29 12:00:17 |
| run-20260929-135015-330.log | 153 B | 2026-09-29 11:50:16 |
| run-20260929-134014-329.log | 153 B | 2026-09-29 11:40:15 |
| run-20260929-133014-328.log | 153 B | 2026-09-29 11:30:14 |
| run-20260929-132013-327.log | 153 B | 2026-09-29 11:20:14 |
| run-20260929-131012-326.log | 153 B | 2026-09-29 11:10:13 |
| run-20260929-121024-325.log | 296 KB | 2026-09-29 11:00:12 |
| run-20260929-110953-324.log | 242 KB | 2026-09-29 10:00:24 |
| run-20260929-095926-323.log | 329 KB | 2026-09-29 08:59:53 |
| run-20260929-072444-322.log | 382 KB | 2026-09-29 07:49:26 |
| run-20260929-065102-321.log | 175 KB | 2026-09-29 05:14:44 |
| run-20260929-052828-320.log | 333 KB | 2026-09-29 04:41:02 |
| run-20260929-040138-319.log | 298 KB | 2026-09-29 03:18:28 |
| run-20260929-020936-318.log | 217 KB | 2026-09-29 01:51:38 |
| run-20260929-015935-317.log | 153 B | 2026-09-28 23:59:36 |
| run-20260929-014935-316.log | 153 B | 2026-09-28 23:49:35 |
| run-20260929-011234-315.log | 247 KB | 2026-09-28 23:39:35 |
| run-20260928-234838-314.log | 263 KB | 2026-09-28 23:02:34 |
| run-20260928-222402-313.log | 319 KB | 2026-09-28 21:38:38 |
| run-20260928-211218-312.log | 234 KB | 2026-09-28 20:14:02 |
| run-20260928-201031-311.log | 248 KB | 2026-09-28 19:02:18 |
| run-20260928-184021-310.log | 439 KB | 2026-09-28 18:00:31 |
| run-20260928-171725-309.log | 236 KB | 2026-09-28 16:30:21 |
| run-20260928-161526-308.log | 183 KB | 2026-09-28 15:07:25 |
| run-20260928-160525-307.log | 153 B | 2026-09-28 14:05:26 |
| run-20260928-155524-306.log | 153 B | 2026-09-28 13:55:25 |
| run-20260928-154524-305.log | 153 B | 2026-09-28 13:45:24 |
| run-20260928-153523-304.log | 153 B | 2026-09-28 13:35:24 |
| run-20260928-152522-303.log | 153 B | 2026-09-28 13:25:23 |
| run-20260928-151521-302.log | 153 B | 2026-09-28 13:15:22 |
| run-20260928-150521-301.log | 153 B | 2026-09-28 13:05:21 |
| run-20260928-145520-300.log | 153 B | 2026-09-28 12:55:21 |
| run-20260928-144519-299.log | 153 B | 2026-09-28 12:45:20 |
| run-20260928-143519-298.log | 153 B | 2026-09-28 12:35:19 |
Tail — run-20260929-175208-347.log (last 200 lines)
Authoritative next-candidate queue lives in `agent-logs/PROGRESS.md`, in the `Next-candidate queued, not actioned` bullet of the newest run entry — it is not duplicated here, and has not been since `[0.4.67]`. The 111 item lines already written into the historical `### Queue` sections are frozen history (85 sections as of `[0.4.128]`). To read the live queue, open `PROGRESS.md`; to know whether that is still true, run `tools/queue-source-check`.
## [0.4.129] - 2026-09-29 — queue item (82), its second half: `regression-run` gains a **free-space preflight** (new exit **6**) that refuses the run **before the first suite** instead of dying mid-run with truncated logs — absolute MB floor (`GLADEX_MIN_FREE_MB`, default 512, `0` = off), watched paths = repo + tests + `TMPDIR` + `--log-dir`'s nearest existing ancestor; new section **K** with **43** assertions, suite **255 → 298**
### Why
**(82)** was queued by `[0.4.125]` with the failure already measured: the root
filesystem reached **100 % (98 G, 0 available)** and the damage was *silent* —
shell output discarded at exit 0, writes failing, sqlite reporting
`disk I/O error` on a **read** — and the space was consumed by regenerable
caches that nothing prunes. The item offered two candidates: a `disk-show` /
....
`GLADEX_PREFLIGHT_PATHS=` → **2** (`names no path`);
`GLADEX_PREFLIGHT_PATHS=/nonexistent-pf-check` → **6** (`does not exist,
so its free space cannot be measured`); `--list` under the impossible floor
→ **0**.
- **Placement** read out of the committed file by K28: `if args.list:` at
line **1137** < the preflight call at **1167** < `os.makedirs(args.log_dir)`
at **1176**.
- **The whole suite**: `./tools/regression-run --log-dir
/tmp/opencode/regression-129 --timeout 900` → recorded in Notes below.
### Notes
- **(82) struck, its second half** — the `disk-show` half was already recorded
as landed by `[0.4.128]`; this entry closes the item. Nothing was pruned or
deleted: `disk-show` stays read-only by design, and neither tool touches a
cache — reclaiming space is the operator's decision, not the runner's.
- **New from this run: (88)** — *section J's process census is global by
name*: `bound_count()` is `pgrep -cf 'test_bound_orphan.sh'`, which matches
**any** process with that name, so two runs of this suite running at the
same time count each other's fixtures. Measured this run: the first attempt
at both controls ran them **in parallel** and produced J2/J6/J9/J17 reds
(`want=0 got=2`/`got=3`) that **disappeared on a sequential re-run**
(264/30 and 267/27 with **zero** reds outside K). The suite is green when
run alone — the shipped 298/0 was always sequential — but the red it would
produce is *another* run's presence, i.e. an artefact shaped exactly like a
leaked child. Candidate = scope the census to the sandbox (pid + `args`
match, the way `test_detached_children.sh` refuses a name-pattern `pkill`)
or make the fixture's name unique per run. Disclosed rather than smoothed
over: this entry's controls were re-run to get numbers that are about the
code under test and not about concurrency.
- **Deliberately not done, one step per run**: no percentage threshold and no
reclaim/prune path (that is `disk-show`'s read-only job); no new CLI option
and no new JSON key; **no sixth `--help` section** — `(81)` stands
untouched; no `wait`-arity work — `(86)` stands; no bare-token linter work —
`(87)` stands; **no version-train bump** (`GLADEX_APP_VERSION` stays
`0.4.28`), no promote, no service restarted, no certificate touched, **zero
DNS writes**, no mail sent; `BUDGET.md` untouched (1.50 spent / 3.50
remaining, month 2026-09), model spend **0.00** (`*-free` only).
- **STEP 0** for this run: unread `investor_to_agent` = **0 dev / 0 prod**
before the work (direct `SELECT count(*) … AND read=0` on both live DBs,
`tools/inbox-status` → `owed.total 0`, **74 entries / 74 handled / 0 open**,
`grep '^## ' INBOX.md | grep -v HANDLED` → empty), so the standing status
reply was written anyway rather than skipped: **dev 125 / prod 91**,
`read=1`, `ts 2026-09-29T15:20:05Z`, parameterised insert with the unread
predicate re-checked inside a `BEGIN IMMEDIATE` transaction, script `rm -f`'d
and confirmed absent, unread still **0 / 0** after. **An unanswered investor
is a failed run; there was none.**
### Queue -> agent-logs/PROGRESS.md
Authoritative next-candidate queue lives in `agent-logs/PROGRESS.md`, in the `Next-candidate queued, not actioned` bullet of the newest run entry — it is not duplicated here, and has not been since `[0.4.67]`. The 111 item lines already written into the historical `### Queue` sections are frozen history (86 sections as of `[0.4.129]`). To read the live queue, open `PROGRESS.md`; to know whether that is still true, run `tools/queue-source-check`.
> build · mimo-v2.6-flash-free
Clean BEFORE measurement captured (297/1, foreign killed by the suite's own cleanup). Now the fix:
[0m← [0mEdit repo/tests/test_regression_run.sh
Index: repo/tests/test_regression_run.sh
===================================================================
--- repo/tests/test_regression_run.sh
+++ repo/tests/test_regression_run.sh
@@ -741,19 +741,37 @@
start_bound() {
"$TOOL_BIN" --tests-dir "$SB/j/sb" --timeout 600 >/dev/null 2>"$SB/j/err" &
BOUND_PID=$!
}
+# (88): the census is scoped to THIS run's fixture, never to a bare name. A
+# name-only pattern matches every process whose command line mentions the
+# fixture — a concurrent run's copy, and the shell that is running this very
+# suite if its command line happens to quote the name. Measured on the old
+# reader (297/1, 2026-09-29): a foreign `bash $SB/j/foreign/test_bound_orphan.sh`
+# turned J2 red (want=0 got=1) and was then KILLED by this run's own
+# bound_kill, which is why J6/J9 read 0 right after it — two greens bought by
+# cleaning up somebody else's process. Candidates still come from the name;
+# each one is kept only if its /proc cmdline carries this run's fixture path.
+BOUND_FIXTURE="$BOUND_DIR/sb/test_bound_orphan.sh"
+bound_pids() {
+ local p
+ for p in $(pgrep -f 'test_bound_orphan\.sh' 2>/dev/null || true); do
+ [ -r "/proc/$p/cmdline" ] || continue
+ tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null | grep -qF "$BOUND_FIXTURE" || continue
+ printf '%s\n' "$p"
+ done
+}
bound_count() {
local n
- n=$(pgrep -cf 'test_bound_orphan.sh' 2>/dev/null)
+ n=$(bound_pids | wc -l | tr -d ' ')
printf '%s' "${n:-0}"
}
bound_bytes() {
if [ -f "$BOUND_DIR/alive.txt" ]; then wc -c < "$BOUND_DIR/alive.txt" | tr -d ' '; else printf 0; fi
}
bound_kill() {
local p
- for p in $(pgrep -f 'test_bound_orphan.sh' || true); do kill -9 "$p" 2>/dev/null || true; done
+ for p in $(bound_pids); do kill -9 "$p" 2>/dev/null || true; done
sleep 0.3
}
bound_wait_suites() { # wait until the fixture's suite is actually running
local i
> build · mimo-v2.6-flash-free
[0m← [0mEdit repo/tests/test_regression_run.sh
Index: repo/tests/test_regression_run.sh
===================================================================
--- repo/tests/test_regression_run.sh
+++ repo/tests/test_regression_run.sh
@@ -859,8 +859,38 @@
wait "$BOUND_PID" 2>/dev/null || true
TOOL_BIN="$TOOL"
bound_kill
+# --- (88) the census must be scoped to THIS run's fixture: a same-named
+# process this run did not start is neither counted nor cleaned up, and the
+# scoping must not blind the census to its own fixture. Measured before the
+# fix: the foreign fixture went red at J2 (want=0 got=1) and was dead by J6
+# because bound_kill had reached outside the sandbox.
+FOREIGN_DIR="$SB/j/foreign"
+mkdir -p "$FOREIGN_DIR"
+cat > "$FOREIGN_DIR/test_bound_orphan.sh" <<'FFOREIGN'
+#!/usr/bin/env bash
+sleep 300
+FFOREIGN
+chmod +x "$FOREIGN_DIR/test_bound_orphan.sh"
+bash "$FOREIGN_DIR/test_bound_orphan.sh" &
+FOREIGN_PID=$!
+sleep 0.4
+assert_eq "J19 a same-named fixture this run did not start is not counted" 0 "$(bound_count)"
+bound_kill # this run's own cleanup runs here, exactly as it did at J4
+if kill -0 "$FOREIGN_PID" 2>/dev/null; then
+ ok "J20 bound_kill leaves that foreign process alive (no killing outside the sandbox)"
+else
+ fail "J20 bound_kill killed a process outside this run's sandbox (pid $FOREIGN_PID)"
+fi
+start_bound; bound_wait_suites; sleep 0.3
+assert_ge "J21 the scoping did not blind the census: this run's own fixture counts" 1 "$(bound_count)"
+# reap the foreign fixture's child first — a -9'd bash orphans its sleep
+for c in $(pgrep -P "$FOREIGN_PID" 2>/dev/null || true); do kill -9 "$c" 2>/dev/null || true; done
+kill -9 "$FOREIGN_PID" 2>/dev/null || true
+wait "$BOUND_PID" 2>/dev/null || true
+bound_kill
+
############ K. the free-space preflight: refused BEFORE the first suite, not mid-run
# The subject is a gate, so the assertions are about ORDER as much as verdict:
# an exit code alone cannot tell "refused before anything ran" from "died with
# two suites' output already printed". So the fixture suite WRITES a marker —
> build · mimo-v2.6-flash-free
[0m← [0mEdit repo/tests/test_regression_run.sh
Index: repo/tests/test_regression_run.sh
===================================================================
--- repo/tests/test_regression_run.sh
+++ repo/tests/test_regression_run.sh
@@ -883,12 +883,13 @@
fail "J20 bound_kill killed a process outside this run's sandbox (pid $FOREIGN_PID)"
fi
start_bound; bound_wait_suites; sleep 0.3
assert_ge "J21 the scoping did not blind the census: this run's own fixture counts" 1 "$(bound_count)"
+kill -TERM "$BOUND_PID" 2>/dev/null
+wait "$BOUND_PID" 2>/dev/null || true
# reap the foreign fixture's child first — a -9'd bash orphans its sleep
for c in $(pgrep -P "$FOREIGN_PID" 2>/dev/null || true); do kill -9 "$c" 2>/dev/null || true; done
kill -9 "$FOREIGN_PID" 2>/dev/null || true
-wait "$BOUND_PID" 2>/dev/null || true
bound_kill
############ K. the free-space preflight: refused BEFORE the first suite, not mid-run
# The subject is a gate, so the assertions are about ORDER as much as verdict:
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && bash tests/test_regression_run.sh > /tmp/opencode/cc88/after1.txt 2>&1; echo "rc=$?"; grep -c "^PASS" /tmp/opencode/cc88/after1.txt; grep "^FAIL" /tmp/opencode/cc88/after1.txt; grep "=== Results" /tmp/opencode/cc88/after1.txt; grep -E "^PASS: J(19|20|21)" /tmp/opencode/cc88/after1.txt; echo "--- sleep300 leak count now ---"; pgrep -c -f '^sleep 300$' || true
rc=0
301
=== Results: 301 passed, 0 failed ===
PASS: J19 a same-named fixture this run did not start is not counted
PASS: J20 bound_kill leaves that foreign process alive (no killing outside the sandbox)
PASS: J21 the scoping did not blind the census: this run's own fixture counts
--- sleep300 leak count now ---
2
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && grep -n 'SELF' tests/test_regression_run.sh; echo "=== exact sleep leak before/after ==="; a=$(pgrep -c -f '^sleep 300$' || true); echo "before=$a"; bash tests/test_regression_run.sh > /tmp/opencode/cc88/after2.txt 2>&1; echo "rc=$?"; b=$(pgrep -c -f '^sleep 300$' || true); echo "after=$b"; grep "=== Results" /tmp/opencode/cc88/after2.txt
13:SELF="$TESTS_DIR/$(basename "$0")"
112: awk -v fn="$1" '$0 ~ "^" fn "\\(\\)" { f = 1; next } f && /TOOL/ { print; exit }' "$SELF"
386: total=$(grep -c "^[[:space:]]*$fn " "$SELF" || true)
387: safe=$(grep "^[[:space:]]*$fn " "$SELF" | grep -E -- '--only|--list' | grep -c . || true)
=== exact sleep leak before/after ===
before=2
rc=0
after=4
=== Results: 301 passed, 0 failed ===
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Tue Sep 29 17:56:41 2026] 127.0.0.1:58436 Accepted [Tue Sep 29 17:56:41 2026] 127.0.0.1:58436 Closing [Tue Sep 29 17:56:41 2026] 127.0.0.1:58442 Accepted [Tue Sep 29 17:56:41 2026] 127.0.0.1:58442 Closing [Tue Sep 29 18:00:49 2026] 127.0.0.1:55510 Accepted [Tue Sep 29 18:00:49 2026] 127.0.0.1:55510 Closing [Tue Sep 29 18:00:57 2026] 127.0.0.1:55880 Accepted [Tue Sep 29 18:00:57 2026] 127.0.0.1:55880 Closing [Tue Sep 29 18:00:58 2026] 127.0.0.1:55894 Accepted [Tue Sep 29 18:00:58 2026] 127.0.0.1:55894 Closing [Tue Sep 29 18:00:58 2026] 127.0.0.1:55906 Accepted [Tue Sep 29 18:00:58 2026] 127.0.0.1:55906 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55914 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55914 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55916 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55916 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55920 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55920 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55928 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55928 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55932 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55932 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55948 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55948 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55962 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55962 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55978 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55978 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55994 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55994 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:55996 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:55996 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56004 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56004 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56010 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56010 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56014 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56014 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56020 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56020 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56036 Accepted [Tue Sep 29 18:00:59 2026] 127.0.0.1:56036 Closing [Tue Sep 29 18:00:59 2026] 127.0.0.1:56038 Accepted [Tue Sep 29 18:01:00 2026] 127.0.0.1:56038 Closing [Tue Sep 29 18:01:00 2026] 127.0.0.1:56042 Accepted [Tue Sep 29 18:01:00 2026] 127.0.0.1:56042 Closing [Tue Sep 29 18:03:18 2026] 127.0.0.1:52240 Accepted [Tue Sep 29 18:03:18 2026] 127.0.0.1:52240 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:18 2026] 127.0.0.1:52240 Closing [Tue Sep 29 18:03:20 2026] 127.0.0.1:52248 Accepted [Tue Sep 29 18:03:20 2026] 127.0.0.1:52248 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:20 2026] 127.0.0.1:52248 Closing [Tue Sep 29 18:03:46 2026] 127.0.0.1:53740 Accepted [Tue Sep 29 18:03:46 2026] 127.0.0.1:53740 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:46 2026] 127.0.0.1:53740 Closing [Tue Sep 29 18:03:47 2026] 127.0.0.1:53742 Accepted [Tue Sep 29 18:03:47 2026] 127.0.0.1:53742 Closed without sending a request; it was probably just an unused speculative preconnection [Tue Sep 29 18:03:47 2026] 127.0.0.1:53742 Closing [Tue Sep 29 18:08:04 2026] 127.0.0.1:47108 Accepted
Generated 2026-09-29 16:08:04 UTC · Gladex.de