Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 3152 files, 135.9 MB |
| Latest run log | run-20261030-012537-2686.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
Tail — run-20261030-012537-2686.log (last 200 lines)
ActiveState=active
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && wc -l tools/system-status && grep -n 'agent-loop-exit\|loop-outcomes\|in-flight\|lo_prev\|newest\|exit=' tools/system-status | head -80
3922 tools/system-status
103: echo " rc=<ExecMainStatus> shift_exit=<marker>', where"
107: echo " '===== shift <date -Is ts> model=<model> exit=N ====='"
111: echo " is a shift that went red, 'shift_exit=?' a unit"
160: echo "loop-outcomes verdicts (the agent loop's own completion markers, queue item (4)):"
161: echo " ok 'run <n> exit=0, streak=0, ...' - the newest completed"
162: echo " run in the window exited 0. Only a measured exit=0 is"
163: echo " ever claimed: a rate-limited run ends exit=1 and can"
171: echo " 'agent-loop-exit=<rc> run=<id>' when <id> is this log's"
173: echo " loop's own 'exit=<rc>' carrying the SAME rc - the exact"
177: echo " printed mid-run) reads in-flight, never completed, or"
178: echo " the legacy 'exit=<rc>' on logs BEHIND the newest - so"
179: echo " an in-flight transcript ending in an ECHOED exit=N (the"
180: echo " 2026-10-26 run-2185 defect) reads in-flight, never a"
181: echo " completed red; the newest log's unattested legacy"
182: echo " exit= reads in-flight while fresh, no-marker once stale"
185: echo " not a dead loop); or 'stale: newest run log ... old' -"
199: echo " Test hooks (env), loop-outcomes row:"
204: echo " GLADEX_RUNLOG_WINDOW newest N run logs read (default 40)"
207: echo " GLADEX_RUNLOG_STALE seconds after which the newest log's age marks an"
308: echo " and the refusal is DATED from the newest"
311: echo " newest), rendered in minutes below a day and as"
393: echo " ok 'one queue' exit 0 - CHANGELOG's newest '### Queue'"
475: echo " the newest log says 'OK: No drift' and is younger"
482: echo " warning 'stale: newest log <age> old > <n>s bound ...'"
680:# the `===== shift ... exit=N =====` marker the wrapper wrote to
682:# normal (r1) shape); `shift_exit=?` = the unit died before its wrapper could
727:# anyone read its numstat by hand: lena's 2026-10-13 18:06 shift went exit=1
901: # `exit=([0-9]+)\ =====` matching 1,380 lines across the seven
909: # `^===== shift <ts> model=<model> exit=<n> =====$`, nothing
911: # (`===== shift %s model=%s exit=%s =====\n`), and all seven
932: marker_re='^===== shift ([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}([+-][0-9]{2}:[0-9]{2}|Z)?) model=[^[:space:]]+ exit=([0-9]+) =====$'
946: piece="$piece shift_exit=?"
948: piece="$piece shift_exit=$marker"
955: # keeps matching: `rc=… shift_exit=…` identifies the unit, `work=…`
1088:# loop-outcomes — the freshness guard on the AGENT LOOP itself (queue item
1091:# exceeded` / `exit=1` — six hours with zero commits, zero queue updates, zero
1098:# WHAT IS READ: the newest GLADEX_RUNLOG_WINDOW (default 40) `run-*.log` files
1103:# `agent-loop-exit=<rc> run=<id>` the loop's own attestation, written
1104:# since 2026-10-26 as the TRUE last line (after the legacy `exit=`) by
1110:# the loop's own `exit=<rc>` with the SAME rc. That pair is the whole
1111:# write the wrapper performs (`rc=$?; echo "exit=$rc" >> log; echo
1112:# "agent-loop-exit=$rc run=$n" >> log`), back to back with no writer in
1115:# 1014's TEST B: appending a bare `agent-loop-exit=0 run=<id>` to a copy
1116:# of an in-flight log read `run <id> exit=0, streak=0 … 0 in-flight` — a
1131:# `exit=<rc>` the legacy marker, the only shape the 2,652 historical logs
1132:# carry, trusted only on logs BEHIND the newest: the loop starts the next
1133:# log only after writing the previous one's `exit=`, so a log behind the
1134:# newest has certainly been closed by the loop — while a fresh newest log
1147:# to take ANY `exit=<digits>` last line as completed, and run 2185 — still in
1149:# `exit=1`: the first system-status of run 1009 read `streak=1 consecutive
1151:# the same read was `run 2184 exit=0, streak=0 … 1 in-flight` — a transient
1156:# lands on the NEXT loop incarnation, never mid-run), a newest log with a bare
1157:# `exit=` reads as in-flight while fresh and no-marker once stale — one run's
1161:# THE IN-FLIGHT RUN IS EXCLUDED, never counted: the newest log has no TRUSTED
1169:# VERDICTS (streak = consecutive non-zero completed runs ending at the newest
1171:# ok streak 0 — the newest completed run exited 0. The detail still
1191:# claims one measured `exit=0` and nothing wider.
1197:# exceeded`, 39/39 ended `exit=1`** — so the single legacy needle printed
1224: local lo_p lo_f lo_name lo_id lo_last lo_prev lo_rc lo_age lo_now lo_mtime
1225: local lo_total=0 lo_start lo_idx lo_newest=""
1232: # (`/data/agent-loop.sh` writes `agent-loop-exit=<rc> run=<n>` as the TRUE
1238: local lo_attested_re='^agent-loop-exit=([0-9]+) run=([0-9]+)$'
1259: STATUS_ITEMS+=("{\"name\":\"loop-outcomes\",\"status\":\"warning\",\"detail\":\"cannot verify: sandboxed run (no GLADEX_RUNLOG_DIR, no system systemctl) - the live run-log directory was NOT read\"}")
1274: STATUS_ITEMS+=("{\"name\":\"loop-outcomes\",\"status\":\"warning\",\"detail\":\"cannot verify: no run-*.log in ${lo_dir} (the agent loop has not run here)\"}")
1280: lo_newest="${lo_all[$((lo_total - 1))]}"
1282: lo_mtime="$(stat -c %Y "$lo_newest" 2>/dev/null || true)"
1298: lo_prev=""
1300: # lo_prev is the line ABOVE lo_last at the end of the walk: the
1301: # pair half of the marker test (the loop writes exit= then the
1305: # ends with an empty lo_prev and can never match a pair.
1306: lo_prev="$lo_last"
1310: # the in-flight echo defect; 2026-10-28, the marker-shape false green):
1313: # the else-branch — AND with the loop's own `exit=<rc>` directly above
1317: # run 1014's TEST B, fixed here) — or the legacy `exit=` only on logs
1318: # BEHIND the newest, where the loop has certainly closed the file (only
1319: # the newest log can still be open, its last line transcript content).
1323: # applies the in-flight/stale rules below, unchanged.
1325: if [[ "$lo_last" =~ $lo_attested_re ]] && [[ "${BASH_REMATCH[2]}" == "$lo_id" ]] && [[ "$lo_prev" == "exit=${BASH_REMATCH[1]}" ]]; then
1327: elif [[ "$lo_last" =~ ^exit=([0-9]+)$ ]] && [[ "$lo_f" != "$lo_newest" ]]; then
1337: # The newest COMPLETED run id: this is what the ok line names,
[0m$ [0mcd /data/repo && wc -l tests/test_system_status_failed_units.sh && grep -n '^# \|^## \|§' tests/test_system_status_failed_units.sh | head -60
1932 tests/test_system_status_failed_units.sh
2:# test_system_status_failed_units.sh — the `failed-units` row: the scheduled
3:# READER that queue item (r1b) said was missing.
5:# Queue item (r1b), CHANGELOG [0.4.203].
7:# Why this row exists, and what the three verdicts each mean:
9:# [0.4.202] (queue item (r1)) made `identity-run.sh` end in `exit "$RC"`, so a
10:# failed shift leaves `identity-run@<id>.service` in `systemctl --failed`
11:# instead of reporting `Deactivated successfully.`. Nothing SCHEDULED read
12:# that list: the six `check_service` rows ask seven NAMED units whether they
13:# are active (any other failed unit is invisible to them) and `tools/red-watch`
14:# reads the monitor, which reads `system-status` — so the truth was reachable
15:# by hand and never ALERTED. This row is the reader, and this suite pins it:
17:# ok `systemctl --failed` is empty -> `0 failed unit(s)`.
18:# The only verdict that leaves `overall ok` / exit 0.
19:# error + ERRORS++ a unit IS listed. Deliberately an error, never a
20:# warning: a warning leaves `overall: ok` and exit 0
21:# behind a failed shift — the exact silent green (r1)
22:# was written to remove (same reasoning the
23:# `investor-duty` and `git-tree` rows document).
24:# warning `systemctl` is absent, exits non-zero, or prints a line
25:# that is not a unit name. Never ok (an unchecked box is
26:# not a healthy one), never error (nothing was seen to
27:# fail).
29:# The detail carries the distinction (r1) handed to its reader — a red unit
30:# means "`opencode` returned a non-zero rc", not "no work was done" — by
31:# pairing systemd's own `ExecMainStatus` with the `===== shift ... exit=N =====`
32:# marker in `agent-logs/identity-<id>.md`: equal pair = the shift went red,
33:# `shift_exit=?` = the unit died before a marker was written, and
34:# `MISMATCH-unit-not-shift` = the two witnesses disagree.
36:# [0.4.299] added a fourth field beside them, `work=<n>ins/<n>files`, because
37:# agent-logs/identity-shift-red-modes-2026-10-11.md measured ONE dashboard red
38:# standing for three different outcomes (1664 insertions committed and then a
39:# non-zero rc; 2 lines of failure evidence; no commit at all) and a reader
40:# could tell them apart only by reading git by hand. Section 13 pins it, and
41:# the pin that matters most is the negative one: `work=0` is a number git
42:# gave us, `work=?` is a question we could not ask, and a bare `date`
43:# argument (GNU `date` resolves the string "1" to a window around NOW) must
44:# never be able to turn the second into the first.
46:# [0.4.337] splits a real work= reading into `ownlog=`/`other=` at file
47:# granularity, because lena's 2026-10-13 18:06 red shift proved the single
48:# number could be read three different ways (runs 941/942: "no work", then
49:# "shift-authored 0", while the numstat says 1058 own-log capture + 3 swept
50:# concurrent edits, and the shift's real deliverables were invisible to the
51:# grep entirely). Section 13b pins the split, the two BARE verdicts
52:# (work=0 and work=? carry no fields — there is no reading to split), and
53:# two mutants: the classification defeated (a sweep dressed as transcript,
54:# total unchanged) and the subfields dropped (the pre-split misread, back).
56:# Hermetic: `systemctl` itself is the thing under test here, so unlike the
57:# other `test_system_status_*` suites it is NOT stubbed with a blind
58:# `echo active` — this suite's stub answers `--failed`, `show` and `is-active`
59:# separately (a blind stub is exactly the unparseable-line case, section 9).
60:# Everything else (curl, dig, openssl, go, repo-lint) is stubbed as in
61:# `tests/test_system_status_red_watch.sh`, the rows that would probe the live
62:# box are pointed at absent paths, and every run gets a sandbox repo through
63:# GLADEX_REPO_DIR.
65:# [0.4.308] adds a SECOND row to the same subject, `log-truncation` (section
66:# 15): the guard asks the same seven files the other question — not "which
67:# shift went red" but "is the record of any shift still there". It belongs
68:# beside this suite because both read `agent-logs/identity-*.md` and because
69:# the measured losses §5 of agent-logs/shift-exit-probe-2026-10-11.md found are
70:# the same transcripts `failed-units` reads its marker out of. Its fixtures
> build · mimo-v2.6-flash-free
[0m→ [0mRead repo/tools/system-status [90m[offset=1088, limit=420][0m
> build · mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && echo "=== files mentioning agent-loop-exit ==="; grep -rln 'agent-loop-exit' --include='*' . | head -30; echo; echo "=== counts ==="; grep -rc 'agent-loop-exit' tools/* tests/* 2>/dev/null | grep -v ':0' | head -30; echo; echo "=== watchdog mentions ==="; grep -n 'agent-loop-exit\|exit=' /usr/local/sbin/agent-loop-watchdog.sh | head -20; echo; echo "=== run-* globs in tools ==="; grep -rn 'run-\*\.log\|run-\*' tools/ tests/ 2>/dev/null | grep -v '\.log:' | head -30
=== files mentioning agent-loop-exit ===
./CHANGELOG.md
./.git/logs/HEAD
./.git/logs/refs/heads/main
./tools/REGISTRY.md
./tools/system-status
./agent-logs/PROGRESS.md
./tests/test_agent_loop_watchdog.sh
./tests/test_system_status_failed_units.sh
=== counts ===
tools/REGISTRY.md:2
tools/system-status:6
tests/test_agent_loop_watchdog.sh:12
tests/test_system_status_failed_units.sh:8
=== watchdog mentions ===
19:# `exit=1` endings, every one `Error: Free usage limit reached`, while
33:# (iii) fresh + newest run terminal exit=0 -> INFO, "nothing to do" (the old
43:# * `agent-loop-exit=<rc> run=<id>` counts only when <id> equals this log's
47:# * a legacy `exit=<rc>` counts too, NEWEST log included. `loop-outcomes`
50:# from an echoed `exit=` is a false WARNING, which restarts nothing and
93: 3. fresh + newest run exit=0 -> info, nothing to do
147:# See the header for why a legacy `exit=` counts on the newest here.
151: agent-loop-exit=*)
152: rc=${ll#agent-loop-exit=}; rc=${rc%%[!0-9]*}
160: exit=*)
161: rc=${ll#exit=}; rc=${rc%%[!0-9]*}
=== run-* globs in tools ===
tools/REGISTRY.md:3319: newest `GLADEX_RUNLOG_WINDOW` (default 40) `run-*.log` files and takes each
tools/system-status:1098:# WHAT IS READ: the newest GLADEX_RUNLOG_WINDOW (default 40) `run-*.log` files
tools/system-status:1267: # filters it (and any directory named run-*.log).
tools/system-status:1268: for lo_f in "$lo_dir"/run-*.log; do
tools/system-status:1274: STATUS_ITEMS+=("{\"name\":\"loop-outcomes\",\"status\":\"warning\",\"detail\":\"cannot verify: no run-*.log in ${lo_dir} (the agent loop has not run here)\"}")
tests/test_agent_loop_watchdog.sh:6:# Until 2026-10-28 the watchdog asked ONE question — "did any run-*.log start
tests/test_agent_loop_watchdog.sh:425:if [ -z "$(find "$LOGDIR" -maxdepth 1 -name 'run-*.log' -mmin -45 2>/dev/null)" ]; then
tests/test_system_status_failed_units.sh:78:# /data/agent-loop.sh appends to each `run-*.log`. Its measured harm is
tests/test_system_status_failed_units.sh:1464:lexpect "16f-empty" "$LF" "warning" "cannot verify: no run-*.log" "0" "ok" "0"
tests/test_system_status_failed_units.sh:1485:touch -d '2 hours ago' "$LH"/run-*.log
[0m$ [0mcd /data/repo && ./tools/system-status --format json 2>&1 | python3 -c "
import json,sys
d=json.load(sys.stdin)
print('overall:',d.get('overall'),'errors:',d.get('errors'))
for it in d.get('items',[]):
if it.get('status')!='ok': print(' ',it.get('name'),'::',it.get('status'),'::',it.get('detail'))
" 2>&1 | head -30
overall: error errors: 4
> build · mimo-v2.6-flash-free
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail — prod-8001.log (last 60 lines)
[Fri Oct 30 01:32:28 2026] 127.0.0.1:59906 Accepted [Fri Oct 30 01:32:28 2026] 127.0.0.1:59906 Closing [Fri Oct 30 01:32:28 2026] 127.0.0.1:59920 Accepted [Fri Oct 30 01:32:28 2026] 127.0.0.1:59920 Closing [Fri Oct 30 01:32:28 2026] 127.0.0.1:59926 Accepted [Fri Oct 30 01:32:28 2026] 127.0.0.1:59926 Closing [Fri Oct 30 01:32:28 2026] 127.0.0.1:59932 Accepted [Fri Oct 30 01:32:28 2026] 127.0.0.1:59932 Closing [Fri Oct 30 01:32:28 2026] 127.0.0.1:59948 Accepted [Fri Oct 30 01:32:29 2026] 127.0.0.1:59948 Closing [Fri Oct 30 01:32:29 2026] 127.0.0.1:59962 Accepted [Fri Oct 30 01:32:29 2026] 127.0.0.1:59962 Closing [Fri Oct 30 01:32:50 2026] 127.0.0.1:34738 Accepted [Fri Oct 30 01:32:50 2026] 127.0.0.1:34738 Closing [Fri Oct 30 01:33:30 2026] 127.0.0.1:36222 Accepted [Fri Oct 30 01:33:30 2026] 127.0.0.1:36222 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35438 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35438 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35440 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35440 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35456 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35456 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35460 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35460 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35474 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35474 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35486 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35486 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35488 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35488 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35492 Accepted [Fri Oct 30 01:33:47 2026] 127.0.0.1:35492 Closing [Fri Oct 30 01:33:47 2026] 127.0.0.1:35500 Accepted [Fri Oct 30 01:33:48 2026] 127.0.0.1:35500 Closing [Fri Oct 30 01:33:48 2026] 127.0.0.1:35508 Accepted [Fri Oct 30 01:33:48 2026] 127.0.0.1:35508 Closing [Fri Oct 30 01:33:51 2026] 127.0.0.1:35522 Accepted [Fri Oct 30 01:33:51 2026] 127.0.0.1:35522 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55476 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55476 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55478 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55478 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55482 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55482 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55490 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55490 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55496 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55496 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55508 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55508 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55522 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55522 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55524 Accepted [Fri Oct 30 01:35:00 2026] 127.0.0.1:55524 Closing [Fri Oct 30 01:35:00 2026] 127.0.0.1:55532 Accepted [Fri Oct 30 01:35:01 2026] 127.0.0.1:55532 Closing [Fri Oct 30 01:35:01 2026] 127.0.0.1:55546 Accepted [Fri Oct 30 01:35:01 2026] 127.0.0.1:55546 Closing [Fri Oct 30 01:35:28 2026] 127.0.0.1:38354 Accepted
Other app logs: downloads.log
Generated 2026-10-30 00:35:28 UTC · Gladex.de