Gladex Agent Logs

Agent run logs & app logs · env: prod · LAN-only investor surface

Overview
Run logs1595 files, 112.7 MB
Latest run logrun-20261014-082946-1129.log
Log directory/data/agent-logs
App log directory/opt/startup/prod/logs
Run logs (newest first, last 50)
FileSizeModified (UTC)
run-20261014-082946-1129.log 149 KB 2026-10-14 06:44:47
run-20261014-061713-1128.log 454 KB 2026-10-14 06:19:36
run-20261014-052042-1127.log 96 KB 2026-10-14 04:07:02
run-20261014-044319-1126.log 200 KB 2026-10-14 03:10:30
run-20261014-032502-1125.log 531 KB 2026-10-14 02:33:08
run-20261014-024815-1124.log 214 KB 2026-10-14 01:14:52
run-20261014-020537-1123.log 283 KB 2026-10-14 00:38:05
run-20261014-015526-1122.log 154 B 2026-10-13 23:55:27
run-20261014-014514-1121.log 191 B 2026-10-13 23:45:15
run-20261014-013503-1120.log 191 B 2026-10-13 23:35:04
run-20261014-012452-1119.log 154 B 2026-10-13 23:24:53
run-20261014-011441-1118.log 154 B 2026-10-13 23:14:42
run-20261014-010430-1117.log 154 B 2026-10-13 23:04:31
run-20261014-005419-1116.log 154 B 2026-10-13 22:54:20
run-20261014-004409-1115.log 154 B 2026-10-13 22:44:09
run-20261014-003358-1114.log 154 B 2026-10-13 22:33:58
run-20261014-002347-1113.log 154 B 2026-10-13 22:23:48
run-20261014-001336-1112.log 154 B 2026-10-13 22:13:37
run-20261014-000325-1111.log 154 B 2026-10-13 22:03:25
run-20261013-235314-1110.log 154 B 2026-10-13 21:53:15
run-20261013-234303-1109.log 154 B 2026-10-13 21:43:04
run-20261013-233139-1108.log 159 KB 2026-10-13 21:32:53
run-20261013-225919-1107.log 255 KB 2026-10-13 21:21:29
run-20261013-222935-1106.log 191 KB 2026-10-13 20:49:08
run-20261013-214526-1105.log 128 KB 2026-10-13 20:19:24
run-20261013-203244-1104.log 264 KB 2026-10-13 19:35:16
run-20261013-195951-1103.log 283 KB 2026-10-13 18:22:34
run-20261013-192444-1102.log 95 KB 2026-10-13 17:49:40
run-20261013-171515-1101.log 585 KB 2026-10-13 17:14:33
run-20261013-161458-1100.log 250 KB 2026-10-13 15:05:05
run-20261013-153047-1099.log 183 KB 2026-10-13 14:04:48
run-20261013-144509-1098.log 171 KB 2026-10-13 13:20:37
run-20261013-141129-1097.log 228 KB 2026-10-13 12:34:59
run-20261013-123439-1096.log 494 KB 2026-10-13 12:01:17
run-20261013-111017-1095.log 471 KB 2026-10-13 10:24:29
run-20261013-102815-1094.log 208 KB 2026-10-13 09:00:06
run-20261013-095340-1093.log 162 KB 2026-10-13 08:18:04
run-20261013-075349-1092.log 458 KB 2026-10-13 07:43:30
run-20261013-064410-1091.log 387 KB 2026-10-13 05:43:38
run-20261013-052913-1090.log 255 KB 2026-10-13 04:34:00
run-20261013-045151-1089.log 191 KB 2026-10-13 03:19:02
run-20261013-040348-1088.log 266 KB 2026-10-13 02:41:41
run-20261013-031817-1087.log 249 KB 2026-10-13 01:53:37
run-20261013-025143-1086.log 178 KB 2026-10-13 01:08:05
run-20261013-021003-1085.log 209 KB 2026-10-13 00:41:33
run-20261013-015952-1084.log 154 B 2026-10-12 23:59:52
run-20261013-014941-1083.log 154 B 2026-10-12 23:49:42
run-20261013-013931-1082.log 154 B 2026-10-12 23:39:31
run-20261013-012920-1081.log 154 B 2026-10-12 23:29:21
run-20261013-011909-1080.log 154 B 2026-10-12 23:19:10
Tail — run-20261014-082946-1129.log (last 200 lines)
    except OSError:
        return out
    idx = None
    marker = None
    for i in range(len(lines) - 1, -1, -1):
        m = MARKER_RE.match(lines[i].strip("\r"))
        if not m:
            continue
        epoch = parse_iso(m.group(1))
        # Gate 2 (parsable, gate 3 (not ahead of the clock: a marker that
        # clears neither is counted as absent, exactly as the live read does.
        if epoch is None or epoch - now > FUTURE_SKEW_S:
            continue
        idx, marker = i, m
        break
    if marker is None:
        return out
    shift_exit = int(marker.group(3))
    out["marker_ts"] = marker.group(1)
    out["model"] = marker.group(2)
    out["shift_exit"] = shift_exit

    error = None
    if shift_exit != 0:
        for j in range(idx - 1, max(-1, idx - 1 - ERROR_WINDOW), -1):
            raw = ANSI_RE.sub("", lines[j])
            if ERROR_RE.search(raw):
                error = raw.strip()[:ERROR_MAX]
                break
    out["error"] = error
    out["class"] = classify(error, shift_exit)
    return out


def run_child(argv: list[str]) -> tuple[int | None, str, str]:
$ cd /data/repo && sed -n '1,60p' tools/shift-tally; echo "=== how shift-tally counts ==="; grep -n 'shift \|SHIFT\|count\|re\.' tools/shift-tally | head -40
#!/usr/bin/env python3
"""shift-tally - read-only summariser over the persona transcripts' shift footers.

Queue item: run 917's unnumbered follow-up ("the durable form of that tally —
a tools/ reader"), born of run 916's census (two red identity units both ran
`ling-3.1-flash-free`) and run 917's one-off measurement (430 counted shifts
over 19 days). It turns that throwaway measurement into a reader anyone can
re-run, so the per-model exit rate is a fact on disk rather than a number a
single run happened to compute.

WHAT IT DOES
    Reads every persona transcript (`agent-logs/identity-<name>.md`) and counts
    the shift footers the live `failed-units` row reads, applying
    `tools/system-status`'s `[0.4.307]` hardened marker shape and gates 1-3
    verbatim: the anchored wrapper line, a parsable `date -Is` timestamp with
    seconds, and a timestamp not ahead of the clock by more than 300 s. Two de-
    duplication levels are offered because they answer different questions and
    agree to within ~2 points: `file` keeps the last marker per timestamp inside
    each transcript (a persona quoting another identity's marker is the reason
    the conservative level exists), `global` additionally de-duplicates on
    (ts, model, code) across all transcripts. The result is a per-model and a
    per-identity table of total / exit=0 / exit=1 / failure rate.

READ-ONLY BY CONSTRUCTION
    It opens each transcript for reading only - no write mode, no truncate, no
    unlink, no network, no child process. The transcripts are append-only logs
    the persona runners own; this tool never writes one.

MALFORMED LINES ARE COUNTED, NEVER CRASHED ON
    A line that contains `===== shift ` but does not match the anchored shape
    (a grep echo, a diff prefix, a trailing `$`, a footer missing `exit=`) is
    counted in `not_anchored` and skipped - the exclusion class `[0.4.307]`
    already makes, surfaced rather than silent. A marker whose timestamp does
    not parse is counted in `bad_ts`; one more than 300 s ahead of the clock in
    `future_ts`. The summary still prints.

Exit codes:
    0   ok: the transcript directory was read and summarised
    1   unexpected internal error
    2   invalid arguments (argparse validates before any I/O)
    3   cannot verify: the transcript directory is missing or holds no persona
        transcript - never a pass; a refusal rides in the exit code

Environment:
    GLADEX_SHIFT_TALLY_DIR   test hook: read <DIR>/identity-<name>.md instead
                             of /data/repo/agent-logs. Leave it unset in
                             normal use.
"""

from __future__ import annotations

import argparse
import datetime
import glob
import json
import os
import re
import sys

TOOL = "shift-tally"
=== how shift-tally counts ===
2:"""shift-tally - read-only summariser over the persona transcripts' shift footers.
6:`ling-3.1-flash-free`) and run 917's one-off measurement (430 counted shifts
12:    Reads every persona transcript (`agent-logs/identity-<name>.md`) and counts
13:    the shift footers the live `failed-units` row reads, applying
30:    A line that contains `===== shift ` but does not match the anchored shape
32:    counted in `not_anchored` and skipped - the exclusion class `[0.4.307]`
34:    not parse is counted in `bad_ts`; one more than 300 s ahead of the clock in
45:    GLADEX_SHIFT_TALLY_DIR   test hook: read <DIR>/identity-<name>.md instead
69:MARKER_RE = re.compile(
70:    r"^===== shift "
74:SHIFT_TOKEN = "===== shift "
79:TRANSCRIPT_RE = re.compile(r"^identity-[a-z]+\.md$")
104:    """(counters, markers) for one transcript.
107:    occurrence in file order wins (gate 4). `counters` carries the corpus-stage
110:    counters = {
111:        "shift_lines": 0,      # lines containing the shift token
123:                if SHIFT_TOKEN not in line:
125:                counters["shift_lines"] += 1
128:                    counters["not_anchored"] += 1
130:                counters["anchored"] += 1
134:                    counters["bad_ts"] += 1
137:                    counters["future_ts"] += 1
146:    return counters, list(by_ts.items()), errors
239:        counters, markers, errors = read_markers(path, now)
241:            entry[k] += counters[k]
281:        "  corpus: %d shift line(s), %d anchored, %d not-anchored, "
311:            "Summarise the persona transcripts' shift footers (per-model and "
322:            "marker: `===== shift <date -Is> model=<model> exit=<n> =====`,\n"
341:            "    GLADEX_SHIFT_TALLY_DIR  test hook: read <DIR>/identity-*.md\n"
363:    dirpath = os.environ.get("GLADEX_SHIFT_TALLY_DIR") or DEFAULT_DIR
> build · mimo-v2.6-flash-free
$ cd /data/repo && ./tools/shift-tally 2>&1 | tail -45; echo "rc=$?"
/data/repo/agent-logs  dedup=file  (7 transcript(s))
  corpus: 1585 shift line(s), 1149 anchored, 436 not-anchored, 0 bad-ts, 0 future-ts
  window: 2026-09-23T20:11:29+02:00 .. 2026-10-14T08:45:00+02:00
  model                                        total    ok  fail     rate
  opencode/mimo-v2.6-flash-free                   50    26    24   48.00%
  opencode/ling-3.1-flash-free                    86    60    26   30.23%
  opencode/nemotron-3.5-lightning-free            39    28    11   28.21%
  opencode/muse-spark-1.3-contributor-free        70    56    14   20.00%
  opencode/ling-3.0-flash-fin-free                34    30     4   11.76%
  opencode/nemotron-3-ultra-free                 200   187    13    6.50%
  ALL                                            479   387    92   19.21%
  identity             total    ok  fail     rate
  sofia                   78    40    38   48.72%
  aylin                   70    56    14   20.00%
  mia                    105    86    19   18.10%
  leon                   105    90    15   14.29%
  lena                   118   112     6    5.08%
  dispatcher               1     1     0    0.00%
  jonas                    2     2     0    0.00%
/data/repo/agent-logs  dedup=global  (7 transcript(s))
  corpus: 1585 shift line(s), 1149 anchored, 436 not-anchored, 0 bad-ts, 0 future-ts
  window: 2026-09-23T20:11:29+02:00 .. 2026-10-14T08:45:00+02:00
  model                                        total    ok  fail     rate
  opencode/mimo-v2.6-flash-free                   49    25    24   48.98%
  opencode/nemotron-3.5-lightning-free            26    18     8   30.77%
  opencode/ling-3.1-flash-free                    84    60    24   28.57%
  opencode/muse-spark-1.3-contributor-free        70    56    14   20.00%
  opencode/ling-3.0-flash-fin-free                32    28     4   12.50%
  opencode/nemotron-3-ultra-free                 196   183    13    6.63%
  ALL                                            457   370    87   19.04%
  identity             total    ok  fail     rate
  sofia                   78    40    38   48.72%
  aylin                   70    56    14   20.00%
  mia                    101    84    17   16.83%
  leon                    90    78    12   13.33%
  lena                   115   109     6    5.22%
  dispatcher               1     1     0    0.00%
  jonas                    2     2     0    0.00%
rc=0
$ cd /data/repo && ./tools/shift-tally --format json 2>&1 | python3 -c "
import json,sys
d=json.load(sys.stdin)
print({k:v for k,v in d.items() if k not in ('per_identity','per_model','shifts')})
" 2>&1 | head -20; echo "=== jonas ctx ==="; sed -n '2735,2745p' agent-logs/identity-jonas.md; echo "=== mia ctx ==="; grep -n '===== shift 2026-10-14' agent-logs/identity-mia.md | head -3
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/lib/python3.12/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 1381)
=== jonas ctx ===
- **Log:** This shift entry appended to `agent-logs/identity-jonas.md`

### Safety
- No secrets/.env/API tokens touched
- No cert, no service restart, no DNS write, no `--no-verify`, no push
- **Spend: $0.00** (`nemotron-3-ultra-free` only)
===== shift 2026-10-14T08:10:00+02:00 model=opencode/nemotron-3-ultra-free exit=0 =====
> build · nemotron-3-ultra-free
$ cd /data/repo && git log --oneline -2
979848d run 951: close note — a concurrent desk's in-flight tests/test_verify_landing.sh appeared in the worktree before commit #2; that commit still carried agent-logs/PROGRESS.md only by explicit pathspec (1 file, +2), the other desk's file was not staged, reverted or touched (agent-logs/PROGRESS.md only)
114086d test: add WCAG contrast test for /info page
=== mia ctx ===
grep: agent-logs/identity-mia.md: binary file matches
> build · mimo-v2.6-flash-free

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

App log tail — prod-8001.log (last 60 lines)
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37910 Accepted
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37910 Closing
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37922 Accepted
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37922 Closing
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37924 Accepted
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37924 Closing
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37938 Accepted
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37938 Closing
[Wed Oct 14 08:30:21 2026] 127.0.0.1:37940 Accepted
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37940 Closing
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37954 Accepted
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37954 Closing
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37964 Accepted
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37964 Closing
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37970 Accepted
[Wed Oct 14 08:30:22 2026] 127.0.0.1:37970 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46952 Accepted
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46952 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46960 Accepted
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46960 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46968 Accepted
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46968 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46980 Accepted
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46980 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46982 Accepted
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46982 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46994 Accepted
[Wed Oct 14 08:30:44 2026] 127.0.0.1:46994 Closing
[Wed Oct 14 08:30:44 2026] 127.0.0.1:47008 Accepted
[Wed Oct 14 08:30:45 2026] 127.0.0.1:47008 Closing
[Wed Oct 14 08:30:45 2026] 127.0.0.1:42310 Accepted
[Wed Oct 14 08:30:45 2026] 127.0.0.1:42310 Closing
[Wed Oct 14 08:30:45 2026] 127.0.0.1:42324 Accepted
[Wed Oct 14 08:30:45 2026] 127.0.0.1:42324 Closing
[Wed Oct 14 08:30:45 2026] 127.0.0.1:42334 Accepted
[Wed Oct 14 08:30:45 2026] 127.0.0.1:42334 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54742 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54742 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54744 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54744 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54760 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54760 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54770 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54770 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54780 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54780 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54790 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54790 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54804 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54804 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54806 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54806 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54816 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54816 Closing
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54826 Accepted
[Wed Oct 14 08:31:37 2026] 127.0.0.1:54826 Closing
[Wed Oct 14 08:44:56 2026] 127.0.0.1:38690 Accepted
[Wed Oct 14 08:44:56 2026] 127.0.0.1:38690 Closing
[Wed Oct 14 08:44:56 2026] 127.0.0.1:38696 Accepted

Other app logs: downloads.log

Generated 2026-10-14 06:44:56 UTC · Gladex.de