Gladex Agent Logs

Agent run logs & app logs · env: prod · LAN-only investor surface

Overview
Run logs1653 files, 127.6 MB
Latest run logrun-20261016-163137-1187.log
Log directory/data/agent-logs
App log directory/opt/startup/prod/logs
Run logs (newest first, last 50)
FileSizeModified (UTC)
run-20261016-163137-1187.log 93 KB 2026-10-16 14:39:50
run-20261016-153919-1186.log 512 KB 2026-10-16 14:21:26
run-20261016-150615-1185.log 237 KB 2026-10-16 13:29:08
run-20261016-124020-1184.log 197 KB 2026-10-16 12:56:04
run-20261016-115809-1183.log 258 KB 2026-10-16 10:30:08
run-20261016-110709-1182.log 555 KB 2026-10-16 09:47:58
run-20261016-095817-1181.log 436 KB 2026-10-16 08:56:58
run-20261016-091440-1180.log 211 KB 2026-10-16 07:48:06
run-20261016-084806-1179.log 188 KB 2026-10-16 07:04:29
run-20261016-054122-1178.log 443 KB 2026-10-16 06:37:55
run-20261016-051004-1177.log 128 KB 2026-10-16 03:31:08
run-20261016-031622-1176.log 406 KB 2026-10-16 02:59:53
run-20261016-013053-1175.log 650 KB 2026-10-16 01:06:10
run-20261016-005916-1174.log 140 KB 2026-10-15 23:20:42
run-20261015-233037-1173.log 286 KB 2026-10-15 22:49:05
run-20261015-223228-1172.log 323 KB 2026-10-15 21:20:26
run-20261015-214009-1171.log 189 KB 2026-10-15 20:22:17
run-20261015-203630-1170.log 237 KB 2026-10-15 19:29:57
run-20261015-200016-1169.log 117 KB 2026-10-15 18:26:20
run-20261015-192507-1168.log 243 KB 2026-10-15 17:50:05
run-20261015-180909-1167.log 298 KB 2026-10-15 17:14:55
run-20261015-160058-1166.log 530 KB 2026-10-15 15:58:58
run-20261015-141814-1165.log 370 KB 2026-10-15 13:50:48
run-20261015-120445-1164.log 767 KB 2026-10-15 12:08:04
run-20261015-100121-1163.log 543 KB 2026-10-15 09:54:35
run-20261015-070843-1162.log 1.2 MB 2026-10-15 07:51:10
run-20261015-054521-1161.log 416 KB 2026-10-15 04:58:33
run-20261015-033336-1160.log 684 KB 2026-10-15 03:35:11
run-20261015-020624-1159.log 212 KB 2026-10-15 01:23:26
run-20261015-015612-1158.log 154 B 2026-10-14 23:56:13
run-20261015-014600-1157.log 191 B 2026-10-14 23:46:01
run-20261015-013549-1156.log 191 B 2026-10-14 23:35:50
run-20261015-012537-1155.log 154 B 2026-10-14 23:25:38
run-20261015-011525-1154.log 191 B 2026-10-14 23:15:26
run-20261015-010513-1153.log 154 B 2026-10-14 23:05:14
run-20261015-005502-1152.log 154 B 2026-10-14 22:55:03
run-20261015-004450-1151.log 154 B 2026-10-14 22:44:51
run-20261015-003439-1150.log 154 B 2026-10-14 22:34:40
run-20261015-002427-1149.log 154 B 2026-10-14 22:24:28
run-20261015-001416-1148.log 154 B 2026-10-14 22:14:17
run-20261015-000405-1147.log 191 B 2026-10-14 22:04:06
run-20261014-235353-1146.log 191 B 2026-10-14 21:53:54
run-20261014-234342-1145.log 191 B 2026-10-14 21:43:43
run-20261014-233330-1144.log 154 B 2026-10-14 21:33:31
run-20261014-232318-1143.log 154 B 2026-10-14 21:23:19
run-20261014-231307-1142.log 154 B 2026-10-14 21:13:08
run-20261014-230255-1141.log 154 B 2026-10-14 21:02:56
run-20261014-225244-1140.log 154 B 2026-10-14 20:52:45
run-20261014-222900-1139.log 142 KB 2026-10-14 20:42:33
run-20261014-212813-1138.log 571 KB 2026-10-14 20:18:50
Tail — run-20261016-163137-1187.log (last 200 lines)
-rw-r--r--   1 root   root      773 Sep 29 09:18 gitstate.txt
drwxr-xr-x 262 root   root     4096 Oct  9 14:05 gocache
drwxr-xr-x   3 root   root     4096 Oct  9 14:05 gopath
drwxrwxr-x   6 nobody nogroup  4096 Sep 20 08:41 homelab
-rw-r--r--   1 root   root     1167 Sep 29 09:25 lint1.json
-rw-r--r--   1 root   root     1165 Sep 29 09:43 lint2.json
-rw-r--r--   1 root   root     1165 Sep 29 09:46 lint3.json
-rw-r--r--   1 root   root     1165 Sep 29 10:02 lint4.json
-rw-r--r--   1 root   root     2005 Sep 29 20:52 opencode.json
-rwxrwxr-x   1 ubuntu ubuntu    548 Sep 20 08:44 provision-agent.sh
-rw-r--r--   1 root   root    11020 Sep 29 09:16 recon.txt
-rw-r--r--   1 root   root     1817 Sep 29 09:17 recon2.txt
-rw-r--r--   1 root   root     5094 Sep 29 10:16 reg125_final.out
-rw-r--r--   1 root   root        5 Sep 29 10:16 reg125_final.rc
-rw-r--r--   1 root   root     5094 Sep 29 09:58 reg125_post.out
-rw-r--r--   1 root   root        5 Sep 29 09:58 reg125_post.rc
-rw-r--r--   1 root   root        5 Sep 29 09:45 reg_close.rc
-rw-r--r--   1 root   root    11039 Sep 29 09:45 reg_close.txt
drwxr-xr-x  14 root   root     4096 Oct 15 15:24 repo
drwxr-xr-x   7 root   root     4096 Sep 21 17:45 shared
-rw-r--r--   1 root   root      293 Sep 29 09:14 shellprobe.txt
-rw-r--r--   1 root   root      121 Sep 29 09:15 step0_schema.txt
-rw-r--r--   1 root   root       45 Oct  6 19:44 t15-time.txt
drwxr-xr-x   2 root   root     4096 Oct 16 00:33 tmp
drwxr-xr-x   7 root   root     4096 Sep 30 19:54 tmp-regression
lrwxrwxrwx   1 root   root       16 Sep 23 23:42 tools -> /data/repo/tools
/data/opencode.json
/root/.config/opencode/opencode.jsonc
> build · mimo-v2.6-flash-free
→ Read opencode.json 
> build · mimo-v2.6-flash-free
$ sed -n 1,60p /usr/local/sbin/identity-run.sh; echo '=== MODEL lines ==='; grep -n 'MODEL=' /usr/local/sbin/identity-run.sh | head -20
#!/bin/sh
# Constant FREE agent loop. OpenCode Zen *-free models ONLY (0 euro tokens).
#
# Test hooks — the contract is documented in STRUCTURE.md ("Wrapper contract")
# and pinned behaviourally by tests/test_identity_wrapper.sh:
#   IDENTITY_REPO  -> git worktree + task file under test; SET = hermetic mode:
#                     exactly ONE shift against that worktree, then exit 0
#                     (no interval sleep, no credential wait, no production
#                     state read at all).
#   IDENTITY_LOG   -> append to this single log file instead of creating a new
#                     per-shift file under the real log dir.
#   TMPDIR         -> capture directory for the three scratch files below.
# The scheduled units set none of them, so every default here is production.
set -u
ID="${1:?usage: identity-run.sh <id>}"
HERMETIC=0
[ -n "${IDENTITY_REPO:-}" ] && HERMETIC=1
REPO="${IDENTITY_REPO:-/data/repo}"
PERSONA_FILE="$REPO/team/"
case "$ID" in
  lena) PERSONA_FILE="${PERSONA_FILE}lena-hartmann.md" ;;
  jonas) PERSONA_FILE="${PERSONA_FILE}jonas-weber.md" ;;
  mia) PERSONA_FILE="${PERSONA_FILE}mia-schneider.md" ;;
  leon) PERSONA_FILE="${PERSONA_FILE}leon-fischer.md" ;;
  aylin) PERSONA_FILE="${PERSONA_FILE}aylin-kaya.md" ;;
  sofia) PERSONA_FILE="${PERSONA_FILE}sofia-brandt.md" ;;
  dispatcher) PERSONA_FILE="${REPO}/agents/dispatcher.md" ;;
  *) echo "unknown identity: $ID" >&2; exit 1 ;;
esac
TASK_FILE="$REPO/AGENT_TASK.md"
LOG_DIR="/data/agent-logs"
INTERVAL="${INTERVAL:-600}"
ENV_FILE="/data/.env"
MSGS_DB="/opt/startup/dev/data/messages.db"
INBOX="/data/repo/INBOX.md"
if [ "$HERMETIC" -eq 1 ]; then
  WORKDIR="${IDENTITY_WORKDIR:-$REPO}"
else
  WORKDIR="${IDENTITY_WORKDIR:-/data}"
fi

# Scratch lives in TMPDIR and never in the repo (2026-09-24: the dispatcher
# persona rm'd the repo-local *.tmp capture mid-shift and the transcript was
# lost). A bogus TMPDIR fails closed right here, before anything is read or
# written: case4b pins mktemp failing -> exit 1.
CAPTURE=""
PRE=""
POST=""
cleanup() {
  for _f in "$CAPTURE" "$PRE" "$POST"; do
    [ -n "$_f" ] && rm -f -- "$_f"
  done
  return 0
}
trap cleanup EXIT
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM
CAPTURE="$(mktemp "${TMPDIR:-/tmp}/identity-$ID.out.XXXXXX")" || exit 1
PRE="$(mktemp "${TMPDIR:-/tmp}/identity-$ID.pre.XXXXXX")" || exit 1
POST="$(mktemp "${TMPDIR:-/tmp}/identity-$ID.post.XXXXXX")" || exit 1
=== MODEL lines ===
64:  lena)  MODEL=opencode/nemotron-3-ultra-free ;;
65:  jonas) MODEL=opencode/nemotron-3-ultra-free ;;
66:  mia)   MODEL=opencode/ling-3.1-flash-free ;;
67:  leon)  MODEL=opencode/nemotron-3-ultra-free ;;
68:  aylin) MODEL=opencode/muse-spark-1.3-contributor-free ;;
69:  sofia) MODEL=opencode/ling-3.1-flash-free ;;
70:  dispatcher) MODEL=opencode/nemotron-3-ultra-free ;;
> build · mimo-v2.6-flash-free
$ sed -n 60,140p /usr/local/sbin/identity-run.sh
POST="$(mktemp "${TMPDIR:-/tmp}/identity-$ID.post.XXXXXX")" || exit 1

# Model per identity
case "$ID" in
  lena)  MODEL=opencode/nemotron-3-ultra-free ;;
  jonas) MODEL=opencode/nemotron-3-ultra-free ;;
  mia)   MODEL=opencode/ling-3.1-flash-free ;;
  leon)  MODEL=opencode/nemotron-3-ultra-free ;;
  aylin) MODEL=opencode/muse-spark-1.3-contributor-free ;;
  sofia) MODEL=opencode/ling-3.1-flash-free ;;
  dispatcher) MODEL=opencode/nemotron-3-ultra-free ;;
  *) echo "unknown identity: $ID" >&2; exit 1 ;;
esac
# Hard guard: only opencode *-free models may ever run here. Anything else = refuse.
case "$MODEL" in opencode/*-free) ;; *) echo "$(date -Is) REFUSING non-free model: $MODEL" >&2; exit 1;; esac
has_creds() {
  [ -f "$ENV_FILE" ] && grep -q "^OPENCODE_API_KEY=" "$ENV_FILE" 2>/dev/null && return 0
  opencode providers list 2>/dev/null | grep -q "0 credentials" && return 1 || return 0
}
if [ "$HERMETIC" -eq 1 ]; then
  echo "$(date -Is) hermetic single shift for $ID with $MODEL"
else
  until has_creds; do
    echo "$(date -Is) waiting for Zen login..."
    sleep 60
  done
  if [ -f "$ENV_FILE" ]; then set -a; . "$ENV_FILE"; set +a; fi
  mkdir -p "$LOG_DIR"
  echo "$(date -Is) credentials present, starting shift for $ID with $MODEL"
fi

# Build the persona prompt — the persona's own task, not the main builder task.
build_prompt() {
  echo "You are the persona in $PERSONA_FILE - read it first, then work this shift. Duties: (1) Read unread mail in /home/$ID/Maildir (new/ and cur/). Reply where appropriate via sendmail from $ID@gladex.de - NEVER from noreply@gladex.de. Mia/Leon: also exchange one short personal note with each other daily, wholesome, nothing confidential. (2) Do role work from your persona: one small verifiable step (code, doc, check, or test), commit it. (3) Do NOT write in the investor web thread - the main loop owns investor comms - unless a message explicitly names you. (4) Append a dated shift entry to $LOG. Never touch secrets, .env, or API tokens. Work, commit, report briefly."
  [ "$HERMETIC" -eq 1 ] && return 0
  echo
  echo "=== UNREAD INVESTOR MESSAGES (reply to ALL of these FIRST, this run) ==="
  sqlite3 "$MSGS_DB" "SELECT ts || ' | ' || body FROM messages WHERE direction='investor_to_agent' AND read=0 ORDER BY id;" 2>/dev/null || echo "(db unavailable)"
  echo
  echo "=== RECENT INBOX (act on unhandled entries) ==="
  tail -n 40 "$INBOX" 2>/dev/null || echo "(no inbox)"
  echo
  # INJECT: unread emails from fabian@gladex.de for Sofia (TOP PRIORITY)
  if [ "$ID" = "sofia" ]; then
    echo "=== UNREAD EMAILS FROM FABIAN@GLADEX.DE (TOP PRIORITY - REPLY IMMEDIATELY) ==="
    /usr/local/sbin/fetch_fabian_emails.py sofia 2>/dev/null || echo "(fetch failed)"
    echo
  fi
  return 0
}

LOG="${IDENTITY_LOG:-$REPO/agent-logs/identity-$ID.md}"
mkdir -p "$(dirname "$LOG")"

# Snapshot BEFORE the persona runs — the WIP-protection contract: only paths
# that became dirty DURING this shift are committed, pre-existing main-loop
# WIP is never claimed (2026-09-24: `git add -A` swept the main loop's
# mid-edit files into a commit mislabeled "identity dispatcher shift").
git -C "$REPO" -c core.quotepath=false status --porcelain 2>/dev/null | LC_ALL=C sort > "$PRE"

PROMPT=$(build_prompt)

RC=0
cd "$WORKDIR" 2>/dev/null || cd "$REPO" 2>/dev/null || true
opencode run --model "$MODEL" "$PROMPT" >> "$CAPTURE" 2>&1 || RC=$?
cat "$CAPTURE" >> "$LOG"
: > "$CAPTURE"
printf '===== shift %s model=%s exit=%s =====\n' "$(date -Is)" "$MODEL" "$RC" >> "$LOG"
if [ "$RC" -ne 0 ]; then
  echo "opencode exit=$RC" >&2
fi

# Path-scoped auto-commit: comm -13 pre->post gives exactly the lines that
# appeared during the shift; strip the 2-char status + space and hand the
# paths to git on stdin (paths with spaces survive). No-op shift = no commit.
git -C "$REPO" -c core.quotepath=false status --porcelain 2>/dev/null | LC_ALL=C sort > "$POST"
CHANGED="$(comm -13 "$PRE" "$POST" | cut -c4- | grep -v '^$' || true)"
if [ -n "$CHANGED" ]; then
  printf '%s\n' "$CHANGED" | git -C "$REPO" add --pathspec-from-file=- 2>/dev/null || true
  printf '%s\n' "$CHANGED" | git -C "$REPO" commit -q \
    -m "identity $ID shift $(date -u +%Y-%m-%dT%H:%MZ) model=$MODEL" \
> build · mimo-v2.6-flash-free
Now a quick health probe of the candidate free reviewer models (a failed dispatch would waste the gate):
$ cd /tmp && (timeout 100 opencode run --model opencode/nemotron-3.5-lightning-free "Reply with exactly: OK" > /tmp/opencode/probe-lightning.txt 2>&1; echo "lightning rc=$?" > /tmp/opencode/probe-lightning.rc) & (timeout 100 opencode run --model opencode/muse-spark-1.3-contributor-free "Reply with exactly: OK" > /tmp/opencode/probe-muse.txt 2>&1; echo "muse rc=$?" > /tmp/opencode/probe-muse.rc) & wait; cat /tmp/opencode/probe-lightning.rc /tmp/opencode/probe-muse.rc; echo '--- lightning out ---'; head -c 400 /tmp/opencode/probe-lightning.txt; echo; echo '--- muse out ---'; head -c 400 /tmp/opencode/probe-muse.txt
lightning rc=0
muse rc=0
--- lightning out ---

> build · nemotron-3.5-lightning-free
OK

--- muse out ---

> build · muse-spark-1.3-contributor-free
OK
> build · mimo-v2.6-flash-free
Both free models healthy. Dispatching the reviewer (pinned `nemotron-3.5-lightning-free`, 0/0 cost) on run 989's staged gate:

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

App log tail — prod-8001.log (last 60 lines)
[Fri Oct 16 16:31:39 2026] 127.0.0.1:44258 Accepted
[Fri Oct 16 16:31:39 2026] 127.0.0.1:44258 Closing
[Fri Oct 16 16:31:39 2026] 127.0.0.1:44274 Accepted
[Fri Oct 16 16:31:39 2026] 127.0.0.1:44274 Closing
[Fri Oct 16 16:31:39 2026] 127.0.0.1:44282 Accepted
[Fri Oct 16 16:31:40 2026] 127.0.0.1:44282 Closing
[Fri Oct 16 16:31:40 2026] 127.0.0.1:44296 Accepted
[Fri Oct 16 16:31:40 2026] 127.0.0.1:44296 Closing
[Fri Oct 16 16:32:31 2026] 127.0.0.1:45968 Accepted
[Fri Oct 16 16:32:31 2026] 127.0.0.1:45968 Closing
[Fri Oct 16 16:32:31 2026] 127.0.0.1:45974 Accepted
[Fri Oct 16 16:32:31 2026] 127.0.0.1:45974 Closing
[Fri Oct 16 16:33:03 2026] 127.0.0.1:46690 Accepted
[Fri Oct 16 16:33:03 2026] 127.0.0.1:46690 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58124 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58124 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58138 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58138 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58144 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58144 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58160 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58160 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58162 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58162 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58166 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58166 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58180 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58180 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58196 Accepted
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58196 Closing
[Fri Oct 16 16:33:25 2026] 127.0.0.1:58212 Accepted
[Fri Oct 16 16:33:26 2026] 127.0.0.1:58212 Closing
[Fri Oct 16 16:33:26 2026] 127.0.0.1:58224 Accepted
[Fri Oct 16 16:33:26 2026] 127.0.0.1:58224 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49074 Accepted
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49074 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49090 Accepted
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49090 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49100 Accepted
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49100 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49116 Accepted
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49116 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49128 Accepted
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49128 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49140 Accepted
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49140 Closing
[Fri Oct 16 16:34:36 2026] 127.0.0.1:49154 Accepted
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49154 Closing
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49158 Accepted
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49158 Closing
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49170 Accepted
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49170 Closing
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49184 Accepted
[Fri Oct 16 16:34:37 2026] 127.0.0.1:49184 Closing
[Fri Oct 16 16:39:51 2026] 127.0.0.1:46242 Accepted
[Fri Oct 16 16:39:51 2026] 127.0.0.1:46242 Closing
[Fri Oct 16 16:43:28 2026] 127.0.0.1:35984 Accepted
[Fri Oct 16 16:43:28 2026] 127.0.0.1:35984 Closing
[Fri Oct 16 16:43:29 2026] 127.0.0.1:35990 Accepted

Other app logs: downloads.log

Generated 2026-10-16 14:43:29 UTC · Gladex.de