Gladex Agent Logs
Agent run logs & app logs · env: prod · LAN-only investor surface
Overview
| Run logs | 1187 files, 71.1 MB |
| Latest run log | run-20261006-041829-721.log |
| Log directory | /data/agent-logs |
| App log directory | /opt/startup/prod/logs |
Run logs (newest first, last 50)
| File | Size | Modified (UTC) |
|---|---|---|
| run-20261006-041829-721.log | 231 KB | 2026-10-06 02:28:56 |
| run-20261006-031229-720.log | 292 KB | 2026-10-06 02:08:21 |
| run-20261006-022028-719.log | 271 KB | 2026-10-06 01:02:21 |
| run-20261006-013213-718.log | 276 KB | 2026-10-06 00:10:20 |
| run-20261006-004412-717.log | 192 KB | 2026-10-05 23:22:05 |
| run-20261005-234618-716.log | 235 KB | 2026-10-05 22:34:04 |
| run-20261005-225616-715.log | 296 KB | 2026-10-05 21:36:11 |
| run-20261005-215648-714.log | 220 KB | 2026-10-05 20:46:08 |
| run-20261005-205855-713.log | 188 KB | 2026-10-05 19:46:40 |
| run-20261005-201246-712.log | 321 KB | 2026-10-05 18:48:47 |
| run-20261005-190629-711.log | 213 KB | 2026-10-05 18:02:38 |
| run-20261005-174000-710.log | 397 KB | 2026-10-05 16:56:22 |
| run-20261005-164047-709.log | 254 KB | 2026-10-05 15:29:53 |
| run-20261005-160934-708.log | 139 KB | 2026-10-05 14:30:37 |
| run-20261005-155925-707.log | 153 B | 2026-10-05 13:59:26 |
| run-20261005-154917-706.log | 190 B | 2026-10-05 13:49:18 |
| run-20261005-153909-705.log | 153 B | 2026-10-05 13:39:09 |
| run-20261005-152900-704.log | 153 B | 2026-10-05 13:29:01 |
| run-20261005-142051-703.log | 222 KB | 2026-10-05 13:18:53 |
| run-20261005-130731-702.log | 227 KB | 2026-10-05 12:10:44 |
| run-20261005-120453-701.log | 163 KB | 2026-10-05 10:57:24 |
| run-20261005-094142-700.log | 440 KB | 2026-10-05 09:54:46 |
| run-20261005-090130-699.log | 165 KB | 2026-10-05 07:31:34 |
| run-20261005-071341-698.log | 435 KB | 2026-10-05 06:51:22 |
| run-20261005-061647-697.log | 196 KB | 2026-10-05 05:03:34 |
| run-20261005-052959-696.log | 226 KB | 2026-10-05 04:06:39 |
| run-20261005-044026-695.log | 177 KB | 2026-10-05 03:19:52 |
| run-20261005-034927-694.log | 254 KB | 2026-10-05 02:30:18 |
| run-20261005-030059-693.log | 312 KB | 2026-10-05 01:39:20 |
| run-20261004-233305-692.log | 1010 KB | 2026-10-05 00:50:51 |
| run-20261004-230052-691.log | 130 KB | 2026-10-04 21:22:57 |
| run-20261004-221607-690.log | 258 KB | 2026-10-04 20:50:44 |
| run-20261004-210204-689.log | 310 KB | 2026-10-04 20:05:59 |
| run-20261004-184239-688.log | 505 KB | 2026-10-04 18:51:56 |
| run-20261004-172538-687.log | 486 KB | 2026-10-04 16:32:32 |
| run-20261004-165539-686.log | 142 KB | 2026-10-04 15:15:29 |
| run-20261004-161338-685.log | 176 KB | 2026-10-04 14:45:31 |
| run-20261004-160330-684.log | 153 B | 2026-10-04 14:03:31 |
| run-20261004-155322-683.log | 153 B | 2026-10-04 13:53:23 |
| run-20261004-154314-682.log | 190 B | 2026-10-04 13:43:15 |
| run-20261004-153306-681.log | 153 B | 2026-10-04 13:33:07 |
| run-20261004-152258-680.log | 153 B | 2026-10-04 13:22:58 |
| run-20261004-151250-679.log | 190 B | 2026-10-04 13:12:50 |
| run-20261004-150241-678.log | 153 B | 2026-10-04 13:02:42 |
| run-20261004-145233-677.log | 153 B | 2026-10-04 12:52:34 |
| run-20261004-144225-676.log | 190 B | 2026-10-04 12:42:26 |
| run-20261004-143217-675.log | 153 B | 2026-10-04 12:32:17 |
| run-20261004-142209-674.log | 153 B | 2026-10-04 12:22:09 |
| run-20261004-141201-673.log | 153 B | 2026-10-04 12:12:01 |
| run-20261004-140152-672.log | 153 B | 2026-10-04 12:01:53 |
Tail โ run-20261006-041829-721.log (last 200 lines)
' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
' *)\n'
' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;,
' dns_lookup_failed)\n'
' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;\n'
' public_ip_unavailable)\n'
' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
' *)\n'
' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;)
"M28": (' no_a_records)\n'
' DIAGNOSIS="the DNS A record for ${DOMAIN} is GONE (query completed, no answer)."\n'
' ACTION="inspect the zone with \'pdns-api.py records\'; do NOT rewrite DNS from here." ;;\n'
' dns_lookup_failed)\n'
' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;\n'
' public_ip_unavailable)\n'
' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
' *)\n'
' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;,
' no_a_records)\n'
' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;\n'
' dns_lookup_failed)\n'
' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;\n'
' public_ip_unavailable)\n'
' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
' *)\n'
' DIAGNOSIS="the DNS A record for ${DOMAIN} is GONE (query completed, no answer)."\n'
' ACTION="inspect the zone with \'pdns-api.py records\'; do NOT rewrite DNS from here." ;;)
> build ยท mimo-v2.6-flash-free
[0mโ [0mEdit repo/tests/test_ip_drift_cron.sh
Index: repo/tests/test_ip_drift_cron.sh
===================================================================
--- repo/tests/test_ip_drift_cron.sh
+++ repo/tests/test_ip_drift_cron.sh
@@ -1857,8 +1857,102 @@
' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;',
' *)\n'
' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
' ACTION="check egress; no drift verdict can be trusted until this clears." ;;'),
+ # M27 โ the default arm's SECOND exchange (queue item (x), 2026-10-06):
+ # after M25 the `*)` arm has traded with exactly ONE of the three named
+ # arms (`public_ip_unavailable`), so `default<->dns` and `default<->no_a`
+ # are the block's remaining untraded pairs โ this is the first of those
+ # two. Both runs keep their own LABEL, still alert and still exit 3, so
+ # the output's shape is unchanged and only a reader that pins each arm to
+ # its own sentence reddens. The needle is the CONTIGUOUS three-arm span
+ # `dns_lookup_failed` + `public_ip_unavailable` + `*)` with only the two
+ # OUTER arms' `DIAGNOSIS=`/`ACTION=` pairs swapped and
+ # `public_ip_unavailable` in the middle left byte-identical โ that
+ # untouched middle arm is what keeps M27 from collapsing into M18 (moves
+ # `dns`, the span's upper arm) or into M25 (moves `ip`, the same middle
+ # arm M27 must not touch). `${EXIT_CODE}` interpolation is carried across
+ # the move, so the receiving arm still prints the checker's real exit
+ # (counted once in the probe). Pinned to `err_dns โ does not borrow the
+ # default arm's action` (a pin no mutant used before this one), whose
+ # siblings `err_dns โ named arm action orders the lookup chain re-checked
+ # first` and `err_dns โ named arm diagnosis says the lookup never
+ # completed` (M18's pin) redden with it, while the WHOLE `err_ip` and
+ # `err_no_a` blocks stay GREEN โ the middle-arm-untouched proof, read
+ # green rather than merely absent. Probe measured **263 passed / 17
+ # failed** against the control **280 / 0**: 4 per scenario x 3 = 12 in
+ # the three-scenario loop, + the `:362` interpolation assertion, + the
+ # `named branch (dns_lookup_failed)` guard, + 3 in the `err_dns` block.
+ # The queue's own components summed to 18 (it wrote "4 on the named
+ # side") while predicting "17-shaped"; measured is **17**, the named-side
+ # component being **3** โ counted, not carried.
+ "M27": (' dns_lookup_failed)\n'
+ ' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
+ ' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;\n'
+ ' public_ip_unavailable)\n'
+ ' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
+ ' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
+ ' *)\n'
+ ' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
+ ' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;',
+ ' dns_lookup_failed)\n'
+ ' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
+ ' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;\n'
+ ' public_ip_unavailable)\n'
+ ' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
+ ' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
+ ' *)\n'
+ ' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
+ ' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;'),
+ # M28 โ the LAST `case`-block exchange (queue item (x), 2026-10-06): the
+ # final untraded arm pair, `no_a_records` <-> `*)`, over the FULL
+ # four-arm span with `dns_lookup_failed` and `public_ip_unavailable` both
+ # left byte-identical โ those two untouched arms are what keep M28 from
+ # collapsing into M20 (moves `dns`) or M21 (moves `ip`), each of which
+ # trades with the same upper arm from the other side. After it every one
+ # of C(4,2)=6 arm pairs has exchanged, so the exchange family is closed:
+ # the only `case` work left is the three single-half DIAGNOSIS borrows
+ # queued for whoever picks this up next. Same shape as M27 โ arm LABELS
+ # in place, only the outer two arms' `DIAGNOSIS=`/`ACTION=` pairs swapped,
+ # `${EXIT_CODE}` carried across so the receiving arm still prints the
+ # checker's real exit (counted once in the probe). Pinned to `err_no_a โ
+ # does not borrow the default arm's action` (a pin no mutant used before
+ # this one), whose siblings `err_no_a โ named arm action points at the
+ # zone, never at a rewrite` and `err_no_a โ named arm diagnosis says the
+ # A record is gone` (M8's pin) redden with it, while `err_no_a โ does not
+ # borrow the dead-lookup diagnosis` (M20's pin) stays GREEN โ no_a's new
+ # sentences are the default arm's, so the dead-lookup sentence is still
+ # absent, and that green is what distinguishes an *exchange* from a
+ # borrow on this arm โ and while the WHOLE `err_dns` and `err_ip` blocks
+ # stay GREEN (the two untouched arms). Probe measured **263 passed / 17
+ # failed** against the control **280 / 0**: the same 12 + 1 + 1 + 3
+ # shape, with the loop's two unexpected needles now
+ # `does not diagnose a vanished A record` and `does not send the operator
+ # to the zone`.
+ "M28": (' no_a_records)\n'
+ ' DIAGNOSIS="the DNS A record for ${DOMAIN} is GONE (query completed, no answer)."\n'
+ ' ACTION="inspect the zone with \'pdns-api.py records\'; do NOT rewrite DNS from here." ;;\n'
+ ' dns_lookup_failed)\n'
+ ' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
+ ' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;\n'
+ ' public_ip_unavailable)\n'
+ ' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
+ ' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
+ ' *)\n'
+ ' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
+ ' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;',
+ ' no_a_records)\n'
+ ' DIAGNOSIS="checker failed without a classified cause (exit ${EXIT_CODE})."\n'
+ ' ACTION="run \'ip-drift-check --format json\' by hand and read its output." ;;\n'
+ ' dns_lookup_failed)\n'
+ ' DIAGNOSIS="the DNS lookup never completed (dig missing, unreachable or timed out)."\n'
+ ' ACTION="check dig + resolver + WireGuard tunnel before trusting ANY verdict." ;;\n'
+ ' public_ip_unavailable)\n'
+ ' DIAGNOSIS="no public IPv4 could be determined (every echo service rejected)."\n'
+ ' ACTION="check egress; no drift verdict can be trusted until this clears." ;;\n'
+ ' *)\n'
+ ' DIAGNOSIS="the DNS A record for ${DOMAIN} is GONE (query completed, no answer)."\n'
+ ' ACTION="inspect the zone with \'pdns-api.py records\'; do NOT rewrite DNS from here." ;;'),
}
needle, repl = M[os.environ["MUT_ID"]]
path = os.environ["MUT_TARGET"]
src = open(path).read()
> build ยท mimo-v2.6-flash-free
[0mโ [0mEdit repo/tests/test_ip_drift_cron.sh
Index: repo/tests/test_ip_drift_cron.sh
===================================================================
--- repo/tests/test_ip_drift_cron.sh
+++ repo/tests/test_ip_drift_cron.sh
@@ -2008,8 +2008,10 @@
run_mutation M23 'err_ip โ named arm diagnosis says no public IPv4 could be found'
run_mutation M24 'err_no_a โ does not borrow the dead-lookup action'
run_mutation M25 "err_int โ default arm action points at the checker's own output"
run_mutation M26 'err_int โ does not declare the egress untrustworthy'
+ run_mutation M27 "err_dns โ does not borrow the default arm's action"
+ run_mutation M28 "err_no_a โ does not borrow the default arm's action"
fi
# =============================================================================
echo ""
> build ยท mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && bash -n tests/test_ip_drift_cron.sh && echo "bash -n OK"; grep -c '^ run_mutation M' tests/test_ip_drift_cron.sh; echo "=== plain ==="; bash tests/test_ip_drift_cron.sh 2>&1 | tail -2; echo "rc=${PIPESTATUS[0]}"
bash -n OK
28
=== plain ===
=== Results: 283 passed, 0 failed ===
rc=0
> build ยท mimo-v2.6-flash-free
[0m$ [0mcd /data/repo && bash tests/test_ip_drift_cron.sh --mutations > /tmp/opencode/probe40/mut.out 2>&1; echo "rc=$?"; tail -2 /tmp/opencode/probe40/mut.out; echo "=== counts ==="; grep -c 'applied (its needle occurs exactly once)' /tmp/opencode/probe40/mut.out; grep -c 'is syntactically valid' /tmp/opencode/probe40/mut.out; grep -c 'suite goes red' /tmp/opencode/probe40/mut.out; grep -c 'wrong verdict lands on the intended assertion' /tmp/opencode/probe40/mut.out; grep -c 'suite stayed GREEN' /tmp/opencode/probe40/mut.out; grep -c 'precondition: needle occurs' /tmp/opencode/probe40/mut.out; grep -c 'does not parse' /tmp/opencode/probe40/mut.out; echo "=== M27/M28 lines ==="; grep 'M27 \|M28 \|M27 โ\|M28 โ' /tmp/opencode/probe40/mut.out
rc=0
=== Results: 396 passed, 0 failed ===
=== counts ===
28
28
28
28
0
0
0
=== M27/M28 lines ===
ok - M27 applied (its needle occurs exactly once)
ok - M27 is syntactically valid (red comes from behaviour, not a parse error)
ok - M27 โ suite goes red (rc=1), the mutant did not survive
ok - M27 โ wrong verdict lands on the intended assertion: 'err_dns โ does not borrow the default arm's action'
ok - M28 applied (its needle occurs exactly once)
ok - M28 is syntactically valid (red comes from behaviour, not a parse error)
ok - M28 โ suite goes red (rc=1), the mutant did not survive
ok - M28 โ wrong verdict lands on the intended assertion: 'err_no_a โ does not borrow the default arm's action'
> build ยท mimo-v2.6-flash-free
Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.
App log tail โ prod-8001.log (last 60 lines)
[Tue Oct 6 04:15:09 2026] 127.0.0.1:35786 Accepted [Tue Oct 6 04:15:09 2026] 127.0.0.1:35786 Closing [Tue Oct 6 04:15:09 2026] 127.0.0.1:35788 Accepted [Tue Oct 6 04:15:09 2026] 127.0.0.1:35788 Closing [Tue Oct 6 04:15:09 2026] 127.0.0.1:35796 Accepted [Tue Oct 6 04:15:09 2026] 127.0.0.1:35796 Closing [Tue Oct 6 04:15:09 2026] 127.0.0.1:35806 Accepted [Tue Oct 6 04:15:09 2026] 127.0.0.1:35806 Closing [Tue Oct 6 04:15:09 2026] 127.0.0.1:35820 Accepted [Tue Oct 6 04:15:09 2026] 127.0.0.1:35820 Closing [Tue Oct 6 04:15:09 2026] 127.0.0.1:35826 Accepted [Tue Oct 6 04:15:10 2026] 127.0.0.1:35826 Closing [Tue Oct 6 04:15:10 2026] 127.0.0.1:35834 Accepted [Tue Oct 6 04:15:10 2026] 127.0.0.1:35834 Closing [Tue Oct 6 04:15:10 2026] 127.0.0.1:35844 Accepted [Tue Oct 6 04:15:10 2026] 127.0.0.1:35844 Closing [Tue Oct 6 04:15:10 2026] 127.0.0.1:35860 Accepted [Tue Oct 6 04:15:10 2026] 127.0.0.1:35860 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39306 Accepted [Tue Oct 6 04:15:57 2026] 127.0.0.1:39306 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39320 Accepted [Tue Oct 6 04:15:57 2026] 127.0.0.1:39320 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39326 Accepted [Tue Oct 6 04:15:57 2026] 127.0.0.1:39326 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39330 Accepted [Tue Oct 6 04:15:57 2026] 127.0.0.1:39330 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39342 Accepted [Tue Oct 6 04:15:57 2026] 127.0.0.1:39342 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39356 Accepted [Tue Oct 6 04:15:57 2026] 127.0.0.1:39356 Closing [Tue Oct 6 04:15:57 2026] 127.0.0.1:39362 Accepted [Tue Oct 6 04:15:58 2026] 127.0.0.1:39362 Closing [Tue Oct 6 04:15:58 2026] 127.0.0.1:39368 Accepted [Tue Oct 6 04:15:58 2026] 127.0.0.1:39368 Closing [Tue Oct 6 04:15:58 2026] 127.0.0.1:39382 Accepted [Tue Oct 6 04:15:58 2026] 127.0.0.1:39382 Closing [Tue Oct 6 04:15:58 2026] 127.0.0.1:39394 Accepted [Tue Oct 6 04:15:58 2026] 127.0.0.1:39394 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48882 Accepted [Tue Oct 6 04:30:10 2026] 127.0.0.1:48882 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48888 Accepted [Tue Oct 6 04:30:10 2026] 127.0.0.1:48888 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48902 Accepted [Tue Oct 6 04:30:10 2026] 127.0.0.1:48902 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48912 Accepted [Tue Oct 6 04:30:10 2026] 127.0.0.1:48912 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48914 Accepted [Tue Oct 6 04:30:10 2026] 127.0.0.1:48914 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48926 Accepted [Tue Oct 6 04:30:10 2026] 127.0.0.1:48926 Closing [Tue Oct 6 04:30:10 2026] 127.0.0.1:48936 Accepted [Tue Oct 6 04:30:11 2026] 127.0.0.1:48936 Closing [Tue Oct 6 04:30:11 2026] 127.0.0.1:48948 Accepted [Tue Oct 6 04:30:11 2026] 127.0.0.1:48948 Closing [Tue Oct 6 04:30:11 2026] 127.0.0.1:48956 Accepted [Tue Oct 6 04:30:11 2026] 127.0.0.1:48956 Closing [Tue Oct 6 04:30:11 2026] 127.0.0.1:48966 Accepted [Tue Oct 6 04:30:11 2026] 127.0.0.1:48966 Closing [Tue Oct 6 04:30:15 2026] 127.0.0.1:51794 Accepted
Generated 2026-10-06 02:30:16 UTC · Gladex.de