Gladex Agent Logs

    domain          Domain to check (default: gladex.de)

Options:
    --help, -h      Show this help and exit 0

Exit codes:
    0   No drift (egress matches DNS A record) — logged, nothing to read
    1   Drift detected — DRIFT alert written, DNS deliberately left unchanged
    2   Invalid arguments (bad option, more than one domain)
    3   Check error (checker failed, missing or unusable) — CHECK-ERROR alert
        written with the checker's error_code (no_a_records, dns_lookup_failed,
        public_ip_unavailable, internal_error, or unknown)

Outputs:
    /data/agent-logs/ip-drift-YYYYMMDD.log      full run log (append)
    /data/agent-logs/ip-drift-ALERTS.log        DRIFT + CHECK-ERROR alerts

Cron: 0 3 * * 0 /data/repo/tools/ip-drift-cron gladex.de   (Sundays 03:00 UTC)

Test hooks (a real cron run never sets these):
    IPDRIFT_CHECK_BIN  checker binary to run (default: <tools>/ip-drift-check)
    IPDRIFT_LOG_DIR    where logs/alerts go (default: /data/agent-logs)
    IPDRIFT_TOOLS_DIR  tools directory (default: /data/repo/tools)
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "red-watch\|red_watch\|rw_failures_no_want\|redwatch" tools/system-status | head -40; echo "=== total lines ==="; wc -l tools/system-status
316:            echo "red-watch verdicts (the last verdict tools/red-watch recorded, read from"
321:            echo "                          red-watch saw a red tree. ALWAYS a warning and never"
322:            echo "                          an error: red-watch is warn-only by construction and"
327:            echo "                          by rw_failures_no_want first, because this row is the"
340:            echo "Test hooks (env), red-watch row:"
342:            echo "                          /data/agent-logs/red-watch.state - absolute, not"
409:# them, and `tools/red-watch` reads the monitor, which reads this tool — the
411:# here is what makes the 15-minute `red-watch` cron write an ALERT line on
1204:# red-watch's last recorded verdict — queue item (117).
1206:# `tools/red-watch` runs `tests/test_gladex_monitor.sh` every 15 minutes (cron
1207:# `*/15`) and writes its verdict to /data/agent-logs/red-watch.state - outside
1209:# watches. Measured before this row existed: `grep -rln "red-watch" /data
1213:# reached only a human who opened `red-watch-ALERTS.log` by hand. That is
1218:# Contract (pinned by tests/test_system_status_red_watch.sh):
1223:#   `state=red` -> warning, ALWAYS a warning, whatever the age: red-watch is
1237:# The `fails=` line is copied into the detail only after `rw_failures_no_want`
1249:# /data/agent-logs/red-watch.state - deliberately ABSOLUTE, not REPO_DIR
1254:RW_STATE_FILE="${GLADEX_REDWATCH_STATE:-/data/agent-logs/red-watch.state}"
1276:#     -> 1   (quoted from A13's want=, inside the red-watch row detail)
1286:rw_failures_no_want() {
1302:RW_DETAIL="cannot verify: no red-watch state at $RW_STATE_FILE (tools/red-watch has not run)"
1320:    RW_FAILS_CLEAN=$(printf '%s' "$(rw_failures_no_want "$RW_FAILS")" | sed 's/["\\]//g' | cut -c1-400)
1326:        *)         RW_DETAIL="cannot verify: unreadable red-watch state '$RW_STATE_SANE'" ;;
1385:            RW_DETAIL="stale green: last verdict $RW_AGE_TXT old (updated $RW_UPDATED_SANE) - cron rewrites it every 15 min, run tools/red-watch"
1396:STATUS_ITEMS+=("{\"name\":\"red-watch\",\"status\":\"$RW_STATUS\",\"detail\":\"$RW_DETAIL\"}")
=== total lines ===
2418 tools/system-status
$ cd /data/repo && ls tests/ | tr '\n' ' '; echo; echo "=== ip-drift latest log ==="; cat /data/agent-logs/ip-drift-20261004.log; echo "=== ALERTS exists? ==="; ls -la /data/agent-logs/ip-drift-ALERTS.log 2>&1
test_app_contrast.php test_app_contrast_meta.php test_app_version.php test_application_log.php test_applications_hr.php test_budget_show.sh test_changelog_api.php test_changelog_mobile.php test_chat_a11y.php test_chat_frontend.php test_chat_header_320.php test_chat_nojs.php test_cli_version.php test_commit_gate.sh test_comp_review_schedule.php test_detached_children.sh test_dig_tools_transport.sh test_disk_show.sh test_dns_verify.sh test_docs_mobile.php test_domain_availability.sh test_gladex_monitor.sh test_grid_320.php test_handbook_people.php test_heading_order.php test_hire_agent.sh test_identity_wrapper.sh test_immich_roundtrip.sh test_inbox_status.sh test_investor_heading.php test_ip_drift_check.sh test_ip_drift_cron.sh test_ip_drift_safety.php test_lan_middleware.php test_leak_figure_readers.sh test_links_tap_target.php test_mailbox_a11y.php test_main_landmark.php test_never_send_from.php test_no_dsn_reply.php test_onboarding_baseline.php test_page_lang.php test_pdns_api.sh test_php_busy_timeout.php test_promote_gate.sh test_promote_json.sh test_promote_lint_gate.sh test_qa_handover.php test_queue_source.sh test_red_watch.sh test_reduced_motion.php test_registry_coverage.sh test_regression_run.sh test_repo_lint.sh test_smtp_relay_probe.sh test_source_sync_check.sh test_start_320.php test_start_page.php test_stats_dbpath.php test_stats_gocli_path.php test_system_status_cloud.sh test_system_status_dig_single_point.sh test_system_status_dns.sh test_system_status_failed_units.sh test_system_status_go_compile.sh test_system_status_go_tests.sh test_system_status_investor_duty.sh test_system_status_mx_soa.sh test_system_status_mx_soa_transport.sh test_system_status_promote_gates.sh test_system_status_red_watch.sh test_system_status_staged.sh test_system_status_tls_expiry.sh test_system_status_unread.sh test_table_scroll_320.php test_team_page.php test_team_page_privacy.php test_template_sync.sh test_templates_contrast.php test_templates_gallery.php test_tls_check.sh test_trust_a11y.php test_trust_form.php test_trust_mobile.php test_ts_ordering.php test_verify_landing.sh test_version_check.sh test_vhost_ui_smoke.sh test_webmail_session_routing.php 
=== ip-drift latest log ===
[2026-10-04T01:00:01Z] Starting IP drift check for gladex.de
{
  "domain": "gladex.de",
  "timestamp": "2026-10-04T01:00:01Z",
  "public_ip": "77.90.15.49",
  "dns_records": [
    "77.90.15.49"
  ],
  "dns_status": "ok",
  "rejected": [],
  "drift": false,
  "error": null,
  "error_code": null,
  "exit_code": 0
}
[2026-10-04T01:00:01Z] OK: No drift (egress matches DNS A record)
[2026-10-04T01:00:01Z] Drift check complete
=== ALERTS exists? ===
ls: cannot access '/data/agent-logs/ip-drift-ALERTS.log': No such file or directory
> build · mimo-v2.6-flash-free
$ cd /data/repo && grep -n "ipchan_scan\|^# \|section 7\|C1\|C2\|C3\|C4\|C5\|C6\|C7\|C8\|REAL_ALERT" tests/test_ip_drift_cron.sh | sed -n '1,120p'
2:# Behavioral regression test for tools/ip-drift-cron — the weekly ALERT path.
4:# Why this suite exists (2026-09-24): the cron had a 16-assertion STATIC safety
5:# suite (test_ip_drift_safety.php: "never writes DNS", "mentions an alert file")
6:# but zero BEHAVIOURAL coverage of what actually lands in
7:# /data/agent-logs/ip-drift-ALERTS.log — the only channel an investor/operator
8:# ever reads. The gap it exists for:
10:#   1. A CHECK ERROR NEVER ALERTED. ip-drift-check was hardened to separate
11:#      `no_a_records` (the A record vanished) from `dns_lookup_failed` (dig /
12:#      the resolver is dead) from `public_ip_unavailable`, but the cron logged
13:#      those to the dated file and exited 3 WITHOUT touching ALERTS.log. The
14:#      failure modes that are WORSE than drift were the only ones nobody would
15:#      ever see — a silent monitor is worse than no monitor.
16:#   2. EVERYTHING THAT WAS NOT 0 OR 1 WAS THE SAME STRING: a missing checker
17:#      binary (127), a checker arg bug (2) and an unknown code (99) all printed
18:#      one line and exited 3, with no hint which happened.
19:#   3. NO HOOKS: LOG_DIR and the checker path were hardcoded, so the cron could
20:#      not be exercised without writing to the real /data/agent-logs and
21:#      running the real network check. (That is also why it had no suite.)
23:# Hermetic by construction: a scenario-driven `ip-drift-check` STUB stands in
24:# for the real tool (via IPDRIFT_CHECK_BIN) and every log/alert lands in a
25:# sandbox (via IPDRIFT_LOG_DIR) — no network is touched, no real alert file is
26:# written, no real service sees us. Legacy replay: the pre-fix cron has no
27:# hooks, so `IPDRIFT_CRON_LEGACY=1` rewrites ONLY its two hardcoded path
28:# literals into this sandbox (harness redirection, no logic touched) — without
29:# it the suite REFUSES to run (exit 2) rather than append to the real alert log.
31:# Run: bash tests/test_ip_drift_cron.sh
32:#      IPDRIFT_CRON=<blob> IPDRIFT_CRON_LEGACY=1 bash tests/test_ip_drift_cron.sh  # pre-fix replay
33:# Mutations: bash tests/test_ip_drift_cron.sh --mutations
34:# Exit codes: 0 = all pass, 1 = at least one failure, 2 = refused to run.
41:# Set by --mutations: run the mutation battery instead of the suite.
64:# ------------------------------------------------- legacy gate + redirect ----
65:# The pre-fix cron hardcodes both paths. Refuse to run against it (it would
66:# append to the REAL alert log), or — with the explicit LEGACY flag — rewrite
67:# exactly those two literals into this sandbox.
90:# ------------------------------------------------------ scenario stub --------
91:# Stands in for tools/ip-drift-check: emits the documented JSON and exit code
92:# for the requested scenario, records argv so we can assert the hand-off.
115:# The real escalation file must survive this suite untouched — snap it now.
116:REAL_ALERTS="/data/agent-logs/ip-drift-ALERTS.log"
117:REAL_ALERT_N="$(wc -l < "$REAL_ALERTS" 2>/dev/null | tr -d ' ')"
118:REAL_ALERT_N="${REAL_ALERT_N:-0}"
120:# ------------------------------------------------------------- run helper ----
138:# dated run log only — never the ALERTS file (it matches ip-drift-*.log too,
139:# and the escalation line it holds legitimately mentions "ip-drift-ALERTS.log")
142:# =============================================================================
143:# 1. Argument / help contract
144:# =============================================================================
164:# =============================================================================
165:# 2. Happy path: exit 0 — no alert, log says OK
166:# =============================================================================
176:# default domain flows through to the checker
183:# =============================================================================
184:# 3. Drift: exit 1 — DRIFT alert with classification, DNS untouched
185:# =============================================================================
207:# =============================================================================
208:# 4. CHECK ERRORS must alert too — the defect this suite exists for
209:#    (pre-fix: these all log to the dated file only and stay silent)
210:# =============================================================================
230:# the human-meaningful detail rides along too
235:# malformed / empty / unexpected checker output never stays silent
257:# an error must never be mistaken for drift, and vice versa
263:# alerts are append-only across runs
274:# =============================================================================
275:# 5. Hooks, sandboxing, static guards
276:# =============================================================================
290:# no DNS write anywhere in executable code (full-line comments stripped: the
291:# cron deliberately documents the OLD auto-`set-a` behaviour in its header).
292:# A read-only MENTION is fine — the error path names `pdns-api.py records` as
293:# the human's next step — but no write subcommand may be invoked.
307:# the error path must be reachable without ever calling a write tool either
311:# REGISTRY contract keeps pace with the behaviour
322:# =============================================================================
323:# 6. One guarded live case (real checker, sandboxed log dir)
324:# =============================================================================
346:# Whatever happened above, the REAL escalation file must be byte-for-byte
347:# untouched: every run in this suite is sandboxed.
348:NOW_N="$(cat "$REAL_ALERTS" 2>/dev/null | wc -l | tr -d ' ')"
349:assert_eq "the real /data/agent-logs/ip-drift-ALERTS.log was never written" "$REAL_ALERT_N" "$NOW_N"
351:# =============================================================================
352:# 7. Channel sweep — the [0.4.205] reader generalised to THIS writer's two
353:#    channels (ip-drift-ALERTS.log + ip-drift-YYYYMMDD.log).
355:#    WHY (queue item `[0.4.205]`, candidate (c)): `red-watch`'s four full-line
356:#    channels got a reader (G28–G34) because a hand-run sweep had found nothing
357:#    reading them. That sweep was a measurement taken once — a memory. The same
358:#    question for this box's OTHER scheduled writer had never been measured, so
359:#    this run measured it and then wrote the measurement down as assertions:
361:#      1. `want=` counts = 0 in `tools/ip-drift-cron`, in `tools/ip-drift-check`
362:#         and in all four live dated logs — this writer carries no expectation
363:#         for a grep to satisfy itself with.
364:#      2. The only non-comment code anywhere in `tests/` naming a live channel
365:#         is THIS suite's write-guard: the `REAL_ALERTS=` snapshot (line 116)
366:#         and the before/after delta (the "never written" assertion) — nobody
367:#         derives a green/red verdict from a live ip-drift channel.
368:#      3. `tools/` side is set-equality on exactly `ip-drift-cron`, the writer:
369:#         `system-status` and `red-watch` do not consume ip-drift output at all,
370:#         so there was no reader to sanction and no `want=` to strip.
372:#    So unlike `[0.4.204]`'s channels there was nothing to FIX here — the
373:#    correct generalisation is a reader that KEEPS it that way: the next suite
374:#    that opens `ip-drift-ALERTS.log` reddens section 7 instead of quietly
375:#    reading green off a drifting box.
377:#    Rules (same shape as test_red_watch.sh G28–G34): comments skipped and
378:#    single-quoted segments stripped FIRST, because a single-quoted path in this
379:#    tree is a NEEDLE grepped against another file's source, never an open;
380:#    `*.md` excluded by extension, not by exception; the only CODE allowed to
381:#    name a live channel is the writer plus this suite's write-guard, and even
382:#    inside this suite a live-channel line must go through `REAL_ALERT*`.
384:#    Anti-vacuity is the five-twin plant below: an open, a single-quoted needle,
385:#    a comment, a guarded read and an unguarded read. If the scanner ever stops
386:#    reading (python3 gone, a token changed, a rule rewritten) the real-tree
387:#    assertions catch the empty set two ways — the two set-equality expectations
388:#    are NON-EMPTY ("test_ip_drift_cron.sh", "ip-drift-cron") — and C7/C8 pin the
389:#    classifier itself, so the suite cannot pass by looking at nothing.
390:# -----------------------------------------------------------------------------
391:ipchan_scan() { # <dir> <files|guard|names> — one hit per line, "" if clean
400:GUARD = "REAL_ALERT"
429:# C1 — the real tree, tests side: set equality on FILE NAMES, not a zero, so a
430:# second suite opening the channel is named rather than merely counted.
431:assert_eq "C1 tests: the only suite naming a live ip-drift channel is this write-guard suite" \
433:    "$(ipchan_scan "$REPO/tests" files | cut -d: -f1 | sort -u | grep .)"
434:# C2 — and inside this suite every such line must ride the write-guard variable:
> build · mimo-v2.6-flash-free
→ Read repo/tests/test_ip_drift_cron.sh [offset=390, limit=180]
> build · mimo-v2.6-flash-free

Select another run log from the list above. Only files matching run-YYYYMMDD-HHMMSS-N.log are readable.

[Sun Oct  4 23:33:43 2026] 127.0.0.1:48910 Accepted
[Sun Oct  4 23:33:43 2026] 127.0.0.1:48910 Closing
[Sun Oct  4 23:33:43 2026] 127.0.0.1:48918 Accepted
[Sun Oct  4 23:33:43 2026] 127.0.0.1:48918 Closing
[Sun Oct  4 23:33:43 2026] 127.0.0.1:48930 Accepted
[Sun Oct  4 23:33:43 2026] 127.0.0.1:48930 Closing
[Sun Oct  4 23:33:43 2026] 127.0.0.1:48940 Accepted
[Sun Oct  4 23:33:44 2026] 127.0.0.1:48940 Closing
[Sun Oct  4 23:33:44 2026] 127.0.0.1:48942 Accepted
[Sun Oct  4 23:33:44 2026] 127.0.0.1:48942 Closing
[Sun Oct  4 23:34:01 2026] 127.0.0.1:42418 Accepted
[Sun Oct  4 23:34:01 2026] 127.0.0.1:42418 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54164 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54164 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54174 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54174 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54176 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54176 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54192 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54192 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54196 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54196 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54212 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54212 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54216 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54216 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54230 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54230 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54236 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54236 Closing
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54252 Accepted
[Sun Oct  4 23:34:07 2026] 127.0.0.1:54252 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44584 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44584 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44596 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44596 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44610 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44610 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44624 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44624 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44636 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44636 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44642 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44642 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44658 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44658 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44660 Accepted
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44660 Closing
[Sun Oct  4 23:34:57 2026] 127.0.0.1:44668 Accepted
[Sun Oct  4 23:34:58 2026] 127.0.0.1:44668 Closing
[Sun Oct  4 23:34:58 2026] 127.0.0.1:44670 Accepted
[Sun Oct  4 23:34:58 2026] 127.0.0.1:44670 Closing
[Sun Oct  4 23:35:08 2026] 127.0.0.1:36514 Accepted
[Sun Oct  4 23:35:08 2026] 127.0.0.1:36514 Closed without sending a request; it was probably just an unused speculative preconnection
[Sun Oct  4 23:35:08 2026] 127.0.0.1:36514 Closing
[Sun Oct  4 23:35:09 2026] 127.0.0.1:36530 Accepted
[Sun Oct  4 23:35:09 2026] 127.0.0.1:36530 Closed without sending a request; it was probably just an unused speculative preconnection
[Sun Oct  4 23:35:09 2026] 127.0.0.1:36530 Closing
[Sun Oct  4 23:35:36 2026] 127.0.0.1:56790 Accepted