A self-improvement loop silently failed 2,450 times in a row before anyone noticed

The self-improvement loop in the Hermes agents logged 2,450 identical failures over 49 days without triggering any alert.

The production stack

Hermes autonomous agents run on owned hardware through the OpenClaw runtime. A local LLM stack on an NVIDIA GB10 handles inference. The Nerve dashboard surfaces live agent state and a cross-server build log. A twice-daily automated audit process scans logs from every agent.

What the log showed

The file /root/.hermes/plugins/self-evolution/evolution_log.jsonl recorded 2,456 runs between 2026-04-22 and 2026-06-10. 2,450 runs, or 99.8 percent, ended with the same error: ModuleNotFoundError: No module named 'dspy'. The job was scheduled on cron and reported only that it had executed. No success metric existed, so zero successes looked identical to normal operation from the outside.

Root cause

The missing module error had already been fixed on disk. A long-running monitor process continued to execute the older code loaded into memory at startup. Every subsequent cron invocation reused that stale process image. On-disk changes never took effect until the process itself was restarted.

How it was caught

The twice-daily audit agent compared failure counts against success counts across all logged runs. It flagged the sequence of 2,450 failures and zero successes as anomalous. On 2026-06-10 at 23:18 the audit triggered a restart of the monitor. Run 2,456, the first after the restart, completed without the import error.

Changes to failure handling

Previously a fallback agent would quietly complete a failed task, hiding the underlying defect. The escalation path was rewritten so the second agent must first diagnose why the initial agent failed, apply a fix, and only then return the original task. The same blind spot cannot recur without an explicit record in the audit log.

Implications for other builders

Long-running processes plus on-disk code updates create a reload gap that standard cron or container restarts do not close. Observability limited to job completion status will miss persistent silent failure. An independent audit agent that treats zero successes as a first-class signal is required to surface the pattern. The 49-day window demonstrates that production multi-agent systems need explicit memory-versus-disk checks rather than relying on human review of logs.