Testing methodology

Recovery automation must be tested for the actions it takes and the actions it refuses to take.

RecoverNode testing focuses on controlled failure, evidence classification, bounded recovery, reversibility, and verification after every material action.

1. Establish a known baseline

Testing begins with the environment healthy. The observer, protected hosts, DNS records, TLS certificates, HTTP routes, private service ports, systemd services, Nginx configuration, application processes, alert delivery, and watchdog support are inspected before recovery actions are enabled.

A baseline prevents a pre-existing configuration problem from being mistaken for a recovery-system failure.

2. Test observer failures separately

Public HTTP, DNS, TLS, latency, content, TCP, and private-network checks are tested independently. DNS Path Integrity compares named internal-reference and client-facing resolvers, UDP and TCP answers, expected internal and public addresses, and remaining TTLs so stale cached public answers and split-horizon leakage can be classified separately from healthy resolver behavior. Public-integrity testing can evaluate expected delegation, configured authoritative nameservers, multiple public recursive resolvers, expected public IPv4 addresses, and semantic page identity. A successful HTTP status is not sufficient when the returned content belongs to a parking page or another site. These observations remain diagnostic and do not grant remote reboot authority.

This confirms that loss of visibility is not treated as proof that a protected server requires a reboot. DNS, public-content, and network-consistency evidence remains diagnostic. Public recursive resolvers provide independent DNS evidence, but HTTP checks performed from the same observer are not a genuine off-network Internet witness. The assessment can also distinguish effective merged Netplan state from overridden source declarations without automatically rewriting Netplan, DHCP, systemd-resolved, routes, or interface addresses.

3. Test alert correlation and certificate cadence

Multi-site observer and DNS failures are tested together to confirm that matching infrastructure symptoms are consolidated rather than producing a separate alert for every affected hostname. Single-site failures must remain independently visible, and unrelated failures must not be swallowed by correlation.

TLS expiration testing verifies that warning conditions repeat no more than once every seven days, critical and expired conditions repeat no more than once every 24 hours, severity escalation does not create a false recovery, and genuine renewal or clearance produces one recovery notification.

4. Test local service failures

Nginx and application failures are simulated separately. The local guard must identify which service is unhealthy, validate the applicable configuration, restart only the relevant service, and retest it before reporting recovery.

An application failure should not automatically become an Nginx restart. An Nginx configuration failure should not enter an endless restart loop.

5. Test ambiguous evidence

Network-path interruptions, conflicting checks, unreachable observer conditions, invalid configurations, and incomplete evidence are used to confirm fail-open behavior.

In these conditions, RecoverNode should alert, preserve the running state where possible, and withhold disruptive action that cannot be justified.

6. Exhaust the recovery budget

Repeated failures are used to verify consecutive-failure thresholds, exponential backoff, cooldown periods, restart limits, reboot limits, persistent counters, and automatic lockout.

Recovery state must survive service restarts. A restarted guard should not forget that its safe action budget was already consumed.

7. Verify watchdog behavior independently

Where supported and authorized, the hardware watchdog is tested as a separate protection layer. It is intended for a complete operating system or userspace freeze, not as a substitute for service-specific recovery.

Watchdog configuration, timeout behavior, systemd integration, and reboot recovery are verified before production activation.

8. Verify Raspberry Pi power-health reporting

On supported Raspberry Pi hosts, testing confirms that current and historical undervoltage or throttling flags are decoded correctly, written to persistent state, and reported through rate-limited alerts.

Testing does not intentionally undervoltage production hardware. A clean cold-start acceptance test uses an approved power supply, confirmsthrottled=0x0, reviews the current boot journal, and verifies that the power-health timer resumed after startup.

RecoverNode must not attempt to reboot a host merely because its power source may be unstable. It also must not claim that a watchdog can recover a system that never reached the operating system.

9. Verify logs and alerts

Health failures, restart attempts, successful recoveries, cooldown decisions, budget exhaustion, lockout, reconciliation failures, payment events, and backup failures must produce the expected journal entries, state records, or administrative notifications.

A recovery system that acts without leaving evidence is not operationally supportable.

10. Verify rollback and uninstall

Installation is not considered complete until status, rollback, and uninstall procedures have been tested. Generated configuration, systemd units, timers, state directories, permissions, and package contents are reviewed for reversibility.

11. Verify commercial operations

The production service also validates checkout fulfillment, subscription status, canceled-access enforcement, billing portal access, webhook idempotency, payment records, refund and dispute records, administrative alerts, and daily payment reconciliation.

12. Verify backup and restore

Local archives are created, inspected, retained, and copied to a separate host through a restricted backup account. Restore drills validate JSON records, downloadable packages, environment configuration, service units, scripts, and application package files.

Backup failure is deliberately simulated to confirm that the local archive remains intact, the remote-copy failure is reported, and a later successful run restores off-host protection.

Launch-readiness standard

A RecoverNode production deployment should not be considered ready merely because the software builds or a service starts. The complete environment must pass route checks, access controls, configuration checks, record validation, reconciliation checks, backup checks, timer checks, and recovery tests without unresolved failures.