Testing methodology

Recovery automation must be tested for the actions it takes and the actions it refuses to take.

RecoverNode testing focuses on controlled failure, evidence classification, bounded recovery, reversibility, and verification after every material action.

1. Establish a known baseline

Testing begins with the environment healthy. The observer, protected hosts, DNS records, TLS certificates, HTTP routes, private service ports, systemd services, Nginx configuration, application processes, alert delivery, and watchdog support are inspected before recovery actions are enabled.

A baseline prevents a pre-existing configuration problem from being mistaken for a recovery-system failure.

2. Test observer failures separately

Public HTTP, DNS, TLS, latency, content, TCP, and private-network checks are tested independently. The expected result is not always a restart. Many observer-side failures should produce evidence and an alert while leaving the target unchanged.

This confirms that loss of visibility is not treated as proof that a protected server requires a reboot.

3. Test local service failures

Nginx and application failures are simulated separately. The local guard must identify which service is unhealthy, validate the applicable configuration, restart only the relevant service, and retest it before reporting recovery.

An application failure should not automatically become an Nginx restart. An Nginx configuration failure should not enter an endless restart loop.

4. Test ambiguous evidence

Network-path interruptions, conflicting checks, unreachable observer conditions, invalid configurations, and incomplete evidence are used to confirm fail-open behavior.

In these conditions, RecoverNode should alert, preserve the running state where possible, and withhold disruptive action that cannot be justified.

5. Exhaust the recovery budget

Repeated failures are used to verify consecutive-failure thresholds, exponential backoff, cooldown periods, restart limits, reboot limits, persistent counters, and automatic lockout.

Recovery state must survive service restarts. A restarted guard should not forget that its safe action budget was already consumed.

6. Verify watchdog behavior independently

Where supported and authorized, the hardware watchdog is tested as a separate protection layer. It is intended for a complete operating system or userspace freeze, not as a substitute for service-specific recovery.

Watchdog configuration, timeout behavior, systemd integration, and reboot recovery are verified before production activation.

7. Verify logs and alerts

Health failures, restart attempts, successful recoveries, cooldown decisions, budget exhaustion, lockout, reconciliation failures, payment events, and backup failures must produce the expected journal entries, state records, or administrative notifications.

A recovery system that acts without leaving evidence is not operationally supportable.

8. Verify rollback and uninstall

Installation is not considered complete until status, rollback, and uninstall procedures have been tested. Generated configuration, systemd units, timers, state directories, permissions, and package contents are reviewed for reversibility.

9. Verify commercial operations

The production service also validates checkout fulfillment, subscription status, canceled-access enforcement, billing portal access, webhook idempotency, payment records, refund and dispute records, administrative alerts, and daily payment reconciliation.

10. Verify backup and restore

Local archives are created, inspected, retained, and copied to a separate host through a restricted backup account. Restore drills validate JSON records, downloadable packages, environment configuration, service units, scripts, and application package files.

Backup failure is deliberately simulated to confirm that the local archive remains intact, the remote-copy failure is reported, and a later successful run restores off-host protection.

Launch-readiness standard

A RecoverNode production deployment should not be considered ready merely because the software builds or a service starts. The complete environment must pass route checks, access controls, configuration checks, record validation, reconciliation checks, backup checks, timer checks, and recovery tests without unresolved failures.