An RPC gateway operations runbook must distinguish transport failures, stale chain data, rate limits and unknown transaction submissions. Each requires a different response. This guide covers method-aware health checks and safe recovery. The gateway architecture guide covers routing, interfaces and credential boundaries.
Check network identity and useful data
An HTTP success response alone does not establish upstream health. Confirm the expected network, recent block progress and a lightweight method the application requires. Keep stale-data failures separate from transport errors and unsupported requests. A diagnostic should be inexpensive enough not to become a significant competing workload.
Classify errors before changing upstreams
Separate an invalid method or parameter from a timeout, a throttled request or a node that is behind. Retrying the same invalid request through every upstream wastes capacity and hides the actual problem. Preserve the original error and request identifier, with credentials removed, so an incident can be investigated later.
Use bounded retries for repeatable reads
Retry eligible transient read failures with bounded backoff and a maximum attempt or time budget. Respect rate-limit behavior and avoid retry storms when several workers fail together. If comparing providers, use a common block for deterministic checks. A newer latest head can explain a difference without proving that either endpoint is faulty.
Reconcile unknown transaction submissions
Store the original signed payload and derived hash before sending. After a timeout, query the original hash and relevant sender state before constructing another transfer. The network may have received the first request even when the gateway did not return a result. Changing upstreams is not evidence that the original submission failed.
Return an upstream to service deliberately
Define the observations that show recovery: useful method responses, acceptable freshness and a stable evaluation window. Ramp traffic back gradually where the routing system supports it. Keep an owner for unresolved data disagreements and record the period affected by stale or unavailable responses.
Run drills with usage accounting visible
Test a stale head, a rate limit, an unsupported method, a connection drop and an ambiguous submission response. Confirm that alerts identify the class of failure and that recovery does not generate uncontrolled repeated work. Include repeated operations in usage analysis according to current accounting rules. Check network-specific RPC support before treating every upstream as interchangeable.
Frequently Asked Questions
Is a fast block-number response enough for a health check?
It is one signal. Also verify the expected network, progress and required method behavior. A node can answer quickly while serving stale data.
Should every RPC error trigger failover?
No. Invalid parameters and unsupported methods may require application or configuration changes. Classify the error before routing another attempt.
What should happen after a submission timeout?
Preserve the original hash and signed payload, check the transaction’s state and reconcile the unknown outcome before creating a different transfer.