r-client High-Availability Design Notes

This document records the assumptions, decision history, and remaining study questions behind the parameterized high-availability usage specified in r-client.md. It is a design workbench, not a normative extension of that specification or wire_protocol.md. The MVP C client implements the selected strategy as its sole resource-request policy; this document explains why that shape was chosen.

1. Scope and vocabulary

The protocol has two client request classes, excluding administrative traffic:

1. Resource request: carries guards and resource quantities and expects a server response. It may change rate-counter state.
2. Latency report: carries service-latency observations and expects no response. It is fire-and-forget and may be sent by a client that never sends resource requests. Conversely, a resource-request client may never send latency reports.

These request classes are independent. A latency report does not complete, retry, delay, or otherwise control a resource request. Latency reports are idempotent within the service’s TTL window: repeated delivery of the same observation is equivalent to one delivery. Resource requests are generally not idempotent at the client-policy level because they may consume rate state. At the wire/server level, replaying the same request identity within the deduplication TTL is suppressed by server deduplication.

2. Single-server resource-request model

When DNS SRV yields one usable r-server, the same parameterized policy still applies:

1. Identify resources r1, r2, … with quantities q1, q2, … and optimistic latency guards such as s1.latency < t1.
2. Construct one resource request containing the guards and resource demands, plus protocol metadata such as a unique request identity and TTL.
3. Send it to the server and wait according to round 0 of the configured schedule.
4. A valid response is necessarily from the oldest server and completes immediately.
5. If the round remains response-free, replay the same logical request with the same deduplication identity while the replay count permits.
6. After the last response-free transmission round, optionally enter the final receive-only interval.
7. Abandon the operation at the derived deduplication deadline. A later response cannot affect the client decision.

The default C-client schedule uses U = 20 ms, one replay, and one final receive-only interval, for a derived wire TTL of 3 * U = 60 ms. The replay datagrams are separate transmissions but one logical resource request.

3. Steering

A server response may include a steering indication controlling whether the client should retain its UDP source port. This is an optimization for server- side XDP behavior, not a resource-request result or a protocol command.

• keep_port = true recommends retaining the current source port.
• keep_port = false recommends switching to another source port for future sends.
• The client decides when and how to apply the recommendation.
• Earlier application is generally preferable, but the client may defer it for socket-safety, in-flight-send, callback, or implementation reasons.

Steering applies to transport behavior and is independent of resource-request replays, admission results, and latency reports.

4. HA assumptions

In the HA model, one API key is assigned two or more r-servers through DNS SRV. The membership and identities can change: servers may be added, removed, replaced, or restarted. The client discovers those changes asynchronously.

The r-servers do not know that they are members of a set and never communicate with one another. HA is implemented entirely by the client:

• The client sends every resource request to all currently known r-servers.
• The client sends every latency report to all currently known r-servers.
• Each server independently deduplicates resource-request replays and incorporates latency reports into its TTL-based tracker.
• In healthy conditions, replicated delivery and TTLs keep the servers approximately synchronized.
• Resource counters and latency trackers eventually converge as old state expires and new requests/reports are replicated.

The HA guarantee is eventual convergence only. This model does not promise linearizability, quorum consistency, failover continuity, or immediate agreement during packet loss, partitions, server restarts, or membership changes.

When all servers are healthy and connectivity is reliable, preferring the response from the oldest r-server is a useful deterministic heuristic. It is not server consensus and cannot eliminate divergence outside those conditions.

5. Resolved MVP policy choices

The MVP resolves the original policy questions as follows:

• fan out the initial logical request to one immutable membership snapshot;
• rank trusted responses by decoded server startup time and then by the lower full server_id;
• return immediately when the globally oldest server responds;
• otherwise retain the oldest valid fallback only until the configured preference deadline;
• treat allow and deny as equally valid responses for ranking and completion;
• replay only after a response-free round, and only to servers still missing a valid response;
• optionally perform outcome-independent completion delivery to missing servers immediately before returning a selected response;
• apply asynchronous DNS membership changes only to later requests; and
• keep DNS refresh and steering in persistent transport configuration rather than the request-selection policy.

The MVP does not use quorum, per-server reliability scoring, recent-error ranking, server-stability filtering, or a policy discriminator.

6. One policy, multiple numeric configurations

The original candidate policies are expressible as configurations of the same algorithm rather than separate execution paths:

Former candidate	Parameterization
Strict-oldest	Set P(k) = B(k) so a fallback waits for the full round.
Oldest-preferred	Set 0 < P(k) < B(k) so the oldest server has a bounded preference interval.
Fastest-valid	Set P(k) = 0, making the first valid response immediately selectable.
Bounded-collection	Use a nonzero P(k) as the bounded collection interval and choose the oldest response observed within it.

Fixed, linear, and exponential schedules change replay pacing without creating new policy kinds. The C API therefore exposes one flat r_request_policy_t and no compatibility wait/quorum/retry policy.

7. Correctness and efficiency metrics

Each parameter configuration should be evaluated using reproducible traces and fault scenarios. Initial metrics are:

• admission correctness with synchronized healthy servers;
• false-allow and false-deny rates while servers temporarily disagree;
• resource-counter over-consumption caused by policy decisions or unsafe replays;
• request latency and tail latency;
• availability when the preferred server is slow or unavailable;
• duplicate delivery and deduplication-window utilization;
• convergence time after membership changes or packet loss;
• steering recommendation application latency; and
• handling of late, invalid, and contradictory responses.

8. Implementation status and next work

Completed:

1. The protocol model and server-local deduplication constraints were checked against the wire specification and Rust server.
2. Initial deterministic traces and event vocabulary were defined.
3. The parameterized oldest-first algorithm was selected and specified in r-client.md.
4. rl-c-client PR #25 implemented it as the sole C request policy, with fixed, linear, and exponential schedules, derived TTL validation, missing-only replays, and completion delivery.
5. Deterministic C tests cover the default schedule, custom schedules, preference timing, TTL bounds, response identity, and completion delivery.

Remaining:

1. Tune U and schedule defaults from measured client/server round trips.
2. Build a deterministic evaluator for broader loss, divergence, restart, and DNS-change traces.
3. Derive cross-language behavior tests for Rust, Java, and JavaScript clients.
4. Reevaluate only with evidence whether future policy parameters are needed; do not add parallel compatibility execution paths.

9. Current decision

The current decision is the parameterized strategy in Section 9.3 of r-client.md. It is the baseline against which future traces and metrics are evaluated. Alternative consistency/latency points are selected by numeric parameters, not by choosing a separate policy implementation.

10. Initial deterministic traces

The traces below define shared inputs for the first policy comparison. A is the oldest server and B is newer. The resource request has one stable logical request identity and is sent to both servers.

Trace	Events	Expected observation
Healthy synchronized servers	Fan out to A and B; both return valid, equivalent responses before the preference deadline.	Select A immediately when its response arrives.
Slow oldest server	Fan out; B returns a valid response quickly; A returns before or after the preference deadline.	Select A if it arrives by the preference deadline; otherwise select B at that deadline.
Preferred-server failure	Fan out; A produces no response; B returns a valid response.	Select B at the preference deadline, then best-effort completion-deliver to missing A when enabled.

For each trace, the evaluator will record send times, validated responses, decision time, selected server identity, replays, abandonment, and any steering action. Late responses remain observable events but must not change an already-final client decision.

11. Verification findings

The first comparison against the current specifications and Rust server gives these corrections and constraints:

1. Resource replay window: Rate Request.dedup_ttl_ms is required. The client must stop retransmitting at the request timestamp plus the accepted replay window. The server currently rejects a zero value or a value above either the tenant quota or listener cap; it does not silently guarantee a 300 ms window for every request. The 300 ms value is the current default/cap in the Rust implementation and test fixtures, not a universal protocol constant.
2. Deduplication scope: the Rust server keys rate-response replay by (pdu_type, tenant_mgmt_flag, key_id, unique_id) in each listener’s local deduplication cache. Thus identical broadcasts are independently deduplicated by each r-server; there is no cross-server deduplication.
3. Resource semantics: a healthy deduplication cache replays the cached result within the accepted window. If replay state is lost or evicted, the wire specification explicitly suspends that guarantee; a client policy must treat this as a degraded condition rather than assume safety indefinitely.
4. Latency reports: the latency-report PDU has no resource-request response and no equivalent rate-request deduplication identity. Its idempotence comes from the server’s TTL-based latency-tracker semantics, so broadcasting and repeating reports is safe within the tracker TTL but is a different mechanism from rate-request replay.
5. Server identity: responses carry the responding listener’s full 64-bit server_id in the response Tenant Header key_id. The high bits encode startup time, which can support an oldest-server heuristic, but identity trust still requires matching the asynchronously discovered SRV identity.
6. Steering: steering_feedback = 0 means change-port is recommended and 1 means keep-port is recommended. The Rust server derives this from packet steering state; the client remains free to apply it later.
7. HA commit safety: the former normative “HA Commit Safety” text in wire_protocol.md was not supported by the protocol or Rust implementation. It has been replaced with an open-study note. The wire protocol now states explicitly that cross-server commit safety is undefined and that stronger coordinated modes are only potential future directions.
8. C-client implementation: rl-c-client PR #25 implements one flat parameterized request policy. It removed the former wait, quorum, reliability selection, retry/backoff, resend, and server-stability paths. DNS refresh pacing moved to client configuration because it is not part of response selection.

12. Persistent client state API

The original API sketch expressed reusable client resources as explicit function state, similar to an Erlang process state:

(state1, result) = execute_rate_request(state0, rate_request, api_key)
state2           = close(state1)

The persistent state may contain:

• the UDP socket and current source-port/steering state;
• the resolved r-server set and its DNS expiry time;
• the API-key identity associated with that resolution; and
• optional diagnostic statistics.

The individual request remains temporary and contains its unique request ID, logical payload, membership snapshot, deduplication deadline, replay schedule, and responses observed for that request.

The proposed interface is:

execute_rate_request(state, request, api_key) -> (new_state, result)
close(state)                                  -> closed_state

execute_rate_request should reuse the socket and DNS result while they remain valid, refresh DNS when its TTL expires or the API-key identity changes, and return the updated state on both success and failure. Every replay reuses one logical request ID until the deduplication deadline. The function returns the response selected by the oldest-preference deadlines, or a failure if no valid response is selected before that deadline.

close owns cleanup of persistent resources. It should close the socket, discard cached endpoint state, be idempotent, and return a terminal state that cannot be used for another request. Latency reports remain independent and are not represented as response-bearing entries in the resource-request state.

The C API realizes the same ownership split asynchronously rather than using this tuple-return syntax. r_client_t owns credential, DNS membership, policy, and in-flight request state; the host supplies event-loop I/O and timers. r_runtime_client_t is the optional layer that also owns reusable UDP sockets. r_client_destroy() or r_runtime_client_destroy() provides the corresponding cleanup operation.