Observability, Reliability, and Acceptance

Focuses on audit evidence, service objectives, capacity inputs, and release gates that demonstrate operational readiness.

12. Observability and Audit

12.1 Observability and Audit Center

The Observability and Audit Center accepts only events with an explicit source: data-plane runtime and decision events, ordinary control-plane changes and in-transaction audit, and local break-glass audit copies. It performs redaction, correlation, and evidence output, and does not create another control-plane audit-write path.

Audit coverage is measured against configuration-write transactions, while alerting and compliance queries are measured against correlated results from the Observability Center. The two share request_id, configuration versions, and resource identifiers, but their responsibilities do not overlap.

12.2 Core Metrics

• Request volume: grouped by region, platform, route, merchant, and outcome.
• Latency: total gateway latency, authorization latency, and backend latency, each measured at P50, P95, and P99.
• Error rate: grouped by 401, 403, 404, 429, 502, 503, 504, and internal reason code.
• Authorization: unknown Key, digest failure, IP mismatch, insufficient Scope, Key expiration, and Key revocation.
• Configuration: latest control-plane version, applied version in every region, propagation latency, consumer backlog, and reconciliation discrepancy.
• Routing: number of healthy backend instances, circuit-breaker state, retry count, and failover count.
• Security: Key-enumeration characteristics, anomalous IP changes, sudden increases in denial rate, and high-risk administration operations.

12.3 Logging and Tracing Fields

Data-plane structured logs contain at least: timestamp, request_id, trace_id, region, environment, route_name, platform, merchant_id, redacted Key ID, canonicalized source IP, authorization outcome, internal reason code, configuration epoch, configuration sequence, HTTP status, request duration, backend duration, and failure domain.

Administrative audit logs contain at least: actor identity, approver identity, resource, operation, reason, before-and-after digests, request identifier, source system, and time.

12.4 Alerts

The following events trigger real-time alerts:

1. The authorization-denial rate in any region rises significantly relative to its baseline over the preceding 30 minutes.
2. A regional configuration version trails the control plane by more than 30 seconds.
3. Security-change propagation exceeds 10 seconds.
4. Any platform has no healthy backend or triggers cross-region failover.
5. Gateway P99 latency or 5xx ratio exceeds the SLO.
6. One Key fails authentication repeatedly from multiple unbound IPs.
7. A high-risk administration operation lacks dual approval or audit writing fails.

13. Recommended SLOs and Capacity Principles

The following metrics are design recommendations, not commitments confirmed in the meeting:

The denominator of the gateway-availability SLI is requests that pass basic protocol validation and reach the regional ingress. Client-caused 4xx and 5xx marked as a backend failure domain do not count as gateway failures, but a separate end-to-end availability SLI is required. A 503 caused by API Key validation service fail-closed behavior is a gateway failure and consumes error budget.

Over 30 days, 99.99% permits only about 4.32 minutes of unavailability. The edge ingress, gateway, and API Key validation service may each consume no more than one-third of the error budget, requiring monthly availability of at least 99.997% for each component. Redis must not be a strongly synchronous dependency of every request; the in-process configuration snapshot isolates Redis failures. Recalculate this budget with the actual dependency graph after technology selection; do not simply multiply component SLOs and declare success.

A healthy region is one that has continuously reported data-plane health, configuration version, and clock-synchronization state over the most recent 30 seconds. The propagation SLI excludes only regions formally removed by the ingress execution point of the Traffic Routing Policy Center. It must not temporarily mark a lagging region unhealthy to evade measurement.

The meeting materials provide no peak QPS, average request body, response body, connection duration, merchant count, API Key count, or regional list. Therefore, credible instance sizing and hardware capacity planning cannot yet be provided.

Before capacity planning, collect from each platform its peak QPS over the most recent 30 days, P95 and P99 request-body size, P95 and P99 backend latency, connection-reuse rate, merchant count, Keys per merchant, and IP-binding count. Load testing must verify SLOs at 70% target capacity, retain at least 30% in-region burst headroom, and preserve remaining capacity after one availability-zone failure.

14. Verification and Acceptance

14.1 Authorization-Matrix Tests

The following automated tests must be covered:

1. A valid Key, bound IP, correct environment, and possession of the required scope pass.
2. A valid Key combined with another IP of the same merchant that is not bound returns 401.
3. Another Key of the same merchant that is not bound to the current IP returns 401.
4. When a Key and IP are bound but the required scope is missing, return 404; the response must match one for a truly non-existent route.
5. An unknown Key, incorrect Secret, or malformed credential format uniformly returns 401.
6. A Key in pending, suspended, revoked, or expired status is rejected.
7. Both new and old Secrets work during the rotation window; only the new Secret works after the window ends.
8. A client-forged X-Forwarded-For cannot change the source IP used for authorization.
9. A route that does not declare a required scope cannot be published.
10. A production Key cannot access a non-production environment, and a non-production Key cannot access production.
11. An unauthenticated request returns the same 401 for an existing and a non-existent route, preventing route enumeration.
12. After a Pepper upgrade, a Secret that references an older Pepper version and remains within its validity period continues to validate; new Secrets use the new Pepper version.
13. IPv4 bindings smaller than /29 and IPv6 bindings smaller than /64 are rejected, and overlapping IP ranges cannot be written.
14. A payment write request is rejected when its signature timestamp exceeds five minutes, its Nonce repeats within 10 minutes, its body digest differs, or its signature is invalid.

14.2 Multi-Region and Failure Tests

1. Stop one gateway instance. Requests fail over transparently with no authorization bypass.
2. Stop one availability zone. Remaining regional capacity still meets the SLO.
3. Stop Redis in one region. Known requests are processed by the bounded-snapshot strategy and unknown Keys are rejected.
4. Stop the message bus. Configuration is marked pending publication; after recovery, no event is lost and convergence follows version order.
5. Revoke a Key. Verify that all healthy regions reject it within the 10-second target.
6. Simulate the unavailability of an entire region. Perform failover only for policy-qualified reads or idempotent writes.
7. Simulate a lagging business database in a standby region. Confirm that the gateway does not perform blind cross-region writes.
8. Restore the original region. Perform controlled failback and verify configuration version and backend data state.
9. Before promoting a standby control plane, verify that writes by the former primary are fenced. The new control plane uses a higher epoch, and every region rejects events from the old epoch.
10. When both the control plane and PostgreSQL are unavailable, use break-glass to block a test Key in every healthy region, then archive it as a formal revocation record after recovery.
11. After a Redis failure, verify that the sum of local token-bucket quotas does not exceed the global quota and that recovery does not create a sudden traffic amplification.
12. When the last authorization snapshot is older than 15 minutes, high-risk write requests are rejected and read requests follow approved risk policy.
13. When a standby-region replication-state signal expires or lag exceeds the route threshold, read requests must not fail over automatically.
14. Reject an attempt by an old configuration version to overwrite a new one; 429 and backend 409 semantics remain stable, and atomic IP-binding replacement never exposes a partial set.

14.3 Release Gates

The first production version must satisfy all of the following:

1. The authorization-matrix automated-test pass rate is 100%.
2. Security-change propagation, gateway latency, and availability meet the targets in Section 13.
3. Scanning gateway, WAF, APM, and audit logs finds no plaintext Secret.
4. Complete a single-availability-zone failure drill, an actual control-plane takeover drill, an actual break-glass blocking drill, and a region-level data-plane tabletop exercise.
5. Owners of each initial platform sign off on routing, required scopes, idempotency, and failover policy.
6. The security team completes threat modeling and closes high-risk penetration-test findings.
7. The operations team confirms that dashboards, alerts, runbooks, and rollback procedures are available.