Delivery Plan, Risks, and Architecture Decisions

Brings together implementation phases, known risks, open questions, architecture decision records, and review conclusions.

15. Phased Implementation Plan

Phase 0: Requirements Freeze and Threat Modeling

Produce the merchant-scale and traffic baseline, platform route catalog, Scope catalog, region catalog, backend idempotency-capability catalog, data-compliance constraints, and STRIDE threat model. Do not perform production capacity planning without these inputs.

Phase 1: Single-Region Vertical Slice

In a non-production environment, implement the minimum closed control-plane loop, Key lifecycle, Key-IP binding, Scope authorization, local configuration snapshot, audit, rate limiting, and one real platform route. Prefer Zero Confirmation as the first vertical slice because the meeting explicitly stated that its multi-database and multi-region design will be reused by BPS.

Phase acceptance is based on a real client completing the full process of Key creation, IP binding, authorized invocation, rejection of invalid combinations, rotation, and revocation.

Phase 2: Unified Integration of the Initial Platforms

In a non-production environment, integrate the routes of WebPay Inbox, BPS, and Zero Confirmation after confirmation by each platform owner. Define each route's required scope, timeout, retry, idempotency, and rate-limit policy; complete merchant migration, shadow traffic, and compatibility testing.

Phase 3: Multi-Region Data Plane

Deploy a second region, establish configuration distribution, fenced control-plane takeover, regional version reconciliation, break-glass, the two execution points of the Traffic Routing Policy Center, merchant regional assignment, and read-only failover, and complete availability-zone and region-level drills. The first production gateway version may be released only after every release gate in this phase passes.

Phase 4: Controlled Cross-Region Writes and Platform Expansion

Enable cross-region payment writes only for platforms that have demonstrated global idempotency, unique constraints, and an explicit primary-region switching mechanism. Establish a standard onboarding template so future platforms can integrate through routes, Scopes, service registration, and a verification checklist.

16. Key Risks and Mitigations

17. Items for Business and Platform-Team Confirmation

The following information is absent from both meeting materials and has no current factual basis:

1. Complete API routes, protocols, and versions for WebPay Inbox, BPS, and Zero Confirmation.
2. Actual read/write semantics, idempotency capabilities, timeout caps, and retry constraints of every platform.
3. The production-region, availability-zone, private-connectivity, and public-network ingress inventory.
4. Whether the transcript's reference to three Zero Confirmation databases is accurate, along with their topology, replication method, write primary, and switching procedure.
5. Whether BPS already has multi-database or multi-region capability.
6. Merchant count, API Key count, IP count per Key, peak QPS, and message size.
7. Whether merchants have stable egress IPs, and whether shared NAT, dynamic IPs, or third-party proxying exist.
8. Whether API Keys are created by internal personnel or through merchant self-service, and who owns approval responsibility.
9. The legal retention period, data residency, and privacy requirements for access and audit logs.
10. Inventory of current Kubernetes, PostgreSQL, Redis, message-bus, KMS, WAF, and global traffic-management products. The global traffic product serves as the ingress execution point of the Traffic Routing Policy Center.
11. External domain names, certificate system, and migration compatibility window for legacy ingress.
12. Whether the recommended SLOs, Key lifetime, 24-hour rotation window, and 15-minute offline-snapshot policy are acceptable.
13. Whether external TLS terminates at the WAF or passes through to the gateway, and the logging, key, and access-control responsibility at every termination point.
14. Whether payment write requests use mTLS or Ed25519 request signing, and whether read-only APIs accept residual replay risk under TLS and IP binding.
15. Every platform's read-consistency level, replication-lag signal source, signal-validity period, and automatic read-failover threshold.
16. The route-hint mechanism usable for ingress-region selection before authentication, including anti-enumeration and privacy boundaries for hint information.
17. Which merchants, platforms, and request types are suitable for fixed preferred-region, active-standby failover, and weighted or proximity routing respectively.

These items do not block logical-architecture review, but they block production capacity planning, detailed interface contracts, cross-region payment writes, and final technology selection. Items 1, 2, 4, 8, 14, 15, 16, and 17 must be confirmed before Phase 1 begins.

18. Architecture Decision Records

ADR-001: Unified Gateway with Fine-Grained Platform Authorization

• Decision: every platform shares a unified ingress and authorization model, while every Key and route retain explicit Scopes.
• Reason: meets the meeting requirement for a unified cross-platform ingress while preventing a Key for one platform from automatically gaining permission to other platforms.
• Consequence: a unified route and Scope registration process is required, and no platform may privately publish a merchant ingress that bypasses the gateway.

ADR-002: Bind IPs to a Specific API Key

• Decision: an IP-binding record uses api_key_id as its foreign key rather than a merchant-level implicit allowlist.
• Reason: directly solves the problem of mixing multiple Keys and multiple IPs of the same merchant.
• Consequence: a merchant must explicitly configure IPs when adding a Key; reusing the same IP across multiple Keys requires multiple audit records.

ADR-003: Centralized Control Plane and Regionally Autonomous Data Plane

• Decision: centrally manage the configuration source of truth and distribute it through versioned events to configuration-snapshot caches in every region. The request hot path does not access a cross-region database.
• Reason: balances unified control, low multi-region latency, and isolation from regional failures.
• Consequence: reliable configuration propagation, revocation invalidation, version reconciliation, and bounded-staleness policy must be built.

ADR-004: The Gateway Does Not Perform Platform Database Dual Writes

• Decision: the gateway chooses one explicit backend for every request and does not write directly to multiple business databases.
• Reason: an API Gateway lacks the context required to resolve payment-business cross-database transactions and data conflicts.
• Consequence: before cross-region payment writes are allowed, a platform must provide idempotency, replication, and primary-region switching capability.

ADR-005: Separate a Mature Proxy from an Independent API Key Validation Service

• Decision: use a mature L7 proxy for networking and traffic governance, and use an independent API Key validation service for Secret digests, Key status, environment, and Key-IP bindings. It returns a Scope set to the gateway; gateway authorization policy enforces route visibility and required scope.
• Reason: reduces the risk of a self-developed proxy while giving credential validation and route-permission decision separate testability and room to evolve.
• Consequence: the API Key validation service becomes a critical dependency and must be deployed with multiple instances, low-latency cache, and fail-closed behavior.

ADR-006: Prioritize Authentication Failure and Route Non-Visibility

• Decision: Key, status, environment, and IP-binding failures uniformly return 401; insufficient required scope and route absence uniformly return 404.
• Reason: prevents use of status codes to probe valid Keys, source-IP bindings, and unauthorized platform routes.
• Consequence: merchants may request the internal support team to inspect the precise reason code by request_id, but cannot infer IP or Scope configuration from an external error.

ADR-007: Fenced Single-Writer Disaster Recovery for the Control Plane

• Decision: deploy the control plane as cross-region active-and-standby, use a single-writer lease and (epoch, sequence) versions, and prohibit takeover before the old primary is fenced.
• Reason: an ordinary monotonic sequence cannot resolve dual-primary version divergence after a network partition.
• Consequence: control-plane takeover can sacrifice some management availability, but it does not allow two regions to publish conflicting configuration simultaneously.

ADR-008: Add Proof of Possession to Payment Write Requests

• Decision: production payment-write routes must use mTLS or Ed25519 request signing in addition to API Key and IP binding.
• Reason: even when a static Bearer Key is transported through TLS, it cannot withstand replay after credential exposure in a trusted network or TLS termination point.
• Consequence: merchants must manage client certificates or signing private keys, and the gateway needs certificate revocation, public-key lifecycle, clock, and Nonce-state capabilities.

ADR-009: Business Idempotency Belongs to Backend Platforms

• Decision: the gateway only enforces and forwards Idempotency-Key; it does not store request results or determine content conflicts.
• Reason: region-local deduplication state cannot guarantee consistency for cross-region payment writes.
• Consequence: platforms must provide global idempotency and unique constraints, and the gateway only forwards backend-generated 409.

19. Codex and Claude Architecture Peer-Review Record

19.1 Review Method

• Codex produced the first design version from the two meeting materials and ran local consistency checks.
• The Claude Code version was 2.1.201. Both requested and actual models were claude-fable-5, with medium effort.
• Claude used only Read, Grep, and Glob to read the meeting materials and the design document. It had no write or Shell permission and did not modify files.
• The first round ran an adversarial architecture review, the second verified closure of each finding, and the third rechecked only three residual risks closed after review.

19.2 Accepted and Corrected

1. Authentication and routing order could expose the existence of a Key or route through status codes; this was closed by ADR-006 and Section 6.
2. The control plane lacked cross-region disaster recovery, single-writer fencing, and non-divergent versioning; this was closed by Section 7.5 and ADR-007.
3. A leaked Key could not be blocked urgently during control-plane failure; this was closed by the regional deny-only channel in Section 10.3.
4. Pepper rotation could not validate existing Keys; pepper_version and old-version retention rules were added.
5. Idempotency ownership between gateway and backend was unclear; this was clarified by Section 6 and ADR-009.
6. Rate-limit behavior during Redis failure, environment isolation, CIDR limits, SLI measurement, Bearer replay protection, read-consistency signals, and transcript-evidence strength were all completed.
7. Emergency-deny renewal, quota during Redis failure plus scaling, and idempotency-key applicability, which remained after the second round, were closed before the final scoped review.

19.3 Rejected Challenges

1. Fail open when the authorization component fails: rejected. A payment ingress must fail closed and cannot trade authorization bypass for an availability target.
2. Simplify Key-level IP binding to a merchant-level allowlist: rejected. This directly violates the meeting requirement that multiple Keys and multiple IPs must not be mixed.
3. Let the gateway coordinate writes to three backend databases: rejected. Cross-database transactions and ledger consistency are platform-boundary responsibilities.
4. Lock in a specific product stack despite lacking infrastructure evidence: rejected. This document retains only a replaceable recommended technology baseline.

19.4 Convergence Conclusion

After three rounds of read-only review, no unresolved blocker or substantive contradiction was found. This document may proceed to business and engineering review, but its status remains “design recommendation,” not “production ready.” The factual gaps listed in Section 17 must be closed before their corresponding implementation phases, and all recommended SLOs, technology choices, and failure policies must be verified against real infrastructure, load tests, and drills.

The third round left only two non-blocking detailed-design items: define the duration of one break-glass renewal and the audit measurement for cumulative renewals; write an operator runbook for throughput degradation when a long Redis failure coincides with scale-out. These do not change the current logical architecture or delivery conclusion.

19.5 Traffic Routing Policy Center Specialized Peer Review

• The Claude Code version was 2.1.201. Both requested and actual models were claude-fable-5, with medium effort. It used only Read, Grep, and Glob and did not modify files.
• The first specialized review identified two blockers: ingress-region selection and post-authentication backend selection had been merged, and a pre-authentication routing hint had been treated as a trusted identity. This document closed both issues by adopting “one policy, two execution points.”
• The six substantive issues raised in the first round, covering ingress-failure semantics, candidate filtering, bounded staleness, compliance constraints, terminology consistency, and marking of proposed options, were all corrected.
• After reviewing the actual file in the second round, Claude confirmed no remaining blocker or substantive issue, with high confidence. The document may proceed to business and engineering review.
• Two non-blocking wording improvements from final review were adopted locally after final review: add a writer-gate fallback for a misrouted write request before authentication, and clarify that reason_code is only for internal logs and audit. Neither adjustment changes architectural semantics, so no further external review was started.

19.6 Specialized Peer Review of Abstract Layering and Node Expansion

• The first read-only review accepted the direction of “an eight-node overview, expansion of six abstract nodes, and one cross-node sequence diagram,” and required eight changes: anchor the ingress execution point, establish governance order, clean internal identity headers, define the two-component Access Control boundary, specify break-glass inputs, separate audit responsibilities, remove dangling overview inputs, and separate mTLS from Ed25519 boundaries.
• This document implemented every required change. Static checks confirmed a one-to-one mapping between eight Mermaid blocks and eight Mermaid containers in HTML, exactly eight nodes in the overview, locatable expansion sections for all six nodes, and zero occurrences of ambiguous region-related terminology from the earlier version.
• The second read-only review checked ten facts individually and concluded accept with high confidence. It found no blocker, substantive issue, ungrounded module, duplicated responsibility, or incorrect connection.
• Two non-blocking readability recommendations from the second round were adopted locally after review: describe the sequence diagram as “abstract boundaries and key dependencies,” and place the verified-identity node explicitly between the API Key validation-service component and the gateway authorization-policy component.
• Before this round, the original single-page HTML had no browser-level visual-acceptance evidence: Chrome automation rejected a local file:// URL, and Mermaid CLI lacked the specified version of Headless Chrome. This round completed verification through a local HTTP preview: the overview home page, topic navigation, and all eight Mermaid diagrams were rendered and checked in a real browser. This verification covers only the local static site and does not represent production readiness; it must be repeated in the target deployment environment before external deployment.