# Plan: add IPsec ingress as a zrok-orchestration frontend type

> Companion to `HANDOFF-ipsec-tiering.md` (architecture) and
> `TESTING-ipsec-harness.md` (validation topology). This doc is the
> implementation spec.
>
> **Scope boundary** (product direction): zrok-orchestration is a
> low-level infrastructure lever. Higher-level abstractions — tenants,
> customer sites, per-customer policy — live in `zrok-connector`, which
> invokes this service. That shapes the scope below: the handoff's tier
> model is useful conceptual framing, but most of the tiers resolve
> outside this API. Only the raw AWS infra for IPsec ingress is
> net-new Pulumi-managed-by-this-service.

## What this is

zrok-orchestration today exposes two "frontend types" (its API surface):

| Route | Controller | Template | Purpose |
|---|---|---|---|
| `POST /frontend` (alias `/stacks`) | `CreateStack` / `stacks.go` | `templates/zrok-custom-domains/` | Custom-domain HTTPS ingress |
| `POST /tcp-share` (alias `/frontdoor-tcp-share`) | `CreateFrontdoorTcpShare` / `frontdoor-tcpshares.go` | `templates/frontdoor-tcp-shares/` | TCP share on shared NLB |

Both templates create per-call AWS resources on top of a **shared
substrate stack** — the separate Pulumi program under `pulumi/` that owns
the shared VPC, ECS cluster, IAM roles, egress-all SG, and the
Global-Accelerator-fronted NLB. The orchestration-API stacks resolve that
substrate via `StackReference` (see
`templates/frontdoor-tcp-shares/import.go:13-96`).

The new IPsec ingress flow is a third frontend type in the same mould:
one new template, one new API endpoint, one new service, plus a small
refactor of `frontdoor-tcp-shares` so a share can target the new NLB.

### Handoff tier model — where each tier actually lands

| Handoff tier | Owner |
|---|---|
| Platform | Existing manual / NetEng process. Unchanged. |
| Hosting Region | `pulumi/` shared-infra program (current convention). Any VPC/cluster adjustments for IPsec land there, not in this API. See Gap 1. |
| Tenant | `zrok-connector`. Account tokens are passed per-call; no AWS resources per tenant. |
| **Endpoint Site = the IPsec frontend** | **`templates/ipsec-frontend/`** (new, this plan). Raw AWS ingress infra: TGW, VPN, CGW, subnet, NLB, SGs. |
| Share | `templates/frontdoor-tcp-shares/` (existing; narrow refactor to target an IPsec NLB). |

zrok-connector holds the mapping from customer-site / tenant IDs down to
specific IPsec-frontend stack IDs, peer IPs, PSKs, and share tokens. This
API never sees those business identifiers — only the infra inputs for
the stack it's asked to create.

## Scope

**In scope**
- `templates/ipsec-frontend/` — new Pulumi template. TGW, TGW
  attachments, Customer Gateway, VPN Connection, subnet, route table,
  internal NLB (pinned IP), NLB SG, task ingress SG.
- `POST/GET/DELETE /ipsec-frontend` — new controller + service
  following the `/tcp-share` / `TcpShareService` pattern.
- Narrow refactor of `templates/frontdoor-tcp-shares/` and
  `/tcp-share` request schema to accept an IPsec-frontend stack ID so a
  share can target an IPsec NLB instead of the shared GA NLB.
- Migration of live prototype resources (`vpc-0ddf3f3271533cbb4`,
  `subnet-026a6f24ca320a3da`, `nlb 10.1.5.10`, TGW
  `tgw-088243e203635af8c`, VPN `vpn-0982939c03412d0d2`) into the new
  template via `pulumi import`.

**Out of scope**
- Tenant / site / customer abstractions. zrok-connector territory.
- Hosting Region as an API-managed resource. VPC + cluster + IAM +
  egress SG continue to live in `pulumi/` shared infra. Any
  IPsec-specific additions to that substrate are a separate change
  against that program (see Gap 1).
- Service Site (tenant-owned backend + `zrok share private`). Tenant
  infra.
- `frontdoor-mtls` container changes. Image contract fixed by
  `../frontdoor-mtls/CHANGE-support-disabled-tls.md`.
- Retiring `ip-sec-tf-prototype`. Keep as smoke-test playground until
  the new template stands up clean.

## Target shape

Per-call dependency graph for the new flow:

```
zrok-connector
    │
    │  POST /ipsec-frontend               POST /tcp-share
    │  { id, peer_public_ip,              { ..., ipsec_frontend_id: "<id>" }
    │    peer_bgp_asn, peer_cidr,                      │
    │    psk_secret_arn }                              │
    ▼                                                  ▼
templates/ipsec-frontend/  ◄── StackReference ──  templates/frontdoor-tcp-shares/
    │                                                  │
    └── StackReference ──► pulumi/ shared infra ◄──────┘
        (VPC, cluster, IAM, sgEgressAllId, task exec role)
```

An ipsec-frontend stack is the substrate for zero-or-more tcp-share
stacks. A tcp-share stack targets either the shared GA NLB (existing
path, unchanged behavior) or an ipsec-frontend stack (new path) —
selected by whether the request carries `ipsec_frontend_id`.

## Implementation

### `templates/ipsec-frontend/` — new template

Layout follows `templates/frontdoor-tcp-shares/`:

```
templates/ipsec-frontend/
  go.mod
  Pulumi.yaml
  main.go
  provider.go
  import.go          # StackRef to pulumi/ shared infra
  output.go          # exports consumed by tcp-share stacks
  nlb.go
  securityGroups.go
  tgw.go
  vpn.go
  subnet.go
```

Resources (ported from prototype `nlb.tf`, `security_groups.tf`, plus
TGW/VPN that the prototype stubbed as data sources and are now
orchestration-owned per handoff):

- `aws_ec2_transit_gateway` (create-new default; existing-TGW override
  via config — Gap 4)
- `aws_ec2_transit_gateway_vpc_attachment`
- `aws_customer_gateway` (peer public IP + BGP ASN)
- `aws_vpn_connection` (PSK via Secrets Manager ARN — Gap 3)
- `aws_ec2_transit_gateway_vpn_attachment`
- `aws_ec2_transit_gateway_route_table` + associations + routes
- `aws_subnet` in the shared VPC
- `aws_route_table` + assoc + `<peer_cidr> → TGW` + egress route
- `aws_lb` (network, internal, `subnet_mapping.private_ipv4_address`
  pinned)
- `aws_security_group` nlb (NLB-scoped)
- `aws_security_group` tcp_share (task-scoped, **CIDR-based** because
  `preserve_client_ip=true`) — `name_prefix` + `create_before_destroy`
  per handoff Gotcha 6

Stack-ref inputs (from `pulumi/` shared infra):
`vpcId`, `vpcCidr`, `sgEgressAllId`, `taskExecRoleArn`,
`awsCustAccountId`.

Outputs (consumed by tcp-share stacks):
`nlbArn`, `nlbDnsName`, `nlbPrivateIp`, `nlbSgId`, `tcpShareSgId`,
`subnetId`, `peerCidr`, `vpnConnectionId`, `tgwId`.

Request-body inputs (from zrok-connector):
`id`, `peer_public_ip`, `peer_bgp_asn`, `peer_cidr`, `psk_secret_arn`,
`nlb_private_ip` (optional; template picks a free address in the new
subnet if omitted).

### `templates/frontdoor-tcp-shares/` — narrow refactor

Do **not** fork the template. Branch on a new optional config
`ipsec_frontend_id`:

- Unset → current behavior unchanged (StackRef shared-infra stack,
  GA NLB, shared SGs).
- Set → additionally StackRef
  `organization/<prefix>ipsec-frontend-<env>/<ipsec_frontend_id>` and
  use `nlbArn`, `nlbSgId`, `tcpShareSgId`, `subnetId`, `peerCidr` from
  there. Continue to StackRef shared infra for `taskExecRoleArn`,
  `sgEgressAllId`, `awsCustAccountId`.

Other changes:

1. Add `FRONTDOOR_TLS_ENABLED` as explicit config (default `true` to
   preserve current behavior; IPsec path callers pass `false`). Reject
   `FRONTDOOR_DEBUG=true` + `FRONTDOOR_TLS_ENABLED=false` at the
   controller (handoff Gotcha 2 — prototype-breaking combo).
2. Per-share listener-port ingress rules on the ipsec-frontend's `nlb`
   + `tcp_share` SGs as Share-scoped SG-rule-only resources, keyed off
   the stack-ref output SG IDs. This is the only cross-stack mutation
   the share performs; it stays within the handoff's "Share adds
   listener-port rules to Endpoint-owned SGs" pattern.

### Go service + controller

New files:

- `services/ipsecfrontend.go` — `IpsecFrontendService` with
  Create/Get/List/Delete. Mirrors `services/tcpshares.go`. Reuses
  `baseService`, `getWorkspace`, `selectStack`, `refreshStackConfig`,
  `streamEvents` from `services/common.go`.
- `controllers/ipsec-frontend.go` — HTTP handlers + request structs.
  Mirrors `controllers/frontdoor-tcpshares.go`.

Modified:

- `controllers/frontdoor-tcpshares.go` — add optional
  `ipsec_frontend_id` to `CreateTcpShareReq`.
- `services/tcpshares.go` — plumb `IpsecFrontendId` through to the
  Pulumi config map (`CreateTcpShareParams` + the `configMap` at
  `services/tcpshares.go:357-371`).
- `main.go` — register routes:
  ```go
  router.POST  ("/ipsec-frontend", controller.CreateIpsecFrontend)
  router.GET   ("/ipsec-frontend", controller.GetIpsecFrontend)
  router.DELETE("/ipsec-frontend", controller.DeleteIpsecFrontend)
  ```
- `controllers/controller.go` — add `IpsecFrontends *services.IpsecFrontendService`.
- `main.go` — instantiate `IpsecFrontendService` and pass into the
  Controller.

Success-resource-count threshold for the new service (parallel to
`FRONTDOOR_TCP_SHARE_SUCCESSFUL_RESOURCE_COUNT = 10` at
`services/tcpshares.go:17`): estimate ~18 from prototype + TGW/VPN.
Confirm after first clean deploy (Gap 7).

### Cross-cutting

- **Pulumi project name**: `ipsec-frontend-<env>` (mirrors
  `frontdoor-tcp-shares-<env>` per `services/common.go:143`).
- **Stack name**: caller-supplied `id` (slug-case), same as existing
  frontend types.
- **PSK handling**: Secrets Manager ARN arrives via request body; the
  Pulumi program reads it at runtime via `aws:secretsmanager`. Raw PSK
  never crosses the orchestration wire.
- **Env-var additions in `config/config.go`**: none required.

## Gaps and open questions

Numbered. **(blocker)** = must land before code.

1. **(blocker) Shared-infra exports audit**: the ipsec-frontend
   template expects `vpcId`, `vpcCidr`, `sgEgressAllId`,
   `taskExecRoleArn`, `awsCustAccountId` from the `pulumi/` shared
   stack. Confirm each exists as a current output, and PR any
   additions against `pulumi/` **before** scaffolding the new
   template. Separate project lifecycle, separate review.
2. **(blocker) Stack-reference naming scheme**: tcp-share stacks need
   a deterministic string to resolve an ipsec-frontend. Today's
   shared-infra ref hardcodes an env map
   (`templates/frontdoor-tcp-shares/import.go:22`:
   `staging → nf-staging-cust`). Proposed for ipsec-frontend:
   `organization/<prefix>ipsec-frontend-<env>/<ipsec_frontend_id>`.
   Confirm against the Pulumi org this service authenticates as.
3. **VPN PSK transport**: Secrets Manager ARN in request body
   (recommended) vs inline PSK field (sensitive, scrubbed from logs).
   SM ARN keeps PSK off the orchestration wire, lets NetEng rotate
   without redeploying, and fits the existing server/key/client-cert
   ARN pattern on `/tcp-share` (see `CreateTcpShareReq` at
   `controllers/frontdoor-tcpshares.go:22`).
4. **TGW ownership**: create-new per ipsec-frontend (default, handoff's
   isolation story) vs consume existing TGW via optional request-body
   override. Default recommended; keep override escape-hatch for
   NetEng-owned shared-TGW deployments.
5. **Peer-CIDR overlap enforcement**: two ipsec-frontend stacks with
   overlapping `peer_cidr` in the same VPC break routing. Validate at
   the controller (walk existing stacks) or trust zrok-connector
   upstream? Per Gap 10's general rule ("orchestration is a lever"),
   recommendation: **connector enforces**; orchestration logs a soft
   warning if it spots overlap but doesn't reject.
6. **Prototype import plan**: `pulumi import` for the live resources
   (account `925908137963`, region `us-east-1`) into a new
   ipsec-frontend stack. Scripted import doc (resource → URN map)
   required before cutover. Owner: whoever lands the ipsec-frontend
   template.
7. **Success-count threshold** for `IpsecFrontendService` — estimate
   ~18, confirm after first clean deploy.
8. **Share DELETE ordering vs ipsec-frontend DELETE**: DELETE on an
   ipsec-frontend with live shares targeting it leaves orphan
   cross-stack listener rules. Options:
   - Orchestration validates and returns `409 Conflict` with blocking
     share IDs (add `ErrStackHasDependents` to `services/common.go`).
   - Trust connector to order DELETEs and let Pulumi error at apply.
   Per Gap 10, lean toward trusting connector but still surface a
   helpful error rather than a raw Pulumi trace.
9. **(blocker) Reserved-token requirement at `/tcp-share`**: prototype
   uses ephemeral tokens; they regenerate on restart and silently
   break the stack (harness "Gotchas"). Does `/tcp-share` require
   reserved tokens from v1 of the IPsec path? Recommended: **yes** —
   backend-side systemd work tracked at
   `../ip-sec-tf-prototype/PLAN-reserved-share-tokens.md` is a
   prerequisite anyone publishing an IPsec share has to do anyway.
10. **Orchestration-vs-connector enforcement boundary**: multiple
    gaps (5, 8) hinge on "does zrok-orchestration validate this or
    trust the caller?" General rule from the scope boundary:
    orchestration is a lever — **trust the caller for semantics,
    enforce AWS-safety locally** (Pulumi preview catches CIDR
    collisions at apply-time anyway). Default thin validation; don't
    duplicate connector's business-rule enforcement.
11. **Validator shape** for `POST /tcp-share/{id}/test` (see Test +
    validation): sidecar in the share task def vs separate
    short-lived validator task. Sidecar measures real roundtrip at
    constant cost; short-lived task is cheap-at-rest but cold-starts.
    Default: **sidecar** behind a share-stack config flag so it's off
    in production until proven.
12. **Customer-side DNS/name**: IPsec peers whitelist the pinned NLB
    private IP, not a hostname. If the customer wants a friendly
    runbook name, that's a connector-side record, not orchestration's
    concern. Flag only.
13. **Version variant** (`ZROK_BASE_VERSION` v1 vs v2): does the IPsec
    flavor require v2? Confirm with zrok-controller team; if yes,
    reject v1 requests at the controller with 400 rather than letting
    the stack fail at apply.

## Gotchas to bake in

From the prototype (handoff §"Gotchas") — non-negotiable:

- Pin `frontdoor-mtls` image to `v2-494b642-0846c86` or newer for the
  TLS-disabled path.
- Refuse `FRONTDOOR_DEBUG=true` + `FRONTDOOR_TLS_ENABLED=false` at the
  controller.
- Task SG ingress must be CIDR-based (not SG-referenced) whenever
  `preserve_client_ip=true`.
- `name_prefix` + `create_before_destroy` on any SG that might be
  description-changed.

## Implementation sequence

1. **Resolve blockers 1, 2, 9.**
2. **PR against `pulumi/` shared infra** for any missing exports the
   ipsec-frontend template needs (driven by Gap 1).
3. **Scaffold `templates/ipsec-frontend/`**. Exercise on a throwaway
   account against the updated shared infra.
4. **Refactor `templates/frontdoor-tcp-shares/`** to accept optional
   `ipsec_frontend_id` stack ref. Test: a share created against the
   new ipsec-frontend stack passes full
   `nc <nlbPrivateIp> <listenerPort>` from the prototype's IPsec peer
   (see `TESTING-ipsec-harness.md` Test 2).
5. **Add `IpsecFrontendService` + controller + routes.** Mirror the
   existing TcpShareService structure.
6. **Scripted `pulumi import`** of the prototype's live resources into
   the new template. Validate Tests 0/1/2 from
   `TESTING-ipsec-harness.md` all pass through the imported stack, not
   the TF state.
7. **Retire the prototype's TF state** only after two successful green
   smoke cycles through the new ipsec-frontend stack.

## Test + validation

`TESTING-ipsec-harness.md` is the operator-facing runbook and source of
truth for the test topology (backend host `mg-ipsec-testing-zrok` /
`100.55.0.139`, pinned NLB `10.1.5.10`, listeners 9001-9003, APP-A/B/C
echo banners). Don't paraphrase it; reference it from runbooks and CI
fixtures.

### The three-tier test ladder (from the harness)

Every ipsec-frontend + share stand-up must pass these in order. Failing
Test 0 makes 1/2 meaningless; failing Test 1 means don't bother with
Test 2.

| # | Ran from | Exercises | Narrows fault to |
|---|---|---|---|
| **0** — zrok loopback | Backend host (`100.55.0.139`) against `127.0.0.1:920x` validators | zrok share → fabric → share-backend → echo | Everything **except** AWS |
| **1** — in-VPC `nc` | Any VPC host (e.g. `vpn-testrouter` @ `10.1.5.69`) against `10.1.5.10:900x` | NLB → SGs → Fargate → zrok → backend | AWS-side ingress only |
| **2** — over IPsec | Host in `172.31.0.0/16` against `10.1.5.10:900x` | TGW VPN → NLB → (as Test 1) | IPsec tunnel + TGW routing + `nlb_from_ipsec` SG rule |

Success signal on all three: distinct `APP-A:` / `APP-B:` / `APP-C:`
banners per listener port. Mis-wired listener → target group surfaces
instantly because the banner comes from the wrong app.

### What to carry into zrok-orchestration proper

From `TESTING-ipsec-harness.md` §"What to carry into zrok-orchestration
proper" — direct consequences of prototype pain, not optional:

1. **First-class validator endpoint.** `POST /tcp-share/{id}/test` on
   the refactored Share controller that runs a Test-0-equivalent `nc`
   probe. Sidecar or separate task — Gap 11. Either way, probe output
   must carry the banner pattern so a passing probe proves the whole
   path end-to-end.
2. **Reserved tokens only, no ephemerals** — Gap 9. Ephemeral is an
   operational landmine (tokens regenerate on restart, silently break
   the stack).
3. **`--network=host`** for any test/dev container this repo's harness
   ever spawns (Fargate uses a different network model — this is for
   the test-side, not production).
4. **Banner-or-equivalent distinguishable responses** in every
   integration test. Generic "TCP accepted" probes blind you during
   multi-app mis-wiring debug.
5. **`host:port` (not `tcp://host:port`)** for `zrok2 share private`
   targets (not zrok-orchestration's job, but if this repo ever
   generates Service-Site commands, enforce).

### Diagnostic flow when a probe fails

From the harness's tcpdump triage pattern. The three fingerprints map
to distinct owners:

- SYN/SYN-ACK/PSH then server FIN with no data back → **zrok or backend**
  (retry Test 0; connector / tenant problem).
- SYN/SYN-ACK then immediate RST → **Fargate task unhealthy or SG denies
  container port** (ipsec-frontend SG rules / share task def).
- No SYN-ACK → **IPsec tunnel, TGW routing, or `nlb_from_ipsec` SG rule**
  (ipsec-frontend tier).

Document this triage tree in the ops runbook so a failing
`/tcp-share/{id}/test` maps to a specific component owner rather than a
general page.

### CI

`go build` at minimum; repo has no tests or linting (per CLAUDE.md) and
this change doesn't introduce any. Manual validation against the
harness's backend host is the gate until Test 0/1 are automated against
throwaway stacks in a sandbox account.

## Coordination

- **zrok-connector team** — the scope boundary proposed here (connector
  owns tenant/site/customer semantics; orchestration owns raw infra
  only) needs explicit confirmation. Connector must be the component
  that maps customer IDs → ipsec-frontend stack IDs and tenant IDs →
  account tokens at call-time. If any of that semantic leaks into this
  API's request schema, the boundary has moved.
- `pulumi/` shared-infra owner — PR for any missing exports (Gap 1)
  needs their review; separate project lifecycle.
- `../frontdoor-mtls-cloud/frontdoor_mtls_entrypoint.py` +
  `../frontdoor-mtls/config.go` define the container env-var contract.
  Any env-var change in the Share refactor must be mirrored there
  first.
- Living-triumvirate: any change to the tier classification propagates
  to `../ip-sec-tf-prototype/ARCHITECTURE.md`, `diagrams/tiers.dot`,
  and the Confluence page.
- NetEng — TGW/VPN ownership transition from hand-created to
  orchestration-owned needs an explicit handoff. Don't surprise them
  with the first automated create.