# MarketLink — Disaster Recovery Plan

**System:** MarketLink — CMS EDE Platform
**Operator:** Wipro Healthplan Services
**Environment:** On-prem deployment
**Document Owner:** Platform Operations Lead
**Classification:** Controlled Unclassified Information (CUI)
**Last Reviewed:** 2026-04-14
**Review Cadence:** Annual; after any DR drill; after any major incident

---

## 1. Purpose & Scope

This Disaster Recovery Plan (DRP) defines how MarketLink restores
service to its brokers and agencies after a disruption that degrades
or eliminates the running environment — hardware failure, data
corruption, ransomware, facility outage, or regional disaster.

**In scope:**

- The MarketLink Next.js application and its CI/CD pipeline
- The PostgreSQL database that backs the application
- Encrypted backups, encryption keys, and the documents vault
- Network ingress (DNS, load balancer, reverse proxy)
- Agent session state and active enrollment workflows

**Out of scope** (governed by separate plans):

- Broader Wipro Healthplan Services infrastructure DR (per HPS DR Plan)
- Third-party services (CMS Marketplace API, Okta SSO) — MarketLink's
  recovery assumes these are reachable, and degrades gracefully if not
- Incident response and breach notification — see the Incident Response
  Plan (IRP)

## 2. Recovery Objectives

| Objective | Target | Rationale |
|---|---|---|
| **RPO** (Recovery Point Objective) | **1 hour** | PostgreSQL archive log shipping plus hourly incremental backups ensures no more than 60 minutes of broker activity is lost in a worst-case disk failure |
| **RTO** (Recovery Time Objective) | **4 hours** | Application is fully stateless apart from the database; CI/CD can redeploy from GitLab inside 30 min, database restoration from warm backup inside 2h, DNS propagation < 15 min |
| **MTPOD** (Max Tolerable Period of Disruption) | **8 hours** | Beyond this window broker enrollments are materially impacted and CMS must be notified |

## 3. System Architecture (Summary)

Full details live in the SAP and SSPP. Key recovery-relevant facts:

- **Application tier:** Next.js 15 + Node.js 20, deployed as a systemd
  service behind an internal reverse proxy. Stateless — all persistent
  data lives in the database or the documents volume.
- **Database tier:** PostgreSQL 16 primary with streaming replica in a
  separate availability zone. Automated daily full backups + hourly
  incremental (WAL archive) backups retained 30 days.
- **Documents:** Encrypted-at-rest file storage (consumer documents,
  compliance artifacts). Rsync'd nightly to the offline backup host.
- **Secrets:** Environment variables injected from the HPS secrets
  vault at service-start time. `PII_ENCRYPTION_KEY` is stored in the
  vault, not co-located with database backups.
- **Source:** GitLab (`gitlab.com/healthplan/hps-ai/marketlink`) is
  the authoritative source of truth for application code.

## 4. Backup Strategy

### 4.1 Database
- **Full backup** — daily at 02:00 local, retained 30 days, verified
  nightly with checksum compare.
- **Incremental (WAL)** — streamed continuously to the backup host,
  retained 7 days.
- **Offsite copy** — encrypted daily full backup rsync'd to the
  secondary site.
- **Replica** — streaming PostgreSQL replica with < 60 s lag.

### 4.2 Application
- Source is GitLab; no additional backup needed.
- Deployed artifact is rebuildable from any tagged commit within 30 min
  via the existing CI/CD pipeline.
- `.env` and `.env.local` are regenerated from the secrets vault on
  every deploy — they are never backed up with the source tree.

### 4.3 Documents Volume
- Rsync nightly to the backup host, retained 90 days.
- Encrypted at rest with a key held separately from the data.

### 4.4 Encryption Keys
- `PII_ENCRYPTION_KEY`, `NEXTAUTH_SECRET`, and database credentials
  live in the HPS secrets vault.
- Key rotation schedule documented separately in the SSPP.
- **Critical:** keys are stored at a different logical location from
  the backups. A single compromise cannot yield both plaintext data
  and the ability to decrypt.

## 5. Recovery Procedures

### 5.1 Database Failure (primary down, replica healthy)
1. Platform Ops promotes the replica to primary (`pg_ctl promote`).
2. Update the application's `DATABASE_URL` to point at the new
   primary; restart the MarketLink service.
3. Verify a health check against `/api/health` returns 200.
4. Verify a broker can log in and list their clients.
5. Spin up a new replica against the promoted primary; resume streaming.
6. Expected RTO: **< 30 min.**

### 5.2 Database Data Loss (both primary and replica corrupted)
1. Halt the MarketLink service.
2. Restore the most recent full backup to a fresh database instance.
3. Replay WAL archives up to the latest consistent point before the
   corruption window (use PITR — point-in-time recovery).
4. Verify schema integrity (`npx prisma migrate status`).
5. Verify row counts against last-known-good snapshots for
   `clients`, `applications`, `policies`, `consent_records`.
6. Update `DATABASE_URL`, restart the service.
7. Expected RTO: **2–4 hours** depending on backup size.

### 5.3 Application Redeployment (host failure, compromised image)
1. Provision a fresh host from the standard image.
2. Clone the tagged release from GitLab.
3. Run `npm ci --production` and `npx prisma generate`.
4. Inject environment variables from the secrets vault.
5. Start the systemd service; wait for `/api/health` 200.
6. Expected RTO: **< 30 min.**

### 5.4 DNS / Ingress Failure
1. Switch the DNS A record to the secondary site's load balancer.
2. TTL is 60 s, so propagation is minutes in practice.
3. Expected RTO: **< 15 min.**

### 5.5 Documents Volume Loss
1. Restore the nightly rsync snapshot to a fresh volume.
2. Remount at the service's expected path.
3. Restart the MarketLink service.
4. Expected RTO: **< 1 hour.**

### 5.6 Post-Recovery Verification Checklist
- [ ] `/api/health` returns 200
- [ ] Database schema matches expected Prisma baseline
- [ ] Broker login works (test account)
- [ ] Creating a test client succeeds end-to-end
- [ ] Compliance vault documents download (auth + access-code gate)
- [ ] Audit log is accepting entries
- [ ] CMS Marketplace API reachable from the restored host
- [ ] Okta SSO login works (production) / dev-bypass credentials work
      (staging)
- [ ] Certificate expiry ≥ 30 days remaining
- [ ] Recent audit log entries match pre-failure broker activity

## 6. Roles & Responsibilities

| Role | Responsibility |
|---|---|
| **Platform Operations Lead** | Owns this DRP; leads DR drills; declares a disaster; coordinates recovery |
| **Platform Security Officer** | Ensures backups are encrypted, keys are safe, incident communication follows the IRP |
| **Database Administrator** | Executes database restore procedures; validates schema and row counts |
| **On-Call Engineer** | First responder; diagnoses failure class; triggers the relevant runbook |
| **Compliance Officer** | Coordinates CMS notification (1-hour SLA for PII-impacting incidents) |
| **Broker Support Lead** | Communicates service status to impacted agencies |

All roles are titles, not individuals. The on-call rotation and the
person holding each title at any time are maintained in the HPS
operational directory — not in this document.

## 7. Communication Plan

### 7.1 Internal
- Incident declared in the #marketlink-oncall channel.
- Platform Ops Lead opens an incident bridge within 15 min.
- Status updates every 30 min during active recovery.

### 7.2 External — Brokers and Agencies
- If broker-facing outage exceeds 15 min, the support team posts a
  status banner on the MarketLink dashboard and emails agency admins.
- Updates every hour until resolved.
- Final "all clear" notification with incident ID for follow-up.

### 7.3 CMS
- PII-impacting incidents: CMS is notified within **1 hour** per IRP
  §5.2 and 45 CFR §155.260.
- Service degradation without PII impact: no CMS notification required.

### 7.4 Consumers
- Consumer notification is triggered only when the IRP determines a
  privacy breach; not all DR events rise to that threshold.

## 8. Testing Schedule

| Test Type | Frequency | Owner |
|---|---|---|
| **Backup restoration drill** — restore last night's backup to an isolated instance, verify schema + row counts | Quarterly | DBA |
| **Full DR drill** — simulate primary-site loss, execute §5.2 procedures end-to-end, measure actual RTO | Annual | Platform Ops Lead |
| **Secrets vault recovery** — verify encryption keys can be retrieved through the documented process | Semi-annual | PSO |
| **Runbook walkthrough** — on-call reviews §5 runbooks, identifies ambiguities | Every on-call rotation | On-Call Engineer |

Each test produces a short written report; failures open a POA&M entry.

## 9. Document Control

| Version | Date | Change |
|---|---|---|
| 1.0 | 2026-04-14 | Initial DR plan for the CMS EDE certification package |
