Operate

Testing

The primary v1 path is a modular monolith. Go and Nest are checked with unit, contract, browser, acceptance, load and recovery tests; optional split-deployment experiments remain available for products that need them.

backend/scripts · frontend/e2e · infra/testing · .github/workflowsChecked against the files 16 October 2026

Layers

Layer	Tool	Runs in CI	Proves
Unit	Go `testing`, Vitest	Yes	Domain rules, handlers and adapters in isolation.
Architecture	Go architecture test, dependency-cruiser	Yes	No cross-module imports; layers point inward; SQL names only its own schema.
Contract	Redocly, oasdiff, parity scripts	Yes	The API spec is valid and not broken; both backends ship the same migrations and catalogs.
Security	Role acceptance, security matrices, ZAP	Selected / manual	Role boundaries, denials, authenticated API responses and configured scan coverage.
End-to-end	Playwright, backend scripts, curl, jq	Selected in CI	Core user journeys run through both frontends and both backends. Main/dev, scheduled and opted-in runs execute the wider browser and backend matrix; most PRs use faster gates.
Load	k6	Manual / selected	Account, Files, Billing, Chat, Webhook, identity and AI workloads have bounded smoke/load profiles. Results qualify tested runs, not production capacity.
Resilience	Toxiproxy, disposable services, recovery harnesses	Manual / selected	Recovery scenarios cover PostgreSQL, NATS, AI providers, Files/S3, Chat, Billing, Webhooks and notifications. Offline orchestration checks run separately from live drills.

Unit and integration tests

Go

Standard testing with no assertion library, next to the code they test. Root CI runs go test -race -p 4 ./..., retaining every package while bounding package concurrency. Database- and Redis-backed suites need their matching test URLs.

Nest

NestJS

Vitest, type/build and architecture checks, plus an in-process HTTP end-to-end spec. PostgreSQL-backed cases run when configured by the test workflow.

Every domain

All fifteen Go/Nest modules have tests in their owning packages. Shared HTTP, event, capability and fixture contracts make the infrastructure suites reusable across backend implementations. The commands below are selected database recovery or boundary suites; the coverage column also includes unit, contract and acceptance checks. Each live suite needs its documented disposable dependencies.

Domain	Coverage	Selected live suite
Identity	Sessions, MFA/passkeys, OAuth, security changes and identity lifecycle	`make test-identity-recovery BACKEND=go`
Account	Membership, ownership, lifecycle, receipts and concurrent writes	`make test-account-recovery BACKEND=go`
Authorization	Permission projections, revocation, replay and provider failures	`make test-authorization-recovery BACKEND=go`
Audit	Hash chains, append/export/prune, event replay and recovery	`make test-audit-recovery BACKEND=go`
Billing	Checkout, subscription and invoice lifecycle, ownership and reconciliation	`make test-billing-recovery BACKEND=go`
Chat	Message order, receipts, attachments, membership, erasure and purge	`make test-chat-recovery BACKEND=go`
Entitlements	Plan projections, grants, revocation and event replay	`make test-entitlements-recovery BACKEND=go`
Usage	Reservations, settlement, storage gauges and month boundaries	`make test-usage-boundaries BACKEND=go`
AI	Worker leases, quota races, cancellation, documents and tool approvals	`make test-ai-recovery BACKEND=go`
Integrations	API keys, signed webhooks, delivery leases, MCP and outbound target checks	`make test-webhook-recovery BACKEND=go`
Notifications	Inbox/preferences, source events, email, retention and privacy	`make test-notification-delivery BACKEND=go`
Files	Uploads, scanner/copy failures, quota, receipts, erasure and purge	`make test-files-recovery BACKEND=go`
Privacy	Exports, erasure fences, retention and restore-ledger replay	`make test-privacy-export-recovery BACKEND=go`
Realtime	Broadcast recovery, authorization, subscriptions and presence	`make test-realtime-recovery BACKEND=go`
Operations	Staff authorization, maintenance, account freezes and recovery	`make test-operations-recovery BACKEND=go`

Use BACKEND=nest for the matching Nest execution. Python, Rust, .NET, Java and Zig require their own adapters and live qualification; shared contracts alone do not establish parity.

Verified checkpoints

The 16 October 2026 core recovery closeout passed all 65 default root CI gates in one execution: 136 Go race-test packages, 1,861 Nest cases and 15,289 Python cases. Nest retained 324 skips and Python 532; skipped database/live tests are not counted as live evidence. Infrastructure checks included 991 resilience helpers, 116 acceptance helpers, 109 schema regressions and six alert harness cases.

The checkpoint fixed PDF.js input-buffer ownership in Nest so scanned PDFs retain their bytes for OCR fallback. Two worker-transfer regressions and a native PDF.js probe verify the fix. Source snapshots, ordered gate results, failures from superseded attempts and cleanup records are retained privately; the public progress record is infra/testing/completion.md.

Native k6 fixture checks and paired live fault runs are recorded separately. Recovery hardening uses one allowance through restoration, retry/replay, durable state, event delivery and final public/liveness checks. A late successful response cannot satisfy the recovery budget. Remaining capacity, production restore and operational work is tracked separately from this checkpoint.

The finite closeout passes 117 new regressions, 316 native fixture cases and all twelve live runs: PostgreSQL read/write completion loss, lock cancellation, transaction crash, migration rollback/retry and remote Authorization recovery on both Go and Nest. Complete recovery takes 0–13 integer seconds within the original 90-second database or 30-second provider allowance. Thirty-six independent resource queries verify cleanup. See infra/testing/resilience/core-dependency-closeout.md.

Where testing stopped

The Go/Nest core dependency recovery tranche is complete at candidate 20261016T111155Z: 65/65 default root CI gates and twelve paired live recovery runs passed. The repository handoff at infra/testing/HANDOFF.md records the evidence, source boundary and remaining work. Later source changes require their own checks.

Representative capacity, wider fault and security coverage, production operations and additional backend qualification remain separately scoped in infra/testing/roadmap.md. The next planned phase is a compliance-readiness assessment, beginning with control-to-evidence mapping and gaps for the selected deployment. That phase has not started; the testing checkpoint is technical evidence and does not establish SOC 2 attestation.

Contract checks

OpenAPI lint with Redocly.
Breaking changes: oasdiff against the merge base on main, failing on errors unless the break is listed in breaking-changes-approved.txt.
Realtime contract: a Python check of the WebSocket protocol and its client library.
Migration parity: the Go and Nest migration folders must be identical.
Catalog parity: entitlements, usage meters, notifications and AI catalogs must match across v1 implementations and their contracts.
Route parity, in optional split-deployment checks: each service's internal route inventory must match the OpenAPI spec.
Property-based API checks: Schemathesis exercises selected authenticated operations against the generated OpenAPI contract.

End-to-end suites

The scripts in backend/scripts exercise HTTP contracts and domain journeys. The browser suite infrontend/e2e runs each web frontend against each v1 backend: Next→Go, Next→Nest, Svelte→Go and Svelte→Nest. Isolated API instances can use alternate ports; see frontend/e2e/README.md.

Suite	Covers
`identity-e2e.sh`	Sign-up, verification, sign-in and sign-out, sessions, password reset.
`accounts-e2e.sh`	Workspaces, roles, invitations, ownership transfer.
`entitlements-e2e.sh`	Billing events through Entitlements to Files limits.
`lifecycle-e2e.sh`	Session lifecycle, workspace close and reopen, deletion, CORS.
`trust-e2e.sh`	Terms, step-up, password and email change, TOTP MFA.
`privacy-e2e.sh`	Exports and the operator commands.
`audit-e2e.sh`	The audit log API and chain verification.
`usage-e2e.sh`	The usage view.
`ai-e2e.sh`	AI tasks and credit metering against a fake model.
`billing-e2e.sh`	Trials, checkout, signed webhooks, subscriptions, invoices and usage reconciliation.
`integrations-e2e.sh`	API keys, signed webhooks, MCP connection lifecycle and protected calls.
`notifications-e2e.sh`	Inbox/preferences and durable email delivery behavior.
`chat-e2e.sh`	Conversations, ordered messages, attachments, read cursors, erasure and account lifecycle.
`staff-e2e.sh` · `impersonation-e2e.sh`	Staff role/MFA gates, support access, approvals and audited impersonation lifecycle.
`frontend/e2e`	Browser journeys for auth, MFA/passkeys, billing, files, chat, integrations, MCP consent/OAuth and AI flows across all four web/backend pairings.
`files-e2e.sh`	Folders, trash, quota, upload CORS.
`receipts-e2e.sh`	Retry-safe writes: replay after a lost response or a stale `If-Match`, key reuse, and the receipt lookup.
`realtime-e2e.sh`	The WebSocket protocol.
`cache-e2e.sh`	Two instances sharing Redis, and one with Redis unreachable.

Optional split-deployment run

split-e2e.sh starts, for each backend, a core service, an Account and Files service, an edge in front of both, and a Realtime-only service behind its own edge, with fresh Ed25519 keys and NATS. It runs every suite except the cache suite, checks that edges return 404 for /internal/* and 401 for a client-supplied assertion, and checks route parity.

Load tests with k6

infra/testing provides typed k6 workloads, disposable dependency stacks, tenant fixtures, privacy-checked reports and bounded traffic profiles. Workloads cover Account, Files, Billing, Chat, Webhooks, identity and AI. A manual smoke workflow offers selected Go/Nest workloads; broader profiles run locally or through selected workflows. These checks target disposable infrastructure, not a production service.

Workload	Does
`account/read`	Reads the fixture workspace.
`account/create`	Creates a workspace, replays the same idempotency key, reads it back.
`account/mixed`	Four reads per create. The default.
`account/list`, `account/manage`, `account/fairness`	Pagination, role-aware updates/invitations and a busy tenant's effect on control traffic.
`identity/read`	Authenticated session and profile reads across fixture owners.
`files/lifecycle`, `files/mixed`, `files/purge`	Upload/download, tenant access, exact bytes and cleanup.
`billing/read`, `billing/invoices`	Plan/subscription reads and exact invoice pagination.
`chat/mixed`, `chat/replicas`	Multiple writers, ordered messages, retries and contention across processes.
`webhooks/dispatch`, `webhooks/source`, `webhooks/source-recovery`	Signed delivery, source events and recovery with tenant-owned endpoints.
`ai/run`, `ai/concurrency`, `ai/quota-race`	Run lifecycle, same-key contention and concurrent quota admission.
`account/rename-race`, `health/live`	Version conflicts and independent public health observations.

Profile	Traffic
`smoke`	1 workflow/s for 15 s
`load`	5 workflows/s for 60 s
`spike`	Ramp to 5× the configured rate, hold for 60 s, then recover
`stress`	Staged 2×/5× traffic, sustained hold and recovery
`breakpoint`	Doubling steps through 8× by default; bounded configuration up to 16×
`soak`	5 workflows/s for 30 min by default; maximum 6 hours

These are the standard shared-profile defaults. Workloads can use dedicated contention scenarios; overrides, effective options and requested versus achieved traffic are recorded in their reports. A workflow can make several HTTP requests.

A run passes only if every check passes, fewer than 1% of requests fail, p95 latency stays under one second overall and per operation, and no iteration is dropped. Reports go to infra/testing/reports/<run>/.

make test-setup
make test-smoke BACKEND=go WORKLOAD=account/read
make test-load  BACKEND=nest PROFILE=spike WORKLOAD=account/mixed

Resilience with Toxiproxy

Recovery scenarios inject dependency faults or process failures, then verify durable state, retries, replay, public behavior, event drain and cleanup. NATS recovery is one case; PostgreSQL response/commit faults, AI provider failure, Files storage, Chat, Billing, Webhooks and notification delivery have dedicated scenarios. For example, make test-resilience BACKEND=go SCENARIO=event-recovery:

BaselineCapture existing state and health.
FaultInject a bounded network, dependency or process fault while the workload runs.
EvidenceCheck persisted rows, events, receipts, privacy boundaries and user-visible outcomes.
RecoveryRestore the dependency/process and require replay or retry within a bounded deadline.
CleanupProve exact final state, healthy control work and independently verified cleanup.

Run-specific evidence and limits are recorded in infra/testing/completion.md and the scenario guides.

CI

Local make ci runs 65 default gates for Go, Nest, Python, Next, Svelte, mobile, CLI, contracts, infrastructure and dependency audits. Native k6 fixture checks, database-backed suites and live fault drills are separate selections. Rust, .NET and Python-live are opt-in; Java and Zig have separate build paths. The hosted workflows below have their own triggers and do not imply a hosted run occurred with each local checkpoint.

Workflow · job	Runs
backend · go	Formatting, the architecture test, vet, build, `go test -race`, govulncheck.
backend · nest	Prettier, dependency-cruiser, Oxlint, type-check, build, Vitest, `npm audit`.
backend · contract	OpenAPI lint, oasdiff, realtime contract, migration and catalog parity.
backend · secrets	gitleaks over the full history.
backend · e2e	Main/dev, scheduled, manual or opted-in PR runs: local dependencies, Go/Nest builds, browser journeys, monolith and optional split suites, plus a Redis-cache leg.
frontend · mobile, next, svelte	Type/static checks and tests for mobile and both web apps; see Frontends.
recovery-drills · testing-smoke	Manual workflows for selected bounded Go/Nest recovery and smoke scenarios; they are not part of every pull request.

Dependabot opens weekly updates for both backends and GitHub Actions.

Gaps and roadmap

Gaps

Manual load and recovery evidence qualifies only the scenarios and source revision recorded in each report; it is not a production SLO or a blanket guarantee.
Hosted AI/MCP outcomes depend on provider/model availability and quality; fake-provider coverage does not validate those external behaviors.
Go and Nest are the v1 backends. Python is progressing toward parity; Rust, .NET, Java and Zig remain separate builds and are not the supported v1 starter path.
Split-service deployment checks exist but are optional for the current modular-monolith-first project path.

Later

Production backup and restore qualification against the selected hosting and storage providers.
Production deployment manifests, release ownership and operations integrations for a chosen hosting target.
Provider-specific AI quality, cost and operational limits for the models each product selects.

Current scope and qualification boundaries

Area	Available now	Project-specific follow-up
Contracts and authorization	OpenAPI/Schemathesis selections, role-specific acceptance and security suites, migration/catalog parity.	Add the operations, roles and product-specific constraints required by each generated app.
Operations	Local observability, alert rules, manual k6 profiles and recovery drills with status-only reports.	Set service objectives, alert destinations, capacity budgets and backup policies for the chosen deployment.
Providers	Fake adapters and local fixtures support deterministic checks; selected hosted checks exist for AI, billing and email behavior.	Supply product credentials and verify the provider, model, plan, region and cost behavior being deployed.

Details: infra/testing/README.md and infra/testing/roadmap.md.