Twelve acceptance tests for an AI-built SaaS before real customers depend on it

Start with one critical journey

Do not begin by testing every screen. Name the one result a paying user must complete, then write the journey as a state transition. A typical SaaS path might be:

create account
→ join the correct workspace
→ submit the core input
→ receive a persisted AI result
→ pay or consume entitlement
→ return later and recover the same state

Every test below should be adapted to that journey. The test is useful only when it has a defined starting state, action, expected result, failure state, and evidence.

1. Identity is established once

Risk: duplicate accounts, mismatched social and email identities, expired links, and partial onboarding create records the user cannot recover.

Acceptance test: create an account through every supported method, repeat the flow with the same email, expire and reuse the verification link, and interrupt onboarding before completion. The system should either resume the same identity or explain the conflict without creating silent duplicates.

Evidence: user records, authentication logs, visible error states, and a repeatable account-recovery path.

2. One tenant cannot cross into another

Risk: a polished interface can hide weak database policies. Changing an ID in a URL or request may expose another customer’s data.

Acceptance test: create two organisations and users with different roles. Attempt to read, update, delete, export, and reference the other organisation’s records through the client, API, background job, storage URL, and any administrative helper.

Expected result: every boundary fails closed. The denial should be enforced by the server or database policy, not only by a hidden button in the interface.

3. Payment changes entitlement exactly once

Risk: the checkout succeeds while the product remains locked, or a repeated event grants the same credit more than once.

Acceptance test: run successful, failed, abandoned, delayed, refunded, and disputed payments. Replay the same provider event. Deliver events out of order. Verify that the customer’s entitlement, invoice, internal ledger, and provider state reconcile.

same provider event + same event identifier
= one economic effect

Never infer a payment from a browser redirect alone. Use the provider’s verified server event and an idempotent internal transition.

4. Webhook retries do not create duplicate work

Risk: providers retry. Networks delay. A handler times out after doing the work but before returning success.

Acceptance test: send the same event multiple times, send later events before earlier ones, interrupt the handler after the database write, and retry while another worker is processing the same object.

Expected result: each event is recorded, deduplicated, and safe to replay. Downstream emails, credits, jobs, or notifications occur once unless the product deliberately supports another rule.

5. AI output conforms to a contract

Risk: a model returns plausible prose where the product expects fields, citations, scores, actions, or a safe refusal.

Acceptance test: test normal, empty, adversarial, multilingual, oversized, ambiguous, and malformed inputs. Validate the model output against a schema. Record which fields may be absent, which values require evidence, and what the interface does when validation fails.

{
  "status": "complete | uncertain | failed",
  "result": {},
  "evidence": [],
  "warnings": []
}

A retry should not silently change a high-stakes answer without preserving the model, prompt, input, validation result, and output version used.

6. Uncertainty is visible rather than disguised

Risk: users treat an attractive result as authoritative even when retrieval failed, evidence was missing, or the model was unsure.

Acceptance test: remove a source document, force a low-confidence retrieval, provide conflicting evidence, and make the model provider unavailable. The product should distinguish “no evidence,” “conflicting evidence,” “provider failure,” and “completed result.”

For legal, financial, medical, hiring, or other consequential workflows, a safe uncertainty state is a product requirement, not a cosmetic detail.

7. Interrupted jobs resume or fail cleanly

Risk: long-running transcription, document analysis, media processing, agent work, or batch evaluation becomes stuck in a permanent “processing” state.

Acceptance test: terminate the worker, restart the deployment, exhaust a rate limit, lose a provider response, and disconnect the client. Verify timeout, retry, cancellation, duplicate prevention, and the final state shown to the user.

Evidence: a job identifier, attempts, last error, next retry, terminal status, and a user-safe recovery action.

8. Rate limits and budgets fail predictably

Risk: a launch creates an unexpected provider bill or collapses when users reach a third-party limit.

Acceptance test: exceed user, workspace, provider, and system limits. Test concurrent requests. Confirm that the product applies quotas before expensive work, communicates reset or upgrade rules, and does not leave partially charged operations without a clear outcome.

Track cost per successful customer outcome, not only cost per model call.

9. The product can explain a failure

Risk: support receives “it did not work” while logs contain no customer, request, job, provider, or deployment context.

Acceptance test: trigger a failure in every external dependency and verify that the team can answer: who was affected, what action failed, when it happened, which version ran, whether money or data changed, and whether a retry is safe.

Logs should use correlation identifiers and exclude secrets, full payment details, private documents, raw tokens, and unnecessary model inputs.

10. Secrets and privileged operations stay server-side

Risk: an AI-generated frontend ships provider keys, administrative database credentials, or privileged logic to the browser or mobile bundle.

Acceptance test: inspect the built assets, network requests, environment handling, source maps, logs, CI output, and client storage. Attempt privileged operations with an ordinary session.

Expected result: public keys are intentionally public, private credentials never reach the client, and sensitive actions are authorised again on the trusted side.

11. Export and deletion match the promise

Risk: the interface says data was deleted while files, embeddings, jobs, backups, analytics, or provider copies remain without an explained retention rule.

Acceptance test: export a complete account, delete a single object, delete the account, revoke a connected provider, and verify every documented retention boundary. Test what happens to queued work and shared links after deletion.

The product should state what is deleted immediately, what is retained, why it is retained, who can access it, and when it expires.

12. A bad deployment can be reversed

Risk: the release changes a schema, job contract, storage format, or entitlement rule that the previous version cannot read.

Acceptance test: deploy the release candidate to a production-like environment, run the critical journey, simulate a failure after migration, and execute the rollback or forward-fix plan. Verify background workers and clients that may remain on the old version.

Release evidence: versioned migrations, backups, health checks, monitored error thresholds, a named decision-maker, and a written rollback condition.