Testing AI apps across deterministic code, evals, and real UI

An AI application combines deterministic software with stochastic model behavior. Combining both into one end-to-end test makes it hard to tell whether a failure came from parsing, network, model, tool, or UI. Snapshotting one answer becomes flaky whenever wording changes acceptably. A layered strategy lets each test answer one question. As of 2026-07-29, OpenAI's evaluation best practices still recommend task-specific evals, representative datasets, appropriate graders, and continuous evaluation. Playwright provides web-first assertions that retry until observable UI state matches. Vitest supports controlled request mocks and warns that mock state must be cleaned between tests. Implementation steps Test deterministic code first: prompt builders, schema validation, redaction, cost calculations, retry decisions, tool authorization, parsers, and state machines. These tests use fixed inputs and exact outputs without a provider call. it("rejects a write tool without approval", () => { const decision = authorizeTool({ tool: "refund_order", approved: false, scopes: ["orders:read"], }) expect(decision).toEqual({ allowed: false, reason: "approval_required" }) }) Test the provider adapter contract using mocked HTTP or SDK responses. Cover normal output, stream chunks, invalid JSON, empty output, tool calls, 429, timeout, 5xx, and cancellation. Assert the model, configuration, prompt version, timeout, and redacted logs. Do not put a live API call in every unit test. Build an offline eval set from production-like redacted data, sliced by use case, risk, language, and edge case. Prefer deterministic graders for JSON schema, citation presence, tool arguments, and forbidden output. Use a rubric plus human labels for semantic quality, calibrating model graders. Record dataset, prompt, model, config, and grader hashes. Simulate tools returning success, denied, not found, conflict, slow, and malicious content. Verify that a model denied one action does not switch to another unauthorized tool, does not treat tool-output injection as a system instruction, and preserves idempotency when a mutation is retried. Run a small live integration suite with a dedicated project, key, budget, and non-sensitive fixtures to validate current provider schemas and streaming. Keep it on a separate schedule so ordinary pull requests are not entirely blocked by provider instability, but require recent live evidence before a release. Use Playwright for user journeys: loading, streaming, cancel, retry, citations, copy, errors, mobile, and keyboard. Use roles, labels, or test IDs and auto-retrying assertions. Wait for observable outcomes rather than fixed sleeps. await page.getByRole("button", { name: "Send" }).click() await expect(page.getByTestId("answer-status")).toHaveText("Complete") await expect(page.getByTestId("answer")).not.toBeEmpty() Finish with a production canary and monitoring. A side-effect-free synthetic input checks availability, response schema, and latency. Track refusals, tool errors, fallback, cost, and feedback. Redact a production failure before adding it to the regression dataset. Failure and recovery When eval scores vary sharply, freeze dataset, configuration, sampling, and grader versions and rerun to measure variance. Never release from one pass. If a model grader drifts, recalibrate it against a human-labeled set. When mocks pass but live integration fails, the adapter fixture is behind the provider schema. Preserve a redacted response shape, update the contract fixture, and add a regression. Do not expose a raw provider error to users. For flaky E2E, wait on status, DOM, or a network response using Playwright's retries instead of arbitrary seconds. Redact tokens and prompt content from traces, screenshots, console, and network logs. For a production regression, use a feature flag to restore the previous prompt, model, or route and disable high-risk tools. Retain version, dataset, and eval evidence. After repair, run the affected slice, full eval, and canary. Verification commands npm run test:run npm run eval -- --dataset evals/golden.jsonl npx playwright test npm run build Report unit and contract tests, eval slices, live integration, and E2E separately. A combined average must not hide critical failures. Acceptance includes prompt injection, personal-data redaction, tool denial, duplicate mutation, timeout and cancel, offline and error UI, mobile, and keyboard paths. Primary sources OpenAI Evaluation best practices Playwright Assertions Vitest Mocking Requests Internal links Browse technical articles for the AI roadmap and agent permissions. Build a layered AI test harness through the course catalog . Share a redacted failing case through the contact page .