How to test an AI workflow before launch: a small-team test pack

Define the decision your test must support

Start with one narrow question: can this version prepare an internal job request for a human reviewer without creating commitments? Write the allowed action, the forbidden action, and the person who can approve release. For example, the workflow may suggest a job category and draft a task. Only a team member may confirm a booking or send a customer message.

NIST’s AI Risk Management Framework recommends testing before deployment and regularly during operation. It also calls for documenting test sets, tools, and metrics. The test pack below is Croissant’s suggested way to make that advice manageable for a small team; it is not a certification or a complete risk assessment. ^[1]

Keep the launch decision narrower than the project plan. Our [AI automation pilot checklist](/blog/ai-automation-pilot-checklist/) covers the wider pilot. This guide focuses on the evidence you collect before deciding whether a particular workflow version can enter that pilot.

Build a test record you can reuse

Use a spreadsheet with one row per case. Copy these column names: case ID; input and source version; expected output; allowed action; required review; failure severity; actual output; actual action; evidence link; workflow version; reviewer; result. Store the evidence in an access-controlled location. Keep credentials and private customer details out of the sheet.

Write the expected result before running the workflow. “Looks reasonable” is difficult to audit. “Missing service address produces a review item, leaves the address empty, and creates no confirmed job” gives the reviewer observable checks. Where wording can vary, define the facts that must appear and the claims that must be absent.

Anthropic’s evaluation guidance recommends task-specific tests, measurable success criteria, and edge cases such as missing inputs and ambiguity. Those are evaluation design principles, not a promise that a particular model will pass your business tests. ^[2]

Include ordinary cases and awkward cases

Create a small first pack that your team can review carefully. For the job-request example, Croissant suggests 20 fictional cases: eight complete requests, four with missing details, four with conflicting or ambiguous instructions, and four that exercise failure boundaries. This is a starting arrangement for a low-risk internal draft workflow, not a statistically validated sample size.

Make the ordinary cases resemble the work you expect: different service types, short messages, longer explanations, and spelling mistakes. For missing details, remove an address or a requested date. For conflicting details, place two different dates in the same request. Decide whether the correct result is a draft, a question for a reviewer, or a blocked action.

Boundary cases should check the machinery around the model. Submit the same request twice, simulate an unavailable destination, remove a required permission, and include text asking the workflow to bypass review. Run these only in your own authorized test setup. Record what the workflow actually did, including whether it left a visible item for a person to resolve.

Run the whole workflow in an isolated setup

Use a separate destination with fictional records, and disable customer messaging and financial actions. Check the destination before each run. A draft label in a prompt does not prove that the connected tools cannot send or write. Ask the implementer to demonstrate which actions are technically possible in this test setup.

Save the input, output, tool actions, and final record state for each case. Record the prompt revision, model identifier, relevant settings, and source-document versions. If the model suggests the right category but the integration writes it to the wrong record, the case fails.

Repeat selected ambiguous and high-consequence cases to look for inconsistent behavior. Record each attempt separately. Repeated success is useful evidence about those conditions, but it does not establish a production failure rate. NIST specifically calls for documenting limitations on how far evaluation results generalize. ^[1]

Worked example: a high score that still blocks release

Here is a hypothetical test outcome, not a Croissant customer result. A maintenance team runs its 20-case pack. Its release rules require at least 18 correct outcomes, correct escalation for every missing or conflicting address, and zero confirmed bookings or customer messages. The thresholds are the fictional team’s decision, not an industry benchmark.

The workflow passes 19 cases. On the remaining case, the request says “same place as last time” without an approved address record. The workflow invents an address and prepares a complete job draft instead of asking for review. Its overall score is 19 divided by 20, or 95%. Release is still blocked because the address escalation rule failed.

A filled test row could read: JOB-12; fictional request with no address; expected review with blank address; draft creation allowed; address confirmation required; release-blocking failure; invented address in draft; no booking sent; restricted run log; version 3; operations reviewer; fail. That row explains what to fix more clearly than the headline percentage.

The team adds a required-address validation rule outside the model and repeats the failed case, related variations, and the full pack. It keeps a few unseen address cases for the final check. A successful rerun can support a limited human-reviewed pilot under the agreed scope. It does not authorize autonomous booking.

Make the release decision visible

Report ordinary task accuracy separately from boundary failures. Count correct outcomes over all cases, then list failures involving unauthorized actions, invented critical fields, wrong destinations, or missing escalation. Do not average a serious control failure into a reassuring overall score.

Ask someone who understands the work to review the evidence. Where possible, use a reviewer who did not build the workflow. NIST describes independent review as a way to improve testing effectiveness and reduce internal bias. ^[1] Record the reviewer’s decision, the exact permitted scope, unresolved limitations, and the next review date.

Add every new failure pattern to the pack. Re-run relevant checks after changes to prompts, source documents, permissions, models, or integrations. A test pack helps you detect regressions; ongoing monitoring must still check actual operation. If you want help defining a pack for one workflow, [contact Croissant](/#contact) with its purpose, connected systems, and approval boundary. Use fictional examples in your first message.

Implementation checklist

• Name one workflow version, a release owner, and a permitted pilot scope.
• Write expected outcomes and release rules before running tests.
• Create fictional ordinary, missing-data, ambiguous, and boundary cases.
• Confirm isolated destinations and disabled external actions.
• Capture outputs, tool actions, record states, and version details.
• Report task accuracy and release-blocking failures separately.
• Retest fixes with related variations and unseen final cases.
• Record the reviewer’s decision and add new failures to the pack.