THE SHORT ANSWER
A pilot asks whether an AI idea can work. Production asks whether it will keep working, within approved boundaries, when real data, exceptions, customers, staff, and failures enter the picture. The change is as much operational as technical.
Define the job before the model
Name the user, the decision or task, and the outcome that would improve. Separate a generated suggestion from an authoritative bank record or customer action. Then identify the data needed and the cases the system must refuse, escalate, or leave unresolved. That boundary makes evaluation possible.
The NIST AI Risk Management Framework offers a voluntary way to govern, map, measure, and manage AI risks. Its generative AI profile adds considerations specific to generative systems. These resources inform design; they do not replace the bank's own policies or applicable requirements.
Test the entire workflow
Measure quality on representative cases, including incomplete documents, conflicting facts, and requests outside scope. Record sources, model and prompt versions, reviewer decisions, and outcomes. Test access controls and the handoff when the model cannot answer. A high average score can hide failures in the cases that matter most.
Before release, assign an operational owner. Decide who can change the system, how drift is detected, how a bad output is corrected, and when the feature is paused. A manual fallback is often part of a responsible first release.
Keep authority visible
An assistant may summarize evidence; a person or an approved deterministic system should make the consequential decision when that is the intended control. The interface should say what the AI observed, what it inferred, and what remains unverified.
Illustrative example: an internal AI tool drafts a vendor-risk summary. A pilot tests summary quality. Production also needs source links, restricted access, a reviewer who can edit or reject the draft, and a way to find every decision affected by a later data correction.
Original sources
The process explanations above are educational summaries. The linked primary sources govern their own terms and can change; check them for current requirements.