A lender runs a credit model trained six months ago that has a strong production record. A newer candidate trained last week scores better on the hold-out set. Leadership wants evidence on live traffic before any customer-facing change. Which rollout is safest?
Show answer and explanation
Answer: C
Shadow testing sends real requests to the candidate as well, but only the incumbent's predictions are returned. Comparing both sets of outputs on live traffic checks whether the offline gain holds under current conditions, and no applicant sees a change. Only after that evidence is in does a gradual traffic split or full promotion make sense.
Why the other options are wrong
- A. A hold-out win does not guarantee live performance, especially after a distribution shift. Immediate replacement puts every customer at risk from an untested change.
- B. Using customer complaints as the detection method is slow and poor. Exposing all traffic to an unverified model means harm reaches applicants before anyone notices, which is much riskier than checking first.
- D. Arbitrary margins are not a validation method and delay improvements indefinitely. The decision should rest on live comparison evidence, not a threshold chosen in advance.