A deployed endpoint's response time rose from 50 ms to 500 ms. Traffic has grown, yet CPU and GPU utilisation on the serving nodes stays low. What is the most likely cause?
Show answer and explanation
Answer: D
Latency rising while processors idle points to a bottleneck before computation: requests wait for a free slot rather than for CPU time. If each instance accepts few concurrent requests, or too few instances are deployed, the queue lengthens as traffic grows. Raising the per-instance concurrency limit or scaling out adds parallel capacity. Retraining, drift and file size affect accuracy or load time, not the size of a queue in front of idle hardware.
Why the other options are wrong
- A. Retraining changes predictions, not how quickly requests are admitted to an under-provisioned service.
- B. Drift would show as worse accuracy in monitoring, while response time and utilisation would look normal.
- C. A larger file can slow cold starts, but it would not leave hardware idle while every request waits.