After a rollout, p95 latency on a serving deployment has climbed sharply, yet accelerator and CPU graphs on the serving nodes look mostly idle. The model call alone benchmarks quickly when run in isolation. What should be checked first?
Show answer and explanation
Answer: C
Idle compute alongside high end-to-end latency suggests the time is spent outside the model forward pass, for example in deserialization, feature parsing, or response encoding inside the serving container. Profiling the full request path identifies that overhead. This diagnostic step should precede any architectural or capacity change.
Why the other options are wrong
- A. Cutting layers changes model quality and addresses a cost the benchmarks have already shown is small. It does not identify where the latency is actually being spent.
- B. Autoscaling controls how many replicas exist, and idle nodes mean capacity is not the constraint. Disabling it would not reduce per-request processing time.
- D. Retraining is slow and aimed at model quality, while the symptom points at the serving path. Changing features without profiling would not reveal the overhead.