Serving and scaling models

Batch and online inference

Always ask: does the prediction need to be fresh at request time? If predictions can be precomputed (for example daily recommendations per user) then batch plus a lookup store (BigQuery, Bigtable, Firestore) is far cheaper and simpler than an online endpoint.

Where batch inference can run

Batch-prediction options to know: machine type and accelerator for the job, number of replicas (starting and max), explanations can be included, and model monitoring for batch jobs is supported. Large batch jobs split inputs, so ordering of output isn’t guaranteed. Include an ID column to join back.

Online prediction

• Deploy a model to a Vertex AI endpoint, then call predict (REST, gRPC, SDK). Use rawPredict for custom request formats and streaming methods for token streaming from generative models.
• Request size limits apply (roughly 1.5 MB per request for online prediction). Send large payloads as batch or Cloud Storage references.
• A deployed model is a model version on a machine type with min and max replicas. An endpoint can hold several deployed models.

Serving frameworks and containers

Pre- and post-processing belongs in the serving path consistent with training: bake it into the model graph (TFT), the custom container, CPR, or a serving pipeline (inference pipelines chain containers, for example preprocessor then model). Avoid reimplementing it in the client.

Framework tips: PyTorch models are packaged as a .mar archive for TorchServe or served from a custom handler. XGBoost and scikit-learn use their native saved formats (model.bst, model.joblib/pickle). TensorFlow uses SavedModel. Vertex expects specific file names for prebuilt containers, so a “model not found” error often means the artifact has the wrong name or is in the wrong path.

Model registry

Vertex AI Model Registry is the central catalogue of models and versions.

• Register trained models from training jobs, BigQuery ML (model_registry), AutoML, or by importing artifacts and a serving container.
• Each model has versions with aliases (for example default, champion, staging). Deploy by alias so promoting a version is an alias change instead of a redeploy of consumers.
• Attach evaluation metrics, labels, lineage back to training pipeline and dataset, and the container image and explanation settings.
• Governance patterns: models trained in a dev project, promoted to a registry in a prod project after tests and approvals. Use IAM for who can register, deploy and read.
• From the registry you can deploy to an endpoint, run batch prediction, or start evaluation.

Organise by one model per business problem, versions per retrain, and keep the production alias explicit. Use model labels and descriptions for ownership and risk class. Include a model card and fairness notes with high-impact models.

A/B testing and rollout of model versions

An endpoint can hold multiple deployed models and a traffic split that sums to 100%.

• Canary / gradual rollout: deploy v2 with 5-10% of traffic, compare metrics, then ramp up. Roll back by moving traffic to v1, with no redeploy.
• A/B test: split between two versions, then compare business metrics (conversion, revenue), not just model metrics, over enough traffic for statistical significance. Keep users sticky to a variant if the experience needs it (the endpoint splits per request, so do sticky assignment in the application).
• Shadow (mirror) testing: send a copy of live traffic to a new model, log its predictions without returning them, and compare offline. Safest, with no user impact. Implement with request logging or a mirror in your service layer.
• Blue/green: deploy a second endpoint, switch the client or load balancer.
• Champion/challenger with a rolling evaluation using ground truth that arrives later.
• Every deployed model on an endpoint must serve the same input and output schema for a split to work. Each consumes its own replicas, so you pay for both.
• Request-response logging to BigQuery supports comparison.

Scaling online serving

Endpoints: public and private

IAM: callers need aiplatform.endpoints.predict (role Vertex AI User). Use service accounts, not user credentials, in production. Combine with VPC Service Controls and CMEK on regulated workloads.

Autoscaling

• You set min replica count and max replica count. Autoscaling adds replicas when utilisation passes a target (default CPU utilisation 60% for CPU machines, GPU duty cycle for GPU models, or a custom metric such as requests per second or queue depth).
• Min replicas of at least 1 keeps a warm instance for latency. Setting min to 0 isn’t available on classic endpoints (Cloud Run does scale to zero), but newer features support scale-to-zero for some deployments. Cold starts with large models are slow, so prefer a warm minimum when latency matters.
• Min of 2 or more across zones gives availability. A single replica is a single point of failure.
• Scaling lags behind spikes, so keep headroom (lower target utilisation) for bursts.
• Throughput planning: measure requests per second one replica handles at target latency (load test), then size max replicas to peak divided by that. Use larger batches or dynamic batching to raise accelerator utilisation, at some cost to latency.
• Multi-model hosting: co-hosting several models on one VM/replica set cuts cost for many small models. Deployment resource pools share replicas across models.
• Use Cloud Monitoring metrics (prediction/online/prediction_latencies, replicas, cpu/utilization, accelerator/duty_cycle) and alerts. Check error rates (429 = throttled/overloaded, 5xx = model or container faults).

Feature Store at serving time

For models with many features computed from history (user purchases in the last 30 days), look the features up at request time:

1. Compute features offline in BigQuery and keep them in a feature group.
2. Sync a feature view to the online store on a schedule.
3. At prediction, the serving code (or the custom container) calls the online store by entity ID (low-latency read), merges with request features, and calls the model.

Use Bigtable-backed online serving for large scale and optimized online serving for the lowest latency (with some constraints). Using the same feature definitions in training and serving avoids skew. Alternatives: Memorystore (Redis), Bigtable, or Firestore directly when you don’t need the feature store’s governance.

Hardware for serving

• Right-size: GPUs are wasted on small models at low QPS, and CPUs can’t meet latency for big networks.
• Pre-warm and keep a warm minimum for large models, since loading weights takes minutes.
• GPU batching raises throughput, and multi-instance GPU/time sharing can host several small models.
• Provisioned Throughput (reserved capacity) gives predictable performance for Gemini on production workloads. Pay-as-you-go quotas can throttle (429s) during spikes.

Optimising models for production

Goals: lower latency, memory and cost, and higher throughput, with minimal accuracy loss.

Match the technique to the constraint: latency-bound means smaller or faster model, compiled graph, better hardware, caching. Memory-bound means quantisation, pruning, sharding. Throughput-bound means batching, more replicas, better utilisation. Cost-bound means right-size, distil, batch instead of online, Spot for batch jobs.