This is among the larger domains (about 18% of the exam). Questions start with a notebook model that works on a sample and ask how to train it on all the data, faster, cheaper or more accurately, or why a job fails or a model misbehaves. Expect hardware choices, distribution strategies and tuning, and a few interpretability trade-offs.
Building models
Choosing the framework and architecture
| Data and task | Reasonable starting point |
|---|---|
| Structured/tabular | Gradient-boosted trees (XGBoost, LightGBM) or scikit-learn. Deep networks rarely beat trees on tabular data without a lot of data |
| Images | CNNs or vision transformers, usually transfer learning from a pretrained model |
| Text | Transformers. Fine-tune a pretrained language model or use embeddings plus a simple classifier |
| Sequences and time series | ARIMA/ETS, gradient boosting with lag features, or sequence models (LSTM, transformer, TFT) |
| Recommendations | Matrix factorisation, two-tower retrieval models plus a ranking model |
| Sparse high-cardinality features | Embeddings, wide-and-deep models |
| Very little labelled data | Transfer learning, pretrained embeddings, data augmentation, few-shot prompting of a foundation model |
Frameworks on Google Cloud: TensorFlow/Keras (strong TFX and TPU support), PyTorch (research favourite, supported with prebuilt containers), JAX (TPU-friendly, large-scale research), scikit-learn/XGBoost (classical), Spark ML (cluster scale). Vertex AI offers prebuilt training containers for TensorFlow, PyTorch, scikit-learn and XGBoost, or you bring a custom container.
Practical rules:
- Start simple. Build a baseline (a majority class, linear model or the existing rule). A complex model must beat the baseline by enough to justify its cost. The exam asks about monitoring “against baselines and simpler models” for this reason.
- Transfer learning beats training from scratch when data is limited. Freeze early layers and train the head, then optionally unfreeze to fine-tune.
- Match loss and metric to the business: precision when false positives are expensive (spam moved to trash), recall when false negatives are expensive (disease screening, fraud), F1 or PR-AUC for imbalance, ROC-AUC for ranking quality across thresholds, RMSE/MAE for regression (MAE is robust to outliers), MAPE for relative error (breaks at zero).
- Bias-variance: high training error means underfit (bigger model, more features, train longer). Low training but high validation error means overfit (more data, regularisation, dropout, early stopping, simpler model, augmentation).
Interpretability requirements
Regulated decisions (credit, hiring, health) need explanations. The trade-off: simpler models are transparent, more complex models are accurate but opaque.
| Approach | Notes |
|---|---|
| Inherently interpretable: linear/logistic regression, decision trees, rule lists, GAMs | Coefficients and rules are directly readable. Pick when explainability is a hard requirement |
| Post-hoc attributions on complex models: Vertex Explainable AI | Sampled Shapley (any model, including ensembles and non-differentiable), Integrated Gradients (differentiable models, tabular and images), XRAI (image regions), and feature importance for AutoML tabular. Pick when accuracy needs a complex model but you must still explain |
| Example-based explanations | Return similar training examples (nearest neighbours) as justification |
| What-If Tool / Vertex AI model evaluation slices | Inspect behaviour by subgroup, change features, test counterfactuals |
| Monotonic constraints | Force a feature’s effect direction (more income never lowers credit score). Supported in boosted trees |
Local explanations are per prediction. Global are overall feature importance. Configure explanations when uploading the model (explanation metadata and parameters), then call explain on the endpoint or ML.EXPLAIN_PREDICT in BQML.
Training models
Organising and ingesting data
- Cloud Storage for files at scale and BigQuery for tabular. Use the BigQuery Storage Read API (through the TensorFlow I/O connector, or a Vertex AI managed dataset) so training streams rows without exporting CSVs.
- Ingestion options by file type:
| Source | How |
|---|---|
| CSV / JSON | Fine for small data. For scale convert to Parquet/Avro/TFRecord |
| Images / audio / video | Files in Cloud Storage with a manifest. Use tf.data or a PyTorch DataLoader with parallel reads and prefetch. Mount with Cloud Storage FUSE |
| Hadoop/HDFS data | Copy with Storage Transfer Service or DistCp, or read through the Cloud Storage connector from Dataproc |
| Operational databases | Replicate with Datastream to BigQuery, or query with BigQuery federation. Don’t hammer production |
| Streaming events | Pub/Sub then Dataflow into BigQuery or Cloud Storage |
- Keep the input pipeline from becoming the bottleneck:
tf.datawithinterleave,map(num_parallel_calls=AUTOTUNE),cache,prefetch, and sharded files. Low GPU/TPU utilisation with a high CPU wait usually points here.
Training on Vertex AI
| Option | What it is | Pick when |
|---|---|---|
| AutoML / Tabular Workflows | Managed training from a dataset | Low-code (previous domain) |
| Custom training: prebuilt container | You supply a Python package or script. Google supplies the framework image | Standard framework, no special system dependencies |
| Custom training: custom container | Your Docker image in Artifact Registry | Special libraries, non-Python languages, exact environments |
| Training pipelines | Training plus model upload in one resource | Simple one-step automation |
| Custom jobs / worker pools | Defines machine type, accelerators, replica count per worker pool | Anything custom, including distribution |
| Ray on Vertex AI | Managed Ray clusters | Ray-based distributed training and tuning |
| Kubeflow on GKE | Self-managed orchestration on your clusters | You must stay on Kubernetes or hybrid. More operations overhead |
| Vertex AI Pipelines | Serverless KFP/TFX | Repeatable multi-step workflows (pipelines domain) |
Details:
- Jobs run in ephemeral compute you pay for only while the job runs. Output a model to a Cloud Storage
AIP_MODEL_DIRpath, then register it. - Use Spot VMs / preemptible capacity for cost savings on fault-tolerant jobs. This needs checkpointing so the job resumes after preemption. Flex-start/DWS and reservations help secure scarce GPUs.
- Run the job with a service account granting only the data and bucket access needed, and attach it to a VPC with private service access/PSC interface when data sources are private. Use CMEK where required.
- Local-to-cloud workflow: test the container locally with a small sample, then submit with
CustomJob,CustomTrainingJoborgcloud ai custom-jobs create. - For large datasets in BigQuery, use the managed dataset reader (
AIP_TRAINING_DATA_URIvariables) so splits are done for you.
Distributed training
Distribute when a model or dataset doesn’t fit one device, or training takes too long.
| Strategy | Idea | Notes |
|---|---|---|
| Data parallelism | Every worker holds a full model copy. Each trains on a different data shard, and gradients are combined | The usual default. Needs the model to fit in one device’s memory |
| Model parallelism (tensor/pipeline/FSDP sharding) | The model is split across devices | For models too big for one accelerator (large language models) |
| Synchronous | All workers update together each step (all-reduce) | Better convergence, needs fast interconnect. MirroredStrategy (one machine, many GPUs), MultiWorkerMirroredStrategy, TPUStrategy |
| Asynchronous (parameter server) | Workers push to parameter servers independently | Tolerates slow or preemptible workers. Convergence can be noisier. ParameterServerStrategy |
On Vertex AI, worker pools define the cluster roles: pool 0 is the chief/primary, pool 1 workers, pool 2 parameter servers, pool 3 evaluators. Vertex sets the TF_CONFIG or equivalent environment so the framework finds peers.
- Reduction Server is a Vertex AI feature that speeds the all-reduce of gradients for GPU data-parallel training (NCCL-based, TensorFlow and PyTorch). It uses extra low-cost CPU nodes as reducers so gradient aggregation uses less bandwidth. Choose it when multi-node GPU training is network-bound on gradient exchange.
- Horovod is an open-source all-reduce framework for TensorFlow, PyTorch and Keras across multiple GPUs and nodes. Pick it when the code already uses it or the team wants one distribution API across frameworks. PyTorch
DistributedDataParallelandtorchrunare the native alternatives. - TPU training: use
TPUStrategy(TensorFlow) or JAX/PyTorch-XLA. TPU pods/slices scale across many chips with a fast interconnect. Data must be fed quickly (TFRecord on Cloud Storage in the same region). - Batch size and learning rate: with N workers the global batch grows by N. Scale the learning rate (with warmup) accordingly.
- Reliability in long jobs: periodic checkpoints to Cloud Storage, automatic restart (
restart_job_on_worker_restart), elastic/fault-tolerant training for preemptible nodes, and avoiding a single point of failure in async training.
Hyperparameter tuning
- Vertex AI hyperparameter tuning jobs run many trials of your training code with different values. Define a metric (reported with the
cloudml-hypertunelibrary or by writing to a metrics file), a goal (maximise or minimise), parameter specs (double, integer, discrete, categorical with scale types such as log),maxTrialCountandparallelTrialCount. - Algorithms: default Bayesian optimisation (learns from earlier trials, so trade-off: more parallel trials mean less learning), grid (small discrete spaces) and random. Use log scale for learning rate.
- Early stopping ends unpromising trials.
- Use a validation set for the tuning metric, never the test set.
- AutoML and BQML (
NUM_TRIALS) tune automatically. Vertex Vizier is the black-box optimisation service underneath, usable for any metric, not just ML.
Common trap: raising parallelTrialCount too high makes Bayesian search nearly random. Raising maxTrialCount is what usually buys better results.
Troubleshooting training failures
| Symptom | Likely causes and fixes |
|---|---|
| Loss is NaN or explodes | Learning rate too high, unnormalised inputs, bad data (inf/NaN), exploding gradients (clip them), numerical issues in mixed precision |
| Loss doesn’t decrease | Learning rate too low or too high, bad labels, wrong loss, vanishing gradients (activation, normalisation, skip connections), bug in preprocessing |
| Training accuracy high, validation low | Overfitting: regularise, add data, augment, early stop, simplify |
| Both low | Underfitting: larger model, better features, longer training, lower regularisation |
| Perfect validation score | Leakage or duplicated rows across splits |
| Out of memory | Reduce batch size, use gradient accumulation, mixed precision, gradient checkpointing, a larger-memory accelerator, or model/sharded parallelism |
| Low accelerator utilisation | Input pipeline bottleneck, tiny batch size, host-to-device transfers, CPU preprocessing. Use the Profiler (TensorBoard) |
| Job fails to start | Quota/stock-out (accelerator quota, region capacity), IAM (service account can’t read data or pull the image), bad container entrypoint, wrong image architecture |
| Permission errors reading data | The job’s service account (not yours) needs access. The Vertex AI service agent needs Artifact Registry access |
| Multi-worker hangs | Mismatched cluster spec, firewall between workers, one worker preempted in synchronous training |
| Different results across runs | Seeds, nondeterministic ops, shuffled splits not fixed |
Look in Cloud Logging (ml.googleapis.com job logs), the job’s error status, Cloud Monitoring metrics for CPU, accelerator and memory, and TensorBoard profiler traces.
Fine-tuning foundation models
- Supervised fine-tuning (SFT) on prompt-response pairs (JSONL in Cloud Storage) adapts a Gemini or open model to a task, style or output format. The managed Vertex AI service does the tuning. You specify dataset, epochs and learning-rate multiplier.
- Parameter-efficient fine-tuning (PEFT / LoRA) trains small adapter weights instead of all parameters: far less memory and cost, and it is the usual choice on open models in Model Garden (self-deploy plus notebook recipes and training containers).
- Full fine-tuning updates all weights. It needs the most compute and data.
- Preference tuning (RLHF-style, DPO) aligns outputs with human preference. Distillation trains a smaller student model from a larger teacher to cut latency and cost.
- Adapt rather than tune when you can: better prompts, few-shot, RAG or grounding. Tune when prompting can’t enforce behaviour, you have a few hundred or more good examples, and you need consistent format, tone or domain vocabulary.
- Evaluate the tuned model against the base model on a held-out set. Watch for catastrophic forgetting and overfitting to a small dataset.
- Deploy tuned open models to a Vertex AI endpoint. Tuned Gemini models are served through managed endpoints.
Choosing hardware
| Accelerator | Strengths | Choose for |
|---|---|---|
| CPU | Flexible, cheap, large memory | Classical ML (scikit-learn, XGBoost), data preprocessing, small models, low-volume inference |
| GPU (NVIDIA T4, L4, A100, H100, H200 and newer) | Parallel float math, broad framework support, large memory on A100/H100 | Deep learning training, medium and large models, most PyTorch work, fine-tuning LLMs. T4/L4: cheaper inference and small training. A100/H100: large training and big models |
| TPU (v5e, v5p, v6e Trillium, and newer) | Matrix-multiply chips built for large-batch deep learning with high interconnect bandwidth | Large TensorFlow, JAX or PyTorch/XLA models, big batches, very large-scale training and serving. Not for models that rely on custom ops with no XLA support, or tiny batch sizes |
| Edge (Edge TPU, mobile NPUs) | Tiny, low power, offline | On-device inference, with models converted to TF Lite or quantised |
Selection guidance:
- Right-size: don’t use A100s for a small tabular model. Check utilisation and move down if the GPU is idle.
- TPUs shine when the model uses large dense matrix operations and the input pipeline can keep up. Wanting the highest performance per dollar on large TensorFlow or JAX jobs points to TPUs.
- GPU memory (not just speed) limits model size and batch. Larger models need A100/H100-class or sharding.
- Cost levers: Spot VMs, committed use discounts, right-sized machine types, smaller models (distillation, quantisation), mixed precision (bfloat16/float16) for roughly 2x speed and half the memory.
- Quota and location: accelerators are regional and quota-limited. Request quota early.
- Multi-GPU on one VM (for example 8xA100) avoids network all-reduce cost compared with many small VMs.
Exam checklist
- Baseline first, transfer learning for small data, trees for tabular, transformers for text.
- Interpretability required: simple models, or Explainable AI (Sampled Shapley, Integrated Gradients, XRAI) on complex ones.
- Use sharded, binary data, same region, a fast input pipeline. BigQuery Storage Read API for tabular.
- Custom training: prebuilt container for standard frameworks, custom container for special dependencies, worker pools for distribution.
- Data parallel is the default. Model parallel only when the model doesn’t fit. Sync needs fast links, async tolerates stragglers.
- Reduction Server speeds multi-node GPU gradient all-reduce. Horovod is a portable all-reduce framework.
- Hyperparameter tuning: Bayesian by default, log scale for learning rate, validation metric, more trials beat more parallelism.
- NaN loss means learning rate or data. Train/validation gap means overfit. Perfect scores mean leakage. OOM means batch or memory.
- Spot VMs need checkpoints. The job’s service account needs the data permissions.
- Fine-tune (SFT, LoRA) for behaviour. RAG for knowledge. Distil for latency and cost.
- CPU for classical, GPU for general deep learning, TPU for large matrix-heavy training, edge for on-device.